The sign of a healthy bed for growing vegetables or flowers is the presence of lots of worms when you dig into it.  But worms must have something to eat or they will migrate out of your beds.  Make sure you have plenty of 'food' for the worms so they don't eat your plants.  Things worms love to eat are:

Compost (finished or unfinished)

Manures (rabbit pellets won't harm your plants)

Mulches (small enough that they break down quickly to a size the worms can handle, like straw)

Wet leaves and decaying matter

They will also eat vegetable scraps if you put them through a blender first, and bury it under the soil.