It seems the first year they need to settle in and grow roots. Then, the second year, they really put on new growth and flowers. This time of year, you can add some compost to the top of the dirt around the plant and work it in with a rake. Be sure to water and mulch. Next spring, try fertilizing with a mix that has a high number for the phosphorous to encourage flowers. I think it will make a difference. Also, vitex likes full sun and heat. If they are in a shady spot, you might look for a different spot in the garden.
Thanks for writing!