Once again, I think moderation is key UNLESS you already have a health reason for avoiding a certain food. If you're prone to migraines, arthritis, IBS, Thyroid trouble and lots of other conditions, certain foods are triggers and may exacerbate this and make you feel much worse.
As for babies, breast is always best, formula milks (whether based on cows milk, soy, or whatever) have all kinds of rubbish in them that's not necessarily right for babies. Soy formula milks tend to be LOADED with sugar for a start. Who knows what that level of all those ingredients does to the hormones and development of new born babies???