For one week I removed wheat only and my symptoms got noticeably better. I still had oats, barley and spelt. However when I cut those out my symptoms pretty much went away.
I believe some people just don't tolerate grains as well as others. White rice seems fine with me and I've read that it is often the easiest to digest. Amaranth, buckwheat and quinoa are not really grains so they probably don't digest the same way. Gluten-containing grains seem to be the worst for me. I never would have guessed this a year ago when I was living on white bread and doing relatively ok with my IBS-D. However I was also drinking tons of teas, using peppermint caps, gas-stop, Imodium, ginger etc...
I have had traces of gluten in sauces or contaminated foods when I'm eating out. I haven't really noticed this making a difference in my symptoms.