There's an obligatory "your mileage may vary" on this one and if you're visiting on a Saturday or during summer holidays, everywhere's liable to be pretty crowded.
However, there are some places that I typically find quieter:
Blockbuster Café in Parc Walt Disney Studios
The Old Mill in Fantasyland
Pizzeria Bella Notte in Fantasyland
Hakuna Matata in Adventureland
The Coffee Grinder in Main Street USA
Plus there's a popcorn and drinks stand near Le Pays des Contes des Fées in Fantasyland that's always quite quiet.
On the other hand, the following are often very crowded:
Last Chance Café in Frontierland (high footfall and standard fast food)
Au Chalet de la Marionette in Fantasyland (lots of seating but never enough tills open)
Casey's Corner in Main Street USA (must be where they put the new CMs)
Restaurant en Coulisse in Parc Walt Disney Studios (very high footfall area so guests gravitate towards it)
I can't say I've ever had much if any delay at the men's toilets; obviously can't comment on ladies' ;p