My boss sent me a few days ago a link to a very useful article regarding robots.txt and how to manage it wisely. As you probably now, robots.txt management and building can be cruacial to your sote preformance. You have to know exactly when to put noindex, nofllow, disallow, allow etc. This file ref