Generate a Robots.txt file for your website. Once you generate this file you will need to save it as robots.txt and upload it to the root of your website
The robots.txt file is a special file that tells search engine and specialty robots, spiders, and crawlers what pages and files they are allowed to index on your website (this is done by disallowing access to certain files and directories). This is a powerful tool, but special care must be taken to make sure it is created correctly.
For more detail see: http://webseodesign.com/resources/robots-txt-files.htm
