SEO Friendly Server Security against spammers and hackers – Robots.txt allowing only known search engines

For SEO Friendly Server Security against spammers and hackers  – Robots.txt allowing only known search engines use the following robots.txt file on your web server.

This will prevent any other web scrapping tool spider to check your website and grab content from it and will even secure site agaionst such scripts detecting security holes in your web applications

# robots.txt allowing only known search engines

User-agent: Googlebot
Disallow:

User-agent: googlebot-image
Disallow:

User-agent: googlebot-mobile
Disallow:

User-agent: MSNBot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: twiceler
Disallow:

User-agent: Gigabot
Disallow:

User-agent: Scrubby
Disallow:

User-agent: Robozilla
Disallow:

User-agent: Nutch
Disallow:

User-agent: ia_archiver
Disallow:

User-agent: baiduspider
Disallow:

User-agent: naverbot
Disallow:

User-agent: yeti
Disallow:

User-agent: yahoo-mmcrawler
Disallow:

User-agent: psbot
Disallow:

User-agent: asterias
Disallow:

User-agent: yahoo-blogs/v3.9
Disallow:

User-agent: *
Disallow: /

# disabled for the moment as exact sitemap url is not known

#Sitemap: http://www.yoursite.com/sitemap.gz