robots.txt - habt ihr oder habt ihr nicht?

The simplest robots.txt file uses two rules:
 
[LIST]
[*][B]User-agent[/B]: the robot the following rule applies to
[*][B]Disallow[/B]: the URL you want to block
[/LIST]
 These two lines are considered a single entry in the file. You can  include as many entries as you want. You can include multiple Disallow  lines and multiple user-agents in one entry.
 Each section in the robots.txt file is separate and does not build upon previous sections. For example:
 User-agent: * Disallow: /folder1/  User-Agent: Googlebot Disallow: /folder2/  In this example only the URLs matching /folder2/ would be disallowed for Googlebot.
 [B]User-agents and bots[/B]


 A user-agent is a specific search engine robot. The [URL='http://www.robotstxt.org/wc/active.html']Web Robots Database[/URL]  lists many common bots. You can set an entry to apply to a specific bot  (by listing the name) or you can set it to apply to all bots (by  listing an asterisk). An entry that applies to all bots looks like this:
 User-agent: *  Google uses several different bots (user-agents). The bot we use for our web search is [B]Googlebot[/B].  Our other bots like Googlebot-Mobile and Googlebot-Image follow rules  you set up for Googlebot, but you can set up specific rules for these  specific bots as well. 
 [B]Blocking user-agents[/B]


 The Disallow line lists the pages you want to block. You can list a  specific URL or a pattern. The entry should begin with a forward slash  (/).
 
[LIST]
[*][B]To block the entire site[/B], use a forward slash. Disallow: /
[*][B]To block a directory and everything in it[/B], follow the directory name with a forward slash. Disallow: /junk-directory/
[*][B]To block a page[/B], list the page. Disallow: /private_file.html
[*][B]To remove a specific image from Google Images[/B], add the following: User-agent: Googlebot-Image Disallow: /images/dogs.jpg
[*][B]To remove all images on your site from Google Images[/B]: User-agent: Googlebot-Image Disallow: /
[*][B]To block files of a specific file type (for example, .gif)[/B], use the following: User-agent: Googlebot Disallow: /*.gif$
[*][B]To prevent pages on your site from being crawled, while still displaying AdSense ads on those pages[/B],  disallow all bots other than Mediapartners-Google. This keeps the pages  from appearing in search results, but allows the Mediapartners-Google  robot to analyze the pages to determine the ads to show. The  Mediapartners-Google robot doesn't share pages with the other Google  user-agents. For example: User-agent: * Disallow: /  User-agent: Mediapartners-Google Allow: /
[/LIST]
 Note that directives are case-sensitive. For instance, Disallow: /junk_file.asp  would block [URL]http://www.example.com/junk_file.asp[/URL], but would allow  [URL]http://www.example.com/Junk_file.asp[/URL]. Googlebot will ignore white-space  (in particular empty lines)and unknown directives in the robots.txt.
 Googlebot supports [URL='http://sitemaps.org/protocol.php#informing']submission of Sitemap files through the robots.txt file[/URL]. 
 [B]Pattern matching[/B]


 Googlebot (but not all search engines) respects some pattern matching.
 
[LIST]
[*][B]To match a sequence of characters, use an asterisk (*).[/B] For instance, to block access to all subdirectories that begin with private: User-agent: Googlebot Disallow: /private*/
[*][B]To block access to all URLs that include a question mark (?)[/B]  (more specifically, any URL that begins with your domain name, followed  by any string, followed by a question mark, followed by any string): User-agent: Googlebot Disallow: /*?
[*][B]To specify matching the end of a URL[/B], use $. For instance, to block any URLs that end with .xls: User-agent: Googlebot  Disallow: /*.xls$ You can use this pattern matching in combination with the Allow  directive. For instance, if a ? indicates a session ID, you may want to  exclude all URLs that contain them to ensure Googlebot doesn't crawl  duplicate pages. But URLs that end with a ? may be the version of the  page that you do want included. For this situation, you can set your  robots.txt file as follows: 
User-agent: * Allow: /*?$ Disallow: /*? The [B]Disallow: / *?[/B] directive will block any URL that  includes a ? (more specifically, it will block any URL that begins with  your domain name, followed by any string, followed by a question mark,  followed by any string).
 The [B]Allow: /*?$[/B] directive will allow any URL that  ends in a ? (more specifically, it will allow any URL that begins with  your domain name, followed by a string, followed by a ?, with no  characters after the ?).
[/LIST]