如何写好WordPress网站的robots.txt文件？
robots.txt文件要放在网站根目录下，最基本的检查方法就是用你的域名后面直接跟上robots.txt访问，如果能访问，那放置的位置就对了。找到份代码，[*]User-agent: *
[*]Disallow: /cgi-bin/
[*]Disallow: /wp-admin/
[*]Disallow: /wp-content/cache/
[*]Disallow: /wp-content/languages/
[*]Disallow: /wp-content/plugins/
[*]Disallow: /wp-content/themes/
[*]Disallow: /wp-content/upgrade/
[*]Disallow: /wp-includes/
[*]Disallow: /comments/
[*]Disallow: /category/
[*]Disallow: /tag/
[*]Disallow: /page/
[*]Disallow: /feed/
[*]Disallow: /author/
[*]Disallow: /trackback/
[*]Disallow: /2010/
[*]Disallow: /2011/
[*]Disallow: /2012/
[*]Disallow: /2013/
[*]Disallow: /*/feed/
[*]Disallow: /*/trackback/
[*]Disallow: /*?
[*]Disallow: /*/*?
[*]Disallow: /*/*/*?
[*]Disallow: /*.php$
[*]Disallow: /*.js$
[*]Disallow: /*.inc$
[*]Disallow: /*.css$
[*]
[*]# Google Image
[*]User-agent: Googlebot-Image
[*]Disallow:
[*]Allow: /
[*]
[*]# Google AdSense
[*]User-agent: Mediapartners-Google*
[*]Disallow:
[*]Allow: /
[*]
[*]# digg mirror
[*]User-agent: duggmirror
[*]Disallow: /
[*]
[*]# Alexa archiver
[*]User-agent: ia_archiver
[*]Disallow: /
[*]
[*]Sitemap:http://www.xxx.com/sitemap.xml
[*]Sitemap:http://www.xxx.com/sitemap_baidu.xml复制代码问题是这份代码适用于中文站用于百度，我是做英文站要适用于google, 以上代码怎样改成适用英文站的？
对于代码 一窍不通...
主要疑问是31----47行的代码，既然是英文站，这几行代码应该是允许的吧？中文站才禁止抓取？
页:
[1]