discourse/app/views/robots_txt/no_index.erb

User-agent: googlebot
Allow: <%= Discourse.base_path + "/" %>
Disallow: <%= Discourse.base_path + "/uploads/*" %>

User-agent: *
Disallow: <%= Discourse.base_path + "/" %>
FEATURE: let Google index pages so it can remove them Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag 2020-05-10 21:14:21 -05:00			`User-agent: googlebot`
Replace `base_uri` with `base_path` (#10879) DEV: Replace instances of Discourse.base_uri with Discourse.base_path This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467. 2020-10-09 06:51:24 -05:00			`Allow: <%= Discourse.base_path + "/" %>`
			`Disallow: <%= Discourse.base_path + "/uploads/*" %>`
FEATURE: let Google index pages so it can remove them Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag 2020-05-10 21:14:21 -05:00
Fix all the errors to get our tests green on Rails 5.1. 2017-08-30 23:06:56 -05:00			`User-agent: *`
Replace `base_uri` with `base_path` (#10879) DEV: Replace instances of Discourse.base_uri with Discourse.base_path This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467. 2020-10-09 06:51:24 -05:00			`Disallow: <%= Discourse.base_path + "/" %>`
FEATURE: let Google index pages so it can remove them Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag 2020-05-10 21:14:21 -05:00