discourse/app/views/robots_txt/index.erb

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
User-agent: *
Disallow: /auth/cas
Disallow: /auth/facebook/callback
Disallow: /auth/twitter/callback
Disallow: /auth/google/callback
Disallow: /auth/yahoo/callback
Disallow: /auth/github/callback
Disallow: /auth/cas/callback
Disallow: /assets/browser-update*.js
Disallow: /users/
Disallow: /u/
Disallow: /badges/
Disallow: /search
Disallow: /search/
Disallow: /tags
Disallow: /tags/
Disallow: /email/
Disallow: /session
Disallow: /session/
Disallow: /admin
Disallow: /admin/
Disallow: /user-api-key
Disallow: /user-api-key/
Disallow: /*?api_key*
Disallow: /*?*api_key*
Disallow: /groups
Disallow: /groups/
Disallow: /uploads/

<%= server_plugin_outlet "robots_txt_index" %>
Initial release of Discourse 2013-02-05 13:16:51 -06:00			`# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file`
			`#`
Disallow auth callbacks in robots.txt 2014-01-14 09:42:22 -06:00			`User-agent: *`
			`Disallow: /auth/cas`
			`Disallow: /auth/facebook/callback`
			`Disallow: /auth/twitter/callback`
			`Disallow: /auth/google/callback`
			`Disallow: /auth/yahoo/callback`
			`Disallow: /auth/github/callback`
FEATURE REMOVAL: persona login see: https://meta.discourse.org/t/pulling-persona-out-of-discourse-core/12613 2014-02-10 23:56:48 -06:00			`Disallow: /auth/cas/callback`
Block browser-update.js in robots.txt. Move noscript block above everything else in application layout. 2014-02-14 14:32:49 -06:00			`Disallow: /assets/browser-update*.js`
Disallow /users/ in robots.txt 2014-05-23 09:28:26 -05:00			`Disallow: /users/`
Convert server side paths to use `/u/` 2017-03-28 13:27:54 -05:00			`Disallow: /u/`
Add /badges to robots.txt for now, we don't have a crawlable view so it's better to exclude it. 2014-10-30 13:32:42 -05:00			`Disallow: /badges/`
better disallow search ... this could get ugly 2015-04-02 01:08:00 -05:00			`Disallow: /search`
			`Disallow: /search/`
FIX: add /tags routes to robots.txt 2017-02-03 10:56:57 -06:00			`Disallow: /tags`
			`Disallow: /tags/`
SECURITY: Prevent robots from indexing more routes These routes could contain sensitive material and should never be indexed for content. 2018-02-04 12:24:36 -06:00			`Disallow: /email/`
			`Disallow: /session`
			`Disallow: /session/`
			`Disallow: /admin`
			`Disallow: /admin/`
			`Disallow: /user-api-key`
			`Disallow: /user-api-key/`
			`Disallow: /?api_key`
			`Disallow: /?api_key*`
FEATURE: disallow groups from being indexed 2018-03-01 20:38:01 -06:00			`Disallow: /groups`
			`Disallow: /groups/`
Prevent robots from indexing uploads Although most user uploads are probably harmless, it's possible someone has (either maliciously or not) uploaded sensitive information. Prevent robots from indexing the uploads route. 2018-03-09 05:49:46 -06:00			`Disallow: /uploads/`
new: server plugin outlet for indexable robots.txt 2017-02-13 06:01:10 -06:00
			`<%= server_plugin_outlet "robots_txt_index" %>`