IntenseWebs/discourse
3
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-26 02:40:53 -06:00
Code Issues Packages Projects Releases Wiki Activity

FEATURE: Disallow tags and categories rss feeds

Browse Source 
This stops crawlers from hitting tags and category rss feeds to discover
new content, instead they should focus on latest/posts if they need to
consume something regular
This commit is contained in:
Sam 2018-04-11 14:35:54 +10:00
parent 284c946128
commit 489c22d93c
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
app/views/robots_txt/index.erb
View File

@ -30,6 +30,8 @@ Disallow: /*?*api_key*
Disallow: /groups
Disallow: /groups/
Disallow: /t/*/*.rss
Disallow: /tags/*.rss
Disallow: /c/*.rss
<% if @disallowed_user_agents %>
<% @disallowed_user_agents.each do |user_agent| %>