Files
discourse/app/controllers
Joshua Rosenfeld b52143feff FIX: Remove paths from robots.txt in favor of noindex header
Google no longer supports the use of robots.txt to block indexing.
See https://support.google.com/webmasters/answer/6062608 and
https://support.google.com/webmasters/answer/93710

Previous commits have added the `noindex` header to appropriate pages,
now we need to remove the paths from robots.txt so the pages can be
crawled.

Follow up to:
13f229808a
b6765aac4b
676be3a853
07b728c5e5
c94e6a9a66
2020-06-25 13:55:06 -04:00
..
2020-06-08 10:26:29 +02:00
2019-10-21 12:08:20 +01:00