Sam d84256a876 FEATURE: add Noindex to robots.txt for disallowed routes ...

This strips pages out of indexes that should not exist see:

https://meta.discourse.org/t/pages-listed-in-the-robots-txt-are-crawled-and-indexed-by-google/100309/11?u=sam

2018-11-02 16:39:47 +11:00

..

index.erb

FEATURE: add Noindex to robots.txt for disallowed routes

2018-11-02 16:39:47 +11:00

no_index.erb

prefix the robots.txt rules with the directory when using subfolder

2018-04-11 22:05:02 +02:00