Revert "FEATURE: add Noindex to robots.txt for disallowed routes"

This reverts commit d84256a876.

This is not supported by Google and causes robots.txt to be flagged as
invalid

Removing Noindex

spec/requests/robots_txt_controller_spec.rb

@@ -52,7 +52,6 @@ RSpec.describe RobotsTxtController do
         Discourse.stubs(:base_uri).returns('/forum')
         get '/robots.txt'
         expect(response.body).to include("\nDisallow: /forum/admin")
-        expect(response.body).to include("\nNoindex: /forum/admin")
       end
     end