IntenseWebs/discourse
3
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2025-02-25 18:55:32 -06:00
Code Issues Packages Projects Releases Wiki Activity

FEATURE: control which web crawlers can access using a whitelist or blacklist

Browse Source
This commit is contained in:
Neil Lalonde
2018-03-15 17:10:45 -04:00
parent cbbeedf53b
commit ced7e9a691
22 changed files with 722 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
spec/fabricators/web_crawler_request_fabricator.rb Normal file
View File

@@ -0,0 +1,5 @@
Fabricator(:web_crawler_request) do
user_agent { sequence(:ua) { |i| "Googlebot #{i}.0" } }
date Time.zone.now.to_date
count 0
end