IntenseWebs/discourse
3
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2025-02-25 18:55:32 -06:00
Code Issues Packages Projects Releases Wiki Activity

FEATURE: Rate limit common AI bots crawlers by default (#26731)

Browse Source 
This commit adds the most common AI bot crawlers seen
on our hosting (claudebot, gptbot, anthropic-ai, brightbot)
to our `slow_down_crawler_user_agents` and `crawler_user_agents`
site settings by default.

This means these AI bots will be rate limited by default instead
of site admins having to remember to do it for themselves.
This commit is contained in:
Martin Brennan
2024-04-24 12:01:42 +10:00
committed by GitHub
parent d2a52c3c35
commit c791346088
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
config/site_settings.yml
View File

@@ -1958,7 +1958,7 @@ security:
list_type: compact
crawler_user_agents:
hidden: true
default: "rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor|lighthouse|google-inspectiontool"
default: "rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor|lighthouse|google-inspectiontool|gptbot|claudebot|anthropic-ai|brightbot"
type: list
list_type: compact
browser_update_user_agents:
@@ -2006,7 +2006,7 @@ security:
list_type: compact
slow_down_crawler_user_agents:
type: list
default: ""
default: "gptbot|claudebot|anthropic-ai|brightbot"
list_type: compact
slow_down_crawler_rate: 60
content_security_policy: