IntenseWebs/discourse
3
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2025-02-16 18:24:52 -06:00
Code Issues Packages Projects Releases Wiki Activity

FIX: Change crawl size to 10k. Youtube for example doesn't work with the

Browse Source 
first 1k
This commit is contained in:
Robin Ward 2014-04-07 16:03:47 -04:00
parent b9799a886d
commit 1e3faddfe4
1 changed files with 14 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
app/jobs/regular/crawl_topic_link.rb
View File

@ -58,7 +58,7 @@ module Jobs
# Using exceptions for flow control is really bad, but there really seems to
# be no sane way to get a stream to stop reading in Excon (or Net::HTTP for
# that matter!)
raise ReadEnough.new if result.size > 1024
raise ReadEnough.new if result.size > 1024 * 10
end
Excon.get(uri.to_s, response_block: streamer, read_timeout: 20, headers: CrawlTopicLink.request_headers(uri))
result