github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2025-05-22 03:51:07 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

FEATURE: flexible crawler detection

Browse Source 
You can use the crawler user agents site setting to amend what user agents
are considered crawlers based on a string match in the user agent

Also improves performance of crawler detection slightly
This commit is contained in:
Sam
2017-09-29 12:31:50 +10:00
parent 05dd97ffe0
commit f6fdc1ebe8
5 changed files with 30 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
spec/components/crawler_detection_spec.rb
View File

@ -3,6 +3,14 @@ require_dependency 'crawler_detection'
describe CrawlerDetection do
describe "crawler?" do
it "can be amended via site settings" do
SiteSetting.crawler_user_agents = 'Mooble|Kaboodle+*'
expect(CrawlerDetection.crawler?("Mozilla/5.0 (compatible; Kaboodle+*/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(CrawlerDetection.crawler?("Mozilla/5.0 (compatible; Mooble+*/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(CrawlerDetection.crawler?("Mozilla/5.0 (compatible; Gooble+*/2.1; +http://www.google.com/bot.html)")).to eq(false)
end
it "returns true for crawler user agents" do
# https://support.google.com/webmasters/answer/1061943?hl=en
expect(described_class.crawler?("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")).to eq(true)