github-mirror/discourse
2
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2025-05-24 13:21:23 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

FIX: Apply crawler rate limits to cached requests (#27174)

Browse Source 
This commit moves the logic for crawler rate limits out of the application controller and into the request tracker middleware. The reason for this move is to apply rate limits to all crawler requests instead of just the requests that make it to the application controller. Some requests are served early from the middleware stack without reaching the Rails app for performance reasons (e.g. `AnonymousCache`) which results in crawlers getting 200 responses even though they've reached their limits and should be getting 429 responses.

Internal topic: t/128810.
This commit is contained in:
Osama Sayegh
2024-05-27 16:26:35 +03:00
committed by GitHub
parent 7992d7a65a
commit 361992bb74
4 changed files with 87 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
app/controllers/application_controller.rb
View File

@ -29,7 +29,6 @@ class ApplicationController < ActionController::Base
end
end
before_action :rate_limit_crawlers
before_action :check_readonly_mode
before_action :handle_theme
before_action :set_current_user_for_logs
@ -1053,28 +1052,6 @@ class ApplicationController < ActionController::Base
Theme.where(id: ids).pluck(:id, :name).to_h.to_json
end
def rate_limit_crawlers
return if current_user.present?
return if SiteSetting.slow_down_crawler_user_agents.blank?
user_agent = request.user_agent&.downcase
return if user_agent.blank?
SiteSetting
.slow_down_crawler_user_agents
.downcase
.split("|")
.each do |crawler|
if user_agent.include?(crawler)
key = "#{crawler}_crawler_rate_limit"
limiter =
RateLimiter.new(nil, key, 1, SiteSetting.slow_down_crawler_rate, error_code: key)
limiter.performed!
break
end
end
end
def run_second_factor!(action_class, action_data: nil, target_user: current_user)
if current_user && target_user != current_user
# Anon can run 2fa against another target, but logged-in users should not.