DEV: Add plugin API to add to robots.txt (#17378)

This plugin API can be used to add to robots.txt. The event handler
receives the complete robots information before it is converted into
robots.txt.

app/controllers/robots_txt_controller.rb

@@ -84,6 +84,8 @@ class RobotsTxtController < ApplicationController
       result[:agents] << { name: 'Googlebot', disallow: deny_paths_googlebot }
     end
 
+    DiscourseEvent.trigger(:robots_info, result)
+
     result
   end
 end