FEATURE: allow search engines to index tag pages. (#12248)

Previously, we blocked search engines in tag pages since they may get marked as a duplicate content. * DEV: block tag inner pages from search engines crawling.
2025-05-29 00:20:54 +08:00 · 2021-03-09 23:55:57 +05:30
parent 7435d55ea6
commit e3d8e828b8
3 changed files with 2 additions and 3 deletions
--- a/app/controllers/robots_txt_controller.rb
+++ b/app/controllers/robots_txt_controller.rb
@ -23,7 +23,7 @@ class RobotsTxtController < ApplicationController
    /u
    /my
    /search
-    /tag
+    /tag/*/l
    /g
    /t/*/*.rss
    /c/*.rss