We were using `OFFSET`/`LIMIT` to query topics without an 'ORDER'. Without an explicit order, postgres makes no guarantees about which rows will be returned for each query. This commit adds `ORDER BY id ASC` so that our sitemaps behave consistently.
All our link validation, and conversion from url -> route/model/query is expensive and prone to bugs. Instead, if people enter a link, we can just use it as-is.
Originally all this extra logic was added to handle unusual situations like `/safe-mode`, `/my/...`, etc. However, all of these are now handled correctly by our Ember router, so there is no need for it.
Now, we just pass the user-supplied `href` directly to the SectionLink component, and let Ember handle routing to it when clicked.
The only functional change here is that we no longer validate internal links by parsing them with the Ember router. But I'd argue this is fine, because the previous logic would cause both false positives (e.g. `/t/123` would be valid, even if topic 123 doesn't exist), and false negatives (for routes which are server-side only, like the new AI share pages).
We were incorrectly using `return` in a block which was causing exceptions at runtime. These exceptions were not causing much issues as they are in defer block.
While working on writing a test for this specific case, I noticed that our `upsert_custom_fields` function was using rails `update_all` which is not updating the `updated_at` timestamp. This commit also fixes it and adds a test for it.
In #26122 we promoted all problem checks defined as class methods on AdminDashboardData to their own first-class ProblemCheck instances.
This PR continues that by promoting problem checks that are implemented as blocks as well. This includes updating a couple plugins that have problem checks.
This is a follow up to e2da72b76c7f49c2c3a159ce48051c4050f3dc3e.
Why this change?
According to https://web.dev/articles/preload-critical-assets,
> By preloading a certain resource, you are telling the browser that you would like to fetch it sooner than the browser would otherwise discover it because you are certain that it is important for the current page.
The preload resource hint is meant to tell the browser to fetch
resources that it would not discover upfront or early. However, we are
not using it the right way because we are literally adding the resource
hint right before a `<script>` tag which means the browser would have
discovered the resource even without the resource hint.
What does this change do?
This commit removes the preload resource hint which are added right
before script tags since the optimization here is highly questionable at the expense of making
our initial DOM larger.
We never use that information and this also fixes an issue with the BCC plugin which ends up triggering a rate-limit because we were publishing a "NEW_PRIVATE_MESSAGE" to the user sending the BCC for every recipients 💥
Internal - t/118283
This commit operates at three levels of abstraction:
1. We want to prevent user history rows from being unbounded in size.
This commit adds rails validations to limit the sizes of columns on
user_histories,
2. However, we don't want to prevent certain actions from being
completed if these columns are too long. In those cases, we truncate
the values that are given and store the truncated versions,
3. For endpoints that perform staff actions, we can further control
what is permitted by explicitly validating the params that are given
before attempting the action,
In AdminDashboardData we have a bunch of problem checks implemented as methods on that class. This PR absolves it of the responsibility by promoting each of those checks to a first class ProblemCheck. This way each of them can have their own priority and arbitrary functionality can be isolated in its own class.
Think "extract class" refactoring over and over. Since they were all moved we can also get rid of the @@problem_syms class variable which was basically the old version of the registry now replaced by ProblemCheck.realtime.
In addition AdminDashboardData::Problem value object has been entirely replaced with the new ProblemCheck::Problem (with compatible API).
Lastly, I added some RSpec matchers to simplify testing of problem checks and provide helpful error messages when assertions fail.
There are a couple of reasons for this.
The first one is practical, and related to eager loading. Since /lib is not eager loaded, when the application boots, ProblemCheck["identifier"] will be nil because the child classes aren't loaded.
The second one is more conceptual. There turns out to be a lot of inter-dependencies between the part of the problem check system that live in /app and the parts that live in /lib, which probably suggests it should all go in /app.
This change creates a user setting that they can toggle if
they don't want to receive unread notifications when someone closes a
topic they have read and are watching/tracking it.
Why this change?
There are two problematic queries in question here when loading
notifications in various tabs in the user menu:
```
SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338 AND (topics.id IS NULL OR topics.deleted_at IS NULL)
ORDER BY notifications.high_priority AND NOT notifications.read DESC,
NOT notifications.read AND notifications.notification_type NOT IN (5,19,25) DESC,
notifications.created_at DESC
LIMIT 30;
```
and
```
EXPLAIN ANALYZE SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338
AND (topics.id IS NULL OR topics.deleted_at IS NULL)
AND "notifications"."notification_type" IN (5, 19, 25)
ORDER BY notifications.high_priority AND NOT notifications.read DESC, NOT notifications.read DESC, notifications.created_at DESC LIMIT 30;
```
For a particular user, the queries takes about 40ms and 26ms
respectively on one of our production instance where the user has 10K notifications while the site has 600K notifications in total.
What does this change do?
1. Adds the `index_notifications_user_menu_ordering` index to the `notifications` table which is
indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read)
DESC, created_at DESC)`.
1. Adds a second index `index_notifications_user_menu_ordering_deprioritized_likes` to the `notifications`
table which is indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read AND notification_type NOT IN (5,19,25)) DESC, created_at DESC)`. Note that we have to hardcode the like typed notifications type here as it is being used in an ordering clause.
With the two indexes above, both queries complete in roughly 0.2ms. While I acknowledge that there will be some overhead in insert,update or delete operations. I believe this trade-off is worth it since viewing notifications in the user menu is something that is at the core of using a Discourse forum so we should optimise this experience as much as possible.
Why this change?
Prior to this change, the `CategoryList#find_relevant_topics` method was
loading and allocating all `CategoryFeaturedTopic` records in the
database to eventually only just use its `category_id` and `topic_id`
column. On a site with many `CategoryFeaturedTopic` records, the loading
of the ActiveRecord objects is a source of bottleneck.
The other problem with the `CategoryList#find_relevant_topics` method is
that it is unconditionally loading all records from the database even if
the user does not have access to the category. This again is wasteful.
What does this change do?
This commit makes it such that `CategoryList#find_relevant_topics` is
called only after `CategoryList#find_categories` in the `CategoryList#initialize`
method so that we can filter featured topics against categories that the
user has access to.
The second change is that Instead of loading `CategoryFeaturedTopic` records, we make an
inner join agains the `topics` table instead and skip any allocation of
`CatgoryFeaturedTopic` ActiveRecord objects.
As part of problem checks refactoring, we're moving some data to be DB backed. In this PR it's the tracking of problem check execution. When was it last run, when was the last problem, when should it run next, how many consecutive checks had problems, etc.
This allows us to implement the perform_every feature in scheduled problem checks for checks that don't need to be run every 10 minutes.
Users can hide their public profile and presence information by checking
“Hide my public profile and presence features” on the
`u/{username}/preferences/interface` page. In that case, we also don't
want to return user status from the server.
This work has been started in https://github.com/discourse/discourse/pull/23946.
The current PR fixes all the remaining places in Core.
Note that the actual fix is quite simple – a5802f484d.
But we had a fair amount of duplication in the code responsible for
the user status serialization, so I had to dry that up first. The refactoring
as well as adding some additional tests is the main part of this PR.
Now forums can enroll their sites to be showcased in the Discourse [Discover](https://discourse.org/discover) directory. Once they enable the site setting `include_in_discourse_discover` to enroll their forum the `CallDiscourseHub` job will ping the `api.discourse.org/api/discover/enroll` endpoint. Then the Discourse Hub will fetch the basic details from the forum and add it to the review queue. If the site is approved then the forum details will be displayed in the `/discover` page.
Also, remove experimental setting and simply use top_menu for feature detection
This means that when people eventually enable the hot top menu, there will
be topics in it
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Previously, problem checks were all added as either class methods or blocks in AdminDashboardData. Another set of class methods were used to add and run problem checks.
As of this PR, problem checks are promoted to first-class citizens. Each problem check receives their own class. This class of course contains the implementation for running the check, but also configuration items like retry strategies (for scheduled checks.)
In addition, the parent class ProblemCheck also serves as a registry for checks. For example we can get a list of all existing check classes through ProblemCheck.checks, or just the ones running on a schedule through ProblemCheck.scheduled.
After this refactor, the task of adding a new check is significantly simplified. You add a class that inherits ProblemCheck, you implement it, add a test, and you're good to go.
Why this change?
Firstly, note that this is not a security commit because this feature is
still in development and should not be used anywhere.
The reason we want to set a limit here is to greatly reduce the
possibility of a DoS attack in the future via `ThemeSetting` where
someone would set an arbituary large json string in
`ThemeSetting#json_value` and causing the server to run out of resources
trying to serialize/deserialize the value.
What does this change do?
Adds an ActiveRecord validation to ensure that the bytesize of the json
string being stored is smaller than or equal to 0.5mb. We believe 0.5mb
is a decent limit for now but we can review the limit in the future if
we believe it is too small.
Why this change?
The logic for validating a theme setting's value and default value was
not consistent as each part of the code would implement its own logic.
This is not ideal as the default value may be validated differently than
when we are setting a new value. Therefore, this commit seeks to
refactor all the validation logic for a theme setting's value into a
single service class.
What does this change do?
Introduce the `ThemeSettingsValidator` service class which holds all the
necessary helper methods required to validate a theme setting's value
When "lazy load categories" is enabled, only the categories present in
the sidebar are preloaded. This is insufficient because the parent
categories are necessary too for the sidebar to be rendered properly.
The strict-dynamic CSP directive is supported in all our target browsers, and makes for a much simpler configuration. Instead of allowlisting paths, we use a per-request nonce to authorize `<script>` tags, and then those scripts are allowed to load additional scripts (or add additional inline scripts) without restriction.
This becomes especially useful when admins want to add external scripts like Google Tag Manager, or advertising scripts, which then go on to load a ton of other scripts.
All script tags introduced via themes will automatically have the nonce attribute applied, so it should be zero-effort for theme developers. Plugins *may* need some changes if they are inserting their own script tags.
This commit introduces a strict-dynamic-based CSP behind an experimental `content_security_policy_strict_dynamic` site setting.
Reactions needs this to be able to filter out likes received
actions, where there is also an associated reaction, since
now most reactions also count as a like.
The `-ping` option significantly speeds up the ImageMagick `identify` command per our testing and the [documentation](https://imagemagick.org/script/command-line-options.php#ping):
> -ping
Efficiently determine these image characteristics: image number, the file name, the width and height of the image, whether the image is colormapped or not, the number of colors in the image, the number of bytes in the image, the format of the image (JPEG, PNM, etc.). Use +ping to ensure accurate image properties.
We already pass the `-ping` option in other places where the `identify` command is used, so it makes sense to use the option everywhere.
Internal topic: t/121431.
This would allow a theme component (or an API call) to reset the bump
date of a topic to a given post's created_at date.
I picked `post_id` as the parameter here because it provides a bit of
extra protection against accidentally resetting the bump date to a date
that doesn't make sense.
This change will allow auto generated video thumbnails to be used
instead of the black video thumbnail that overlays videos.
Follow up to: 2443446e62bff9dd8a0493d10e5359a5e946a84e
When we insert into the hot set we add things with a score of 0
This means that if hot has more than batch size items in it with a score, then the 0s don't get an initial score
This corrects the situation by always ensuring we re-score:
1. batch size high scoring topics
2. (new) batch size recently bumped topics
* Update spec/models/topic_hot_scores_spec.rb
Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>
---------
Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>
Removes duplication from LimitedEdit to see who can edit
posts, and also removes the old trust level setting check
since it's no longer necessary.
Also make it so staff can always edit since can_edit_post?
already has a staff escape hatch.
Why this change?
This commit introduces an experimental `type: objects` theme setting
which will allow theme developers to store a collection of objects as
JSON in the database. Currently, the feature is still in development and
this commit is simply setting up the ground work for us to introduce the
feature in smaller pieces.
What does this change do?
1. Adds a `json_value` column as `jsonb` data type to the `theme_settings` table.
2. Adds a `experimental_objects_type_for_theme_settings` site setting to
determine whether `ThemeSetting` records of with the `objects` data
type can be created.
3. Updates `ThemeSettingsManager` to support read/write access from the
`ThemeSettings#json_value` column.
Affects the following settings:
* whispers_allowed_groups
* anonymous_posting_allowed_groups
* personal_message_enabled_groups
* shared_drafts_allowed_groups
* here_mention_allowed_groups
* uploaded_avatars_allowed_groups
* ignore_allowed_groups
This turns off `client: true` for these group-based settings,
because there is no guarantee that the current user gets all
their group memberships serialized to the client. Better to check
server-side first.
