Commit Graph

7467 Commits

Author SHA1 Message Date
053cbe3112 PERF: Limit characters used to generate headline for search blurb.
We determined using the following benchmark script that limiting to 2500 chars would mean a maximum of
25ms spent generating headlines.

```
require 'benchmark/ips'

string = <<~STRING
Far far away, behind the word mountains...
STRING

def sql_excerpt(string, l = 1000000)
  DB.query_single(<<~SQL)
  SELECT TS_HEADLINE('english', left('#{string}', #{l}), PLAINTO_TSQUERY('mountains'))
  SQL
end

def ruby_excerpt(string)
  output = DB.query_single("SELECT '#{string}'")[0]
  Search::GroupedSearchResults::TextHelper.excerpt(output, 'mountains', radius: 100)
end

puts "Ruby Excerpt: #{ruby_excerpt(string)}"
puts "SQL Excerpt: #{sql_excerpt(string)}"
puts

Benchmark.ips do |x|
  x.time = 10

  [1000, 2500, 5000, 10000, 20000, 50000].each do |l|
    short_string = string[0..l]

    x.report("ts_headline excerpt #{l}") do
      sql_excerpt(short_string, l)
    end

    x.report("actionview excerpt #{l}") do
      ruby_excerpt(short_string)
    end
  end

  x.compare!
end
```

```
actionview excerpt 1000:    20570.7 i/s
actionview excerpt 2500:    17863.1 i/s - 1.15x  (± 0.00) slower
actionview excerpt 5000:    14228.9 i/s - 1.45x  (± 0.00) slower
actionview excerpt 10000:    10906.2 i/s - 1.89x  (± 0.00) slower
actionview excerpt 20000:     6255.0 i/s - 3.29x  (± 0.00) slower
ts_headline excerpt 1000:     4337.5 i/s - 4.74x  (± 0.00) slower
actionview excerpt 50000:     3222.7 i/s - 6.38x  (± 0.00) slower
ts_headline excerpt 2500:     2240.4 i/s - 9.18x  (± 0.00) slower
ts_headline excerpt 5000:     1258.7 i/s - 16.34x  (± 0.00) slower
ts_headline excerpt 10000:      667.2 i/s - 30.83x  (± 0.00) slower
ts_headline excerpt 20000:      348.7 i/s - 58.98x  (± 0.00) slower
ts_headline excerpt 50000:      131.9 i/s - 155.91x  (± 0.00) slower
```
2020-08-07 14:36:52 +08:00
e60c74d3c1 FEATURE: Use PG ts_headline for highlighting topic title in search. 2020-08-07 12:43:09 +08:00
12a00d6dc5 FEATURE: add advanced order to search (#10385)
Similar to `advanced_filter` I introduced `advanced_order`.

I needed a new option because default orders are evaluated after advanced_filter so I couldn't use it.

Also, that part is a little bit more generic
```
elsif word =~ /order:\w+/
  @order = word.gsub('order:', '').to_sym
nil
```

After those changes, I can use them in plugins in this way:
```
Search.advanced_order(:votes) do |posts|
  posts.reorder("COALESCE((SELECT dvvc.counter FROM discourse_voting_vote_counters dvvc WHERE dvvc.topic_id = subquery.topic_id), 0) DESC")
end
```
2020-08-07 12:47:00 +10:00
ab2b6f8dea FIX: Specify config when generating tsquery using ts_headline. 2020-08-07 10:21:14 +08:00
7df57b35da REFACTOR: Remove Discourse.__widget_helpers
It's now a variable in the context where the templates are created.
2020-08-06 14:35:46 -04:00
cd4f251891 FEATURE: Poll breakdown 2.0 (#10345)
The poll breakdown modal replaces the grouped pie charts feature.

Includes:

* MODAL: Untangle `onSelectPanel`
Previously modal-tab component would call on click the onSelectPanel callback with itself (modal-tab) as `this` which severely limited its usefulness. Now showModal binds the callback to its controller.

"The PR includes a fix/change to d-modal (b7f6ec6) that hasn't been extracted to a separate PR because it's not currently possible to test a change like this in abstract, i.e. with dynamically created controllers/components in tests. The percentage/count toggle test for the poll breakdown feature is essentially a test for that d-modal modification."
2020-08-06 17:57:06 +02:00
76c02cac65 DEV: Ensure the correct plugin stylesheets are reloaded in development
If a plugin name contained the name of another plugin, the wrong stylesheets would be reloaded. For example, working on discourse-prometheus-alert-receiver would cause discourse-prometheus stylesheets to be reloaded.
2020-08-06 16:10:57 +01:00
87e2c9de24 DEV: Plugins can extend color definitions (#10383) 2020-08-06 09:46:17 -04:00
6fdc711b4a FEATURE: Allow users to opt out of automatic dark mode (#10377) 2020-08-06 09:45:37 -04:00
8c03868808 DEV: Correctly render data- attributes in widget hbs templates (#10376)
In virtualdom, element 'properties' are not completely synonymous with element 'attributes'. In particular, `data-` properties will not be rendered as attributes. To ensure all attributes are passed through, we need to include them under an `attributes` key. For more info, see https://github.com/Matt-Esch/virtual-dom/blob/master/docs/vnode.md#custom-attributes-data-
2020-08-06 14:33:09 +01:00
2193d02433 PERF: Use PG headlines for blurb generation and highlighting for search. 2020-08-06 14:56:29 +08:00
ba482c251c FIX: stop sync of tracking state when list is filtered
This stops sync of tracking state when list is filtered, in the past this
would cause the tracking state to go off wack.

Additionally this introduces an alias for "filter=tracking", called "f=tracking"

This was done cause the term "filter" is used internally in 2 different ways
the main way is for /unread /new filtering.

Trying to also call a query param "filter" causes enormous amounts of
internal pain, this circumvents the issue.
2020-08-06 16:34:02 +10:00
255b0e9f14 PERF: Replace video and audio links in search blurb while indexing.
In the near future, we will be swtiching to PG headlines to generate the
search blurb. As such, we need to replace audio and video links in the
raw data used for headline generation. This also means that we avoid
replacing links each time we need to generate the blurb.
2020-08-06 12:25:03 +08:00
06ef87da51 DEV: Make rubocop happy. 2020-08-06 10:11:07 +08:00
ee5d8fba0c PERF: Optimize ActionView::Helpers::TextHelper#excerpt. 2020-08-06 09:59:20 +08:00
67e8bc5342 FEATURE - allow category group moderators to split/merge topics (#10351) 2020-08-05 10:33:25 -04:00
cb12a721c4 REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
aee3c2c34d FIX: attempt to output a useful error message
currently AWS problems will just dump a stack trace
2020-08-05 17:49:42 +08:00
850d5696f2 FIX: Disable concurrent migration for multisite:migrate.
Rails migration is not thread safe.
2020-08-05 15:12:56 +08:00
1d281353b6 FIX: Don't run seeds if multisite migration fails. 2020-08-05 15:12:56 +08:00
e8328968bb DEV: allow qunit:test to run concurrently
Previously we were fighting on pid file which makes it hard to run multiple
2020-08-05 16:57:12 +10:00
13feb300a8 FIX: Topic map was incorrectly counting assign actions (#10360)
The assign plugin is one of two situations where a post can be both a whisper and a small-action. Check the action_code field to filter out small-actions.
2020-08-05 11:51:28 +10:00
d0953dcda9 DEV: Fix watching plugin stylesheet changes locally
I broke this in e6dbb4fcf52fa74853ed45616dd7cbf620c7deca
2020-08-04 14:10:25 -04:00
266c0c50d7 FIX: Load base color scheme when default theme is not set
Followup to c937afc
2020-08-04 12:15:07 -04:00
ef92b72ff6 DEV: ensure 'posts:reorder_posts' rake task work on single topic
when the topic_id argument is provided.
2020-08-04 14:57:49 +02:00
8a0478b97d DEV: adds plugin api to add custom recipients of a post revision (#10367)
* DEV: adds plugin api to add custom recipients of a post revision

Usage:

```
add_post_revision_notifier_recipients do |post_revision|
  [78]
end
```
2020-08-04 11:57:33 +02:00
712ab33ff8 DEV: fix 'posts:reorder' rake task
Running the reorder rake task was triggering the following error:

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "post_timings_unique"

I re-worked the queries and refactored to use the same couple of queries for all similar tables/columns.
2020-08-04 11:07:07 +02:00
4c03a944f6 PERF: Move URI regexp in GroupSearchResults.blurb_for into constant
No need to generate the huge regexp over and over again.
2020-08-04 14:38:43 +08:00
3b08b15855 PERF: Remove one extra call to Redis when searching. 2020-08-04 14:02:02 +08:00
5a3494b1e1 FIX: IMAP archive fix and group list mailbox code unification (#10355)
* Fixed an issue I introduced in the last PR where I am just archiving everything regardless of whether it is actually archived in Discourse man_facepalming
* Refactor group list_mailboxes IMAP code to use providers, add specs, and add provider code to get the correct prodivder
2020-08-04 14:19:57 +10:00
c937afc75e FEATURE: automatic dark mode (#10341)
A first step to adding automatic dark mode color scheme switching. Adds a new SCSS file at `color_definitions.scss` that serves to output all SCSS color variables as CSS custom properties. And replaces all SCSS color variables with the new CSS custom properties throughout the stylesheets. 

This is an alpha feature at this point, can only be enabled via console using the `default_dark_mode_color_scheme_id` site setting.
2020-08-03 22:57:10 -04:00
92b2448500 DEV: Fix i18n deprecation warnings 2020-08-03 15:57:29 +02:00
fe83baa9b3 FIX: Exclude DELETE methods from invalid request with payload.
Follow-up 105d560177e166a0491b3f0be04dd068178fb3d5

Our client side code is sending params as part of the request payload so
that is going to be tricky to fix.
2020-08-03 17:05:11 +08:00
105d560177 SECURITY: 413 for GET, HEAD or DELETE requests with payload. 2020-08-03 14:21:33 +08:00
0d8c49c385 DEV: Remove unnecessary rack patches.
PRs have all been merged.
2020-08-03 14:17:10 +08:00
ed68ac054d DEV: Cover other case for generated message ID in Email::Receiver 2020-08-03 13:37:06 +10:00
7f55ed1a4a DEV: Clarify Email::Receiver existing incoming email behaviour 2020-08-03 13:31:34 +10:00
2920988b3a FIX: IMAP sync email update uniqueness across groups and minor improvements (#10332)
Adds a imap_group_id column to IncomingEmail to deal with an issue where we were trying to update emails in the mailbox, calling IncomingEmail.where(imap_sync: true). However UID and UIDVALIDITY could be the same across accounts. So if group A used IMAP details for Gmail account A, and group B used IMAP details for Gmail account B, and both tried to sync changes to an email with UID of 3 (e.g. changing Labels), one account could affect the other. This even applied to Archiving!

Also in this PR:

* Fix error occurring if we do a uid_fetch and no emails are returned
* Allow for creating labels within the target mailbox (previously we would not do this, only use existing labels)
* Improve consistency for log messages
* Add specs for generic IMAP provider (Gmail specs still to come)
* Add custom archiving support for Gmail
* Only use Message-ID for uniqueness of IncomingEmail if it was generated by us
* Various refactors and improvements
2020-08-03 13:10:17 +10:00
e6dbb4fcf5 DEV: Live refresh all themes when watching stylesheets (#10337) 2020-07-30 19:03:24 -04:00
16c65a94f7 PERF: Preload S3 inventory data for multisite clusters 2020-07-29 10:31:55 +01:00
36c11cb62b FIX: Rename delete_when_reminder_sent? bookmark method to avoid conflict with AR (#10333)
I added delete_when_reminder_sent to ignored_columns because it no longer exists and added a shortcut method delete_when_reminder_sent? to the Bookmark model. However I have been seeing some weird errors like:

> Job exception: unknown attribute 'delete_when_reminder_sent' for Bookmark.

So I am very suspicious. I am just renaming the method to auto_delete_when_reminder_sent? to avoid any potential conflicts.

Also found include_bookmark_delete_on_owner_reply? in PostSerializer which is used for nothing; I must have forgotten to delete it before.
2020-07-29 17:02:36 +10:00
c7f1777ac9 DEV: No need to pause Sidekiq during backups.
This was done for historical reasons where we stop all writes going to
the DB during a backup by setting the database to readonly mode.
2020-07-29 10:52:31 +08:00
9e5b213089 FIX: Ensure topic user bookmarked synced on bookmark auto-delete (#10323)
For the following conditions, the TopicUser.bookmarked column was not updated correctly:

* When a bookmark was auto-deleted because the reminder was sent
* When a bookmark was auto-deleted because the owner of the bookmark replied to the topic

This adds another migration to fix the out-of-sync column and also some refactors to BookmarkManager to allow for more of these delete cases. BookmarkManager is used instead of directly destroying the bookmark in PostCreator and BookmarkReminderNotificationHandler.
2020-07-29 09:43:32 +10:00
8f140b8903 TopicView/PostSerializer should be able to handle topics without categories 2020-07-28 19:06:55 -04:00
17ab69929c Handle topics without categories 2020-07-28 17:55:24 -04:00
74ab4f3bff FEATURE - group modetators visual indicator (#10310) 2020-07-28 17:15:04 -04:00
597d542c33 FIX: Improve Topic.similar_to with better Topic#title matches.
This changes PG text search to only match the given title against
lexemes that are formed from the title. Likewise, the given raw will
only be matched against lexemes that are formed from the post's raw.
2020-07-28 12:00:27 +08:00
2e5b2d20ba FIX: Resolve issue where deleted spam topics marked as Not Spam were not being recovered (#10322)
If a user posted a topic and Akismet decided it was spam, the topic gets deleted and put into the review queue. If a category moderator for that category marked the post/topic as "Not Spam" the topic did not get recovered correctly because Guardian.new(@user).can_review_topic?(@post.topic) returned false incorrectly because the topic was deleted.
2020-07-28 12:06:15 +10:00
25f6136b27 Upgrade fastimage and remove our freedom patch 2020-07-27 13:23:17 -04:00
e0d9232259 FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00