Commit Graph

8100 Commits

Author SHA1 Message Date
4670b62969 DEV: IMAP log to database (#10435)
Convert all IMAP logging to write to a database table for easier inspection. These logs are cleaned up daily if they are > 5 days old.

Logs can easily be watched in dev by setting DISCOURSE_DEV_LOG_LEVEL=\"debug\" and running tail -f development.log | grep IMAP
2020-08-14 12:01:31 +10:00
c2f3c0dc44 FIX: generate_topic_thumbnails job infinitely running for corrupted images
It's possible that the original topic image is broken in some form, so
we shouldn't try and generate a topic thumbnail for it. The fix will
prevent the generate_topic_thumbnails job being enqueued every time the
topic is viewed.
2020-08-13 17:08:32 -06:00
d65a839577 FEATURE: allow group membership to unmute categories and tags
For sites that are configured to mute some or all categories and tags
for users by default, groups can now be configured to set members'
notification level to normal from the group manage UI.
2020-08-13 17:20:53 -04:00
7194b31443 FEATURE: don't notify about changed tags for a private message (#10408)
* FEATURE: don't notify about changed tags for a private message

Only staff members observing specific tag should receive a notification

* FIX: remove other category which is not used

* FIX: improved specs to ensure that revise was succesful
2020-08-13 17:22:34 +10:00
310952fd6a FIX: generate topic excerpt when moving posts to new topic.
Currently, it's not generating the excerpt by default. We have to trigger the "Rebuild HTML" action to do it.
2020-08-13 11:30:14 +05:30
ffb31b8d2b FIX: Do not require tagging to be enabled for IMAP archive and delete (#10426)
Previously we did an early return if either SiteSetting.tagging_enabled or SiteSetting.allow_staff_to_tag_pms was false when updating the email on the IMAP server -- however this also stopped us from archiving or deleting emails if either of these were disabled.
2020-08-13 14:04:40 +10:00
bd0a7553c4 DEV: Detect when s3 inventory failure is caused by etag difference (#10427) 2020-08-13 09:30:28 +10:00
ced8cdad24 DEV: Order links by domain and url in spec 2020-08-12 17:46:12 -04:00
c68563a281 DEV: Improve API usage when creating * updating categories
The category model already has a default value for `color` and
`text_color` so they don't need to be required via the API. The ember UI
already requires that colors be selected.

The name of the category also doesn't need to be required when updating
the category either because we are already passing in the id for the
category we want to change.

These changes improve the api experience because you no longer have to
lookup the category name, color, or text color before updating a single
category attribute. When creating a category the name is still required.

https://meta.discourse.org/t/-/132424/2
2020-08-12 12:28:29 -06:00
6dd9f2eca2 FIX: color scheme selection with non-default theme
This fixes an issue where a non-default theme set to use the base color
scheme (i.e. the theme had an empty `color_scheme_id`) was loading the
default theme's color scheme instead.
2020-08-12 08:49:13 -04:00
603a4d1794 DEV: Fix randomly failing spec. 2020-08-12 16:34:56 +08:00
6792d7f50c Fix the build.
Follow-up to 93f8396b4b7cf23399c6b62e7feb51c91bf9814d.
2020-08-12 15:55:22 +08:00
93f8396b4b FIX: Limit PG headline based search blurb generation to 200 characters.
* Recovers omission characters '...' in blurb as well.
2020-08-12 15:34:27 +08:00
1f7f8ee72c FIX: try to make topic_tracking_state_spec stable (#10419)
Sometime parallel spec if failing with error:
```
NoMethodError:
       undefined method `data' for nil:NilClass
     # ./spec/models/topic_tracking_state_spec.rb:339:in `block (4 levels) in <main>'
```

I have a theory that it might be related to instance variables in before block
2020-08-12 15:56:06 +10:00
95b71b35d6 FEATURE: IMAP delete email sync for group inboxes (#10392)
Adds functionality to reflect topic delete in Discourse to IMAP inbox (Gmail only for now) and reflecting Gmail deletes in Discourse.

Adding lots of tests, various refactors and code improvements.

When Discourse topic is destroyed in PostDestroyer mark the topic incoming email as imap_sync: true, and do the opposite when post is recovered.
2020-08-12 10:16:26 +10:00
c05aced094 FIX: Invalidate cache when updating color scheme colors (#10417) 2020-08-11 16:28:59 -04:00
ee366f7ac7 FIX: Ensure load more directory items has a .json url
The `/directory_items` route needs to have a .json url, but the rails
url helper `_path` doesn't return the format of the route.

I tried passing in a format options to `directory_items_path`. Which
works in the rails console

```
[8] pry(main)> directory_items_path(params.merge(:format => :json))
=> "/directory_items.json?page=1"
```

but when I added that some logic to the controller it comes out as

```
/directory_items?format=json&page=1
```

(which is actually how I expect it to work based on how you pass in the
format param). Anyways, because I couldn't figure out how to pass a
format to the `_path` helper I just used URI.parse to append `.json`
manually.
2020-08-11 12:43:55 -06:00
e64e6351bd DEV: Assert for response status in tests. 2020-08-11 13:53:37 +08:00
b950b3fb3f DEV: Add verified to uploads and fill in S3 inventory (#10406)
When we run the S3 inventory, mark uploads that exist as verified true, those that don't as verified false, and uploads not included in the check / not yet checked as verified nil.
2020-08-11 14:43:51 +10:00
2032c11f78 FIX: Return 422 when creating topics with tags w/out permission (#10400)
The UI prevents users from trying to create tags on topics when they
don't have permission, but if you are trying to add tags to a topic via
the API and you don't have permission before this change it would
silently succeed in creating the topic, but it wouldn't have any tags.

Now a 422 error will be returned with an error message when trying to
create a topic with tags when tagging is disabled or you don't have
enough trust level to add tags to a topic.

Bug report: https://meta.discourse.org/t/-/70525/14
2020-08-10 16:14:15 -06:00
d67f7a7984 Allows mod posts to be created for category group moderators on closed/archived topics (#10399) 2020-08-10 15:21:01 -04:00
4476680d4b UX: Set silence_reason using the system locale
Previously we would use the user's locale, which can be confusing for moderators.
2020-08-10 18:51:54 +01:00
fe7a7ecf6c FIX: Include secure media URLs when linking post uploads (#10404)
Normally, secure media urls are linked like `/secure-media-uploads/...`. In this case, uploads were already being linked correctly.

But sometimes (e.g. when pulling hotlinked onebox images) secure media is referenced with a full domain name (`//example.com/secure-media-uploads`). This commit ensures that those uploads are also linked correctly.
2020-08-10 17:59:29 +01:00
b7a092bd28 FEATURE: Group category permissions tab (#10388) 2020-08-10 09:49:05 -05:00
b44748e503 DEV: Order links by domain in spec
Should fix spec failing in macOS pgsql 11.3
2020-08-10 10:25:13 -04:00
33982c0c9e DEV: Document notifications and tags api endpoints
Added some more specs that will be used to auto generate the api docs.
2020-08-07 14:30:54 -06:00
32f6c5c07e DEV: Use explicit ordering so spec is realiable 2020-08-07 15:15:39 -03:00
f179510a68 FIX: include both name and id in color scheme stylesheet filename slugs (#10397) 2020-08-07 13:43:45 -04:00
bc63232d2e FIX: sync reviewable count when opening the hamburger menu (#10368)
When a tab is open but left unattended for a while, the red, green, and blue
pills tend to go out of sync.

So whevener we open the notifications menu, we sync up the notification count
(eg. blue and green pills) with the server.

However, the reviewable count (eg. the red pill) is not a notification and
is located in the hamburger menu. This commit adds a new route on the server
side to retrieve the reviewable count for the current user and a ping
(refreshReviewableCount) from the client side to sync the reviewable count
whenever they open the hamburger menu.

REFACTOR: I also refactored the hamburger-menu widget code to prevent repetitive uses
of "this.".

PERF: I improved the performance of the 'notify_reviewable' job by doing only 1 query
to the database to retrieve all the pending reviewables and then tallying based on the
various rights.
2020-08-07 18:13:02 +02:00
3593e582a3 FIX - limit number of embedded media items in a post (#10391)
* FIX - limit number of embedded media items in a post

* Add renamed settings to DeprecatedSettings
2020-08-07 12:08:59 -04:00
86cb5803ba Convert HEIC uploads to JPG by default (#10395) 2020-08-07 11:17:50 -04:00
9c9aa21726 Add site setting to pick dark mode color scheme (#10390)
Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-08-07 08:52:47 -04:00
053cbe3112 PERF: Limit characters used to generate headline for search blurb.
We determined using the following benchmark script that limiting to 2500 chars would mean a maximum of
25ms spent generating headlines.

```
require 'benchmark/ips'

string = <<~STRING
Far far away, behind the word mountains...
STRING

def sql_excerpt(string, l = 1000000)
  DB.query_single(<<~SQL)
  SELECT TS_HEADLINE('english', left('#{string}', #{l}), PLAINTO_TSQUERY('mountains'))
  SQL
end

def ruby_excerpt(string)
  output = DB.query_single("SELECT '#{string}'")[0]
  Search::GroupedSearchResults::TextHelper.excerpt(output, 'mountains', radius: 100)
end

puts "Ruby Excerpt: #{ruby_excerpt(string)}"
puts "SQL Excerpt: #{sql_excerpt(string)}"
puts

Benchmark.ips do |x|
  x.time = 10

  [1000, 2500, 5000, 10000, 20000, 50000].each do |l|
    short_string = string[0..l]

    x.report("ts_headline excerpt #{l}") do
      sql_excerpt(short_string, l)
    end

    x.report("actionview excerpt #{l}") do
      ruby_excerpt(short_string)
    end
  end

  x.compare!
end
```

```
actionview excerpt 1000:    20570.7 i/s
actionview excerpt 2500:    17863.1 i/s - 1.15x  (± 0.00) slower
actionview excerpt 5000:    14228.9 i/s - 1.45x  (± 0.00) slower
actionview excerpt 10000:    10906.2 i/s - 1.89x  (± 0.00) slower
actionview excerpt 20000:     6255.0 i/s - 3.29x  (± 0.00) slower
ts_headline excerpt 1000:     4337.5 i/s - 4.74x  (± 0.00) slower
actionview excerpt 50000:     3222.7 i/s - 6.38x  (± 0.00) slower
ts_headline excerpt 2500:     2240.4 i/s - 9.18x  (± 0.00) slower
ts_headline excerpt 5000:     1258.7 i/s - 16.34x  (± 0.00) slower
ts_headline excerpt 10000:      667.2 i/s - 30.83x  (± 0.00) slower
ts_headline excerpt 20000:      348.7 i/s - 58.98x  (± 0.00) slower
ts_headline excerpt 50000:      131.9 i/s - 155.91x  (± 0.00) slower
```
2020-08-07 14:36:52 +08:00
e60c74d3c1 FEATURE: Use PG ts_headline for highlighting topic title in search. 2020-08-07 12:43:09 +08:00
12a00d6dc5 FEATURE: add advanced order to search (#10385)
Similar to `advanced_filter` I introduced `advanced_order`.

I needed a new option because default orders are evaluated after advanced_filter so I couldn't use it.

Also, that part is a little bit more generic
```
elsif word =~ /order:\w+/
  @order = word.gsub('order:', '').to_sym
nil
```

After those changes, I can use them in plugins in this way:
```
Search.advanced_order(:votes) do |posts|
  posts.reorder("COALESCE((SELECT dvvc.counter FROM discourse_voting_vote_counters dvvc WHERE dvvc.topic_id = subquery.topic_id), 0) DESC")
end
```
2020-08-07 12:47:00 +10:00
1ca81fbb95 FEATURE: set notification levels when added to a group (#10378)
* FEATURE: set notification levels when added to a group

This feature allows admins and group owners to define default
category and tag tracking levels that will be applied to user
preferences automatically at the time when users are added to the
group. Users are free to change those preferences afterwards.
When removed from a group, the user's notification preferences aren't
changed.
2020-08-06 12:27:27 -04:00
87e2c9de24 DEV: Plugins can extend color definitions (#10383) 2020-08-06 09:46:17 -04:00
6fdc711b4a FEATURE: Allow users to opt out of automatic dark mode (#10377) 2020-08-06 09:45:37 -04:00
df39e372d7 DEV: Add spec for removing and re-adding hotlinked images
Before the recent refactor, this would fail. https://meta.discourse.org/t/154184
2020-08-06 10:01:53 +01:00
2193d02433 PERF: Use PG headlines for blurb generation and highlighting for search. 2020-08-06 14:56:29 +08:00
255b0e9f14 PERF: Replace video and audio links in search blurb while indexing.
In the near future, we will be swtiching to PG headlines to generate the
search blurb. As such, we need to replace audio and video links in the
raw data used for headline generation. This also means that we avoid
replacing links each time we need to generate the blurb.
2020-08-06 12:25:03 +08:00
ceb858c70a PERF: Release post_upload records when downloaded image is removed (#10379)
Previously we would unconditionally keep all images downloaded via pull_hotlinked_images, even if they are later removed from the post. This commit removes that logic, and relies on the existing link_post_uploads process to pick up the downloaded images in `cooked`. Specs are added to ensure this is working correctly for regular hotlinked images, and for oneboxes.
2020-08-06 10:06:34 +10:00
67e8bc5342 FEATURE - allow category group moderators to split/merge topics (#10351) 2020-08-05 10:33:25 -04:00
cb12a721c4 REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
13feb300a8 FIX: Topic map was incorrectly counting assign actions (#10360)
The assign plugin is one of two situations where a post can be both a whisper and a small-action. Check the action_code field to filter out small-actions.
2020-08-05 11:51:28 +10:00
6da7a97eee FIX: Exclude shared drafts from digests 2020-08-04 13:35:48 -04:00
266c0c50d7 FIX: Load base color scheme when default theme is not set
Followup to c937afc
2020-08-04 12:15:07 -04:00
b76731d722 FEATURE: Invite emails to groups from add member modal (#10308) 2020-08-04 10:02:01 -05:00
8a0478b97d DEV: adds plugin api to add custom recipients of a post revision (#10367)
* DEV: adds plugin api to add custom recipients of a post revision

Usage:

```
add_post_revision_notifier_recipients do |post_revision|
  [78]
end
```
2020-08-04 11:57:33 +02:00
1f7b44de67 DEV: Remove flaky secure optimized image spec 2020-08-04 16:29:41 +10:00