Commit Graph

883 Commits

Author SHA1 Message Date
0d3d2c43a0 DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
c218036107 FIX: Make Google Groups scraper work for G Suite users 2019-11-28 02:09:51 +01:00
88ecb650a9 DEV: Implement a faster Discourse.cache
This is a bottom up rewrite of Discourse cache to support faster performance
and a limited surface area.

ActiveSupport::Cache::Store accepts many options we do not use, this partial
implementation only picks the bits out that we do use and want to support.

Additionally params are named which avoids typos such as "expires_at" vs "expires_in"

This also moves a few spots in Discourse to use Discourse.cache over setex
Performance of setex and Discourse.cache.write is similar.
2019-11-27 16:11:49 +11:00
0fb497eb23 DEV: use Discourse.cache over Rails.cache
Discourse.cache is a more consistent method to use and offers clean fallback
if you are skipping redis

This is part of a larger change that both optimizes Discoruse.cache and omits
use of setex on $redis in favor of consistently using discourse cache

Bench does reveal that use of Rails.cache and Discourse.cache is 1.25x slower
than redis.setex / get so a re-implementation will follow prior to porting
2019-11-27 12:36:19 +11:00
3bb7ad4be1 FEATURE: remove support for 'suppress_from_latest' category setting. (#8308) 2019-11-18 12:28:35 +05:30
067696df8f DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
9fea43e46a DEV: Remove use of cd in the app (#8337)
`FileUtils.cd` and `Dir.chdir` cause the working directory to change for the entire process. We run sidekiq jobs, hijacked requests and deferred jobs in threads, which can make working directory changes have unintended side-effects.

- Add a rubocop rule to warn about usage of Dir.chdir and FileUtils.cd
- Added rubocop:disable for scripts used outside the app
- Refactored code using cd to use alternative methods
- Temporarily skipped the rubocop check for lib/backup_restore. This will require more complex refactoring, so I will create a separate PR for review
2019-11-13 09:57:39 +00:00
bf0ef73286 DEV: correct rake task used to grab admin key
We amended it so "api_key:get" is no longer supported and instead we are
more explicit. This matches that change and fixes the bench.
2019-11-11 10:23:14 +11:00
54fe887c44 DEV: Remove prototype theme-watcher script
This has been superseded by the Theme CLI: https://meta.discourse.org/t/82950
2019-11-07 17:22:54 +00:00
d76d0e75ec DEV: Move warmup inside docker rake task 2019-10-25 16:31:05 -03:00
4f452f0205 DEV: Add variable to warmup tmp folder and obtain accurate results when profiling specs 2019-10-25 10:52:23 -03:00
f34a0141c7 FIX: Correct path to ImportExport module (#8227)
During the move from Classic autoloader to Zeitwerk import_export module was moved to correct file name convention.
427d54b2b0 (diff-d896ec33b95afb7fae9f8bfe73d0580b)

Problem is that export/import is still using old path to require that module

Meta: https://meta.discourse.org/t/topic-and-category-export-import/38930/40
2019-10-23 17:27:14 +11:00
1352a5b5fa DEV: undo pluck_first changes to micro benchmark
and add pluck_first benchmark
2019-10-21 12:21:24 +01:00
55a1394342 DEV: pluck_first
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:

pluck_first, equivalent to limit(1).pluck(*columns).first

and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
e1998ef244 FIX: downsize_uploads script
The script will now correct all width/height and thumbnail_width/thumbnail_height properties of all the uploaded images.

The script now uses width * height to filter out all unaffected images.

Also handled the case where a downsized image was already an uploaded record.
2019-10-10 16:37:55 +02:00
4fdad12998 FIX: downsize_uploads script to support external storage
Also ensured we update the sha1 property of the upload record to match the actual file.
2019-10-08 17:54:39 +02:00
2d4c9bbaac import_scripts: add fluxbb prefix to missing query (#8163)
Signed-off-by: Jelle van der Waa <jelle@archlinux.org>
2019-10-08 11:46:00 +11:00
1d5c2b36f6 DEV: improve diagnostics on mem leak checker
This adds mwrap logging to each iteration so we can see how much
leaks per iteration and where it is coming from
2019-10-04 09:47:33 +10:00
038a38ae1c DEV: add debugging scripts for memory leaks
These scripts are somewhat rough but I needed them to help debug a memory
leak we have noticed in rails 6.

The biggest object script finds all the biggest objects we have in memory
after boot.

The test memory leak runs a very simple iteration through all multisites
and observed memory.
2019-10-03 16:36:31 +10:00
35b1185a08 FIX: Revert Demon::DemonBase back to Demon::Base (#8132)
I introduced DemonBase because I had got some conflict between `demon/base.rb` and `jobs/base.rb`, however, to not rename base class, it is possible to use regex on absolute path in Zeitwerk custom inflector.
2019-10-02 14:54:08 +10:00
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
b48ca9dee9 DEV: Simplify username validation in base importer
The `UsernameValidator` does already all the hard work. No need to do any additional checks in the import script. The checks were out-of-date anyway.
2019-10-01 20:33:09 +02:00
ed1e5ef6cc FIX: By default, don't abort Google Groups crawling on error 2019-09-18 18:14:04 +02:00
ab96239f2a FIX: Google Groups crawler failed to login
Trying to automate the login into a Google account is quite hard. This makes the crawler use the content of a cookies.txt file instead. It also removes a couple of deprecation warnings and adds some color to the output.
2019-09-18 13:09:20 +02:00
cd1ab206d9 DEV: add missing ultra low queue to mwrap sidekiq
note: mwrap is used for analysis of memory bloat and leaks of processes
2019-09-18 11:18:35 +10:00
015051ecaf PERF: avoid spinning a thread each time we close a connection
This is a temporary workaround for the issue in https://github.com/rails/rails/pull/36949

Discussing a proper fix in Rails with the Rails team.

Prior to this fix we were spinning up a thread every time we closed a connection
to the db.
2019-09-12 17:34:04 +10:00
1d73754e84 FIX: Modify frozen String and profile_db_generator uses category id (#8080) 2019-09-09 17:38:37 +10:00
ab7038bfc2 DEV: User simulator tried to modify frozen string. 2019-08-16 17:32:17 +03:00
888b635cfc Import avatars and likes in the Zendesk AP importer
Co-authored-by: Justin DiRose <justin@justindirose.com>
2019-08-14 10:42:52 +02:00
4ed517a344 DEV: Make Rubocop happy
Follow-up to 6cc9fe42
2019-08-12 23:10:58 +02:00
6cc9fe42ce add mongo adapter to nodebb importer (#8000) 2019-08-12 14:15:11 -04:00
19dda59932 FIX: add back verbose option to DbHelper.remap 2019-07-31 17:30:08 +02:00
dcb47d902b REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications (#7958)
* REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications

- The older name could cause some confusion because the setting does not disable all edit notifications, only system ones.

* FIX: Add frozen_string_literal: true in the migration

* DEV: Deprecate 'disable_edit_notifications'
2019-07-31 20:20:41 +05:30
89fce2ce71 DEV: remove duplicate Remap class and use DbHelper.remap instead
Follow-up to 9cd3f96dee96d3fdda99174077fb79c1d6f26ea3
2019-07-29 18:43:40 +02:00
fd12c414e7 DEV: Refactor helper methods for upload markdown
Follow-up to a61ff167
2019-07-25 16:36:35 +02:00
a61ff16740 DEV: Make attachment markdown reusable 2019-07-25 14:04:18 +02:00
cc46de8f46 s/discourse-staff-notes/discourse-user-notes (#7936) 2019-07-24 20:04:27 +02:00
f0fea5991f FIX: Latest Selenium gem broke Google Groups import script
Selenium uses Keep-Alive since version 3.141, so the net-http-persistent gem shouldn't be needed anymore.
2019-07-10 09:45:33 +02:00
ab6ad220c7 DEV: Fix user simulator script. 2019-07-09 18:52:08 +03:00
6d30be1f94 Improve XenForo import script.
- ensure only active, unbanned users are imported.
- ensure only visible threads/posts are imported.
2019-06-18 15:52:34 +05:30
77f5577e30 DEV: Improvements to AnswerHub import script. 2019-06-13 11:46:17 +05:30
36c0cfa890 FIX: Use new attachment markdown format in ImportScripts::Uploader. 2019-06-11 14:49:28 +08:00
0955d9ece9 create answerhub importer (#7671) 2019-06-03 12:17:22 +10:00
0f3c3bc309 Make import scripts work with frozen strings 2019-05-30 22:22:24 +02:00
c70d0c6659 Use an invalid domain for fake email addresses in importers 2019-05-30 22:22:24 +02:00
d3ba338144 Make Telligent import script more generic 2019-05-30 22:22:24 +02:00
630e9814bc datetime is not available at this point (#7630) 2019-05-29 14:06:32 +02:00
6439004161 DEV: do not use STDERR to print tests timestamps (#7629) 2019-05-29 13:28:02 +02:00
6be9a6eb2e DEV: adds time logging to docker_test script (#7627) 2019-05-29 12:06:43 +02:00
7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00