Commit Graph

6925 Commits

Author SHA1 Message Date
ed
3f0bec2ee9 webdav: make pacer minSleep configurable
This adds the config argument --webdav-pacer-min-sleep which specifies
the http-request rate limit. Lowering this from the default 10ms can
greatly improve performance when synchronizing small files.

See: https://forum.rclone.org/t/webdav-with-persistent-connections/37024/10
2023-03-27 15:30:02 +02:00
8fb9eb2fee sync: make --suffix-keep-extension preserve 2 part extensions like .tar.gz
If a file has two (or more) extensions and the second (or subsequent)
extension is recognised as a valid mime type, then the suffix will go
before that extension. So `file.tar.gz` would be backed up to
`file-2019-01-01.tar.gz` whereas `file.badextension.gz` would be
backed up to `file.badextension-2019-01-01.gz`

Fixes #6892
2023-03-27 14:24:21 +01:00
01fa15a7d9 Add Aditya Basu to contributors 2023-03-27 14:24:21 +01:00
6aaa5d7a75 Add jumbi77 to contributors 2023-03-27 14:24:21 +01:00
b4d3411637 Add Juang, Yi-Lin to contributors 2023-03-27 14:24:21 +01:00
01ddc8ca6c Add NickIAm to contributors 2023-03-27 14:24:21 +01:00
16c1e7149e Add yuudi to contributors 2023-03-27 14:24:21 +01:00
0374ea2c79 Use jwt-go (golang-jwt) instead of deprecated jws (x/oauth2/jws)
golang.org/x/oauth2/jws is deprecated: this package is not intended for public use and
might be removed in the future. It exists for internal use only. Please switch to another
JWS package or copy this package into your own source tree.

github.com/golang-jwt/jwt/v4 seems to be a good alternative, and was already
an implicit dependency.
2023-03-26 19:20:50 +02:00
2e2451f8ec lib/rest: fix problems re-using HTTP connections
Before this fix, it was noticed that the rclone webdav client did not
re-use HTTP connections when it should have been.

This turned out to be because rclone was not draining the HTTP bodies
when it was not expecting a response.

From the Go docs:

> If the returned error is nil, the Response will contain a non-nil
> Body which the user is expected to close. If the Body is not both
> read to EOF and closed, the Client's underlying RoundTripper
> (typically Transport) may not be able to re-use a persistent TCP
> connection to the server for a subsequent "keep-alive" request.

This fixes the problem by draining up to 10MB of data from an HTTP
response if the NoResponse flag is set, or at the end of a JSON or XML
response (which could have some whitespace on the end).

See: https://forum.rclone.org/t/webdav-with-persistent-connections/37024/
2023-03-26 17:19:48 +01:00
bd1e3448b3 build: add exclude for misspell linter 2023-03-26 17:05:24 +02:00
20909fa294 build: enable misspell linter 2023-03-26 17:05:24 +02:00
c502e00c87 fs: fix infinite recursive call in pacer ModifyCalculator (fixes issue reported by the staticcheck linter) 2023-03-26 14:28:15 +02:00
9172c9b3dd crypt: reduce allocations
This changes crypt's use of sync.Pool: Instead of storing slices
it now stores pointers pointers fixed sized arrays.

This issue was reported by staticcheck:

SA6002 - Storing non-pointer values in sync.Pool allocates memory

A sync.Pool is used to avoid unnecessary allocations and reduce
the amount of work the garbage collector has to do.

When passing a value that is not a pointer to a function that accepts
an interface, the value needs to be placed on the heap, which means
an additional allocation. Slices are a common thing to put in sync.Pools,
and they're structs with 3 fields (length, capacity, and a pointer to
an array). In order to avoid the extra allocation, one should store
a pointer to the slice instead.

See: https://staticcheck.io/docs/checks#SA6002
2023-03-26 14:28:15 +02:00
78deab05f9 netstorage: ignore false positive from the staticcheck linter regarding header name not being canonical 2023-03-26 14:28:15 +02:00
6c9d377bbb vfs: ignore false positive from the unused linter 2023-03-26 14:28:15 +02:00
62ddc9b7f9 vfscache: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
448ae49fa4 webgui: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
5f3c276d0a zoho: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
9cea493f58 union: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
400d1a4468 swift: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
851ce0f4fe seafile: remove unused code for legacy API v2 (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
cc885bd39a hidrive: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
a1a8c21c70 dropbox: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
6ef4bd8c45 cache: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
fb316123ec azureblob: remove unused code (fixes issue reported by the unused linter) 2023-03-26 14:28:15 +02:00
270af61665 smb: code cleanup to avoid overwriting ctx before first use (fixes issue reported by the staticcheck linter) 2023-03-26 14:28:15 +02:00
155f4f2e21 mount: replace deprecated bazil/fuse specific constants with syscall constants 2023-03-26 14:28:15 +02:00
eaf593884b serve/ftp: use io.SeekEnd instead of os.SEEK_END (deprecated since Go 1.7) 2023-03-26 14:28:15 +02:00
930574c6e9 oracleobjectstorage: remove empty branch (fixes issue reported by the staticcheck linter) 2023-03-26 14:28:15 +02:00
c1586a9866 onedrive: report any list errors during cleanup 2023-03-26 14:28:15 +02:00
432eb74814 lib: avoid unnecessary use of fmt.Sprintf for string constant 2023-03-26 14:28:15 +02:00
92fb644fb6 test: use decompressed.String() instead of string(decompressed.Bytes()) 2023-03-26 14:28:15 +02:00
bb92af693a test: do not test deprecated and unused Dial and DialTLS functions on http Transport type 2023-03-26 14:28:15 +02:00
eb5fd07131 mount: error strings should not be capitalized 2023-03-26 14:28:15 +02:00
b2ce7c9aa6 hidrive: error strings should not be capitalized 2023-03-26 14:28:15 +02:00
d6b46e41dd build: replace deprecated linters deadcode, structcheck and varcheckadd with unused
The three linters deadcode, structcheck and varcheck we have been using are as of
gitlabci-lint version 1.49.0 (24 Aug 2022) marked as deprecated, and replaced by unused.

The linters staticcheck, gosimple, stylecheck and unused should combined correspond to
the checks performed by the stand-alone staticcheck tool, which is by default used for
linting in Visual Studio Code with the Go extension. We previously enabled the first
three, but skipped unused due to many reported issues.

See #6387 for more information.
2023-03-26 14:28:15 +02:00
254c6ef1dd build: add lint ignore comment required for golangci-staticcheck in addition to stand-alone staticcheck 2023-03-26 14:28:15 +02:00
547f943851 build: exclude known issues from the staticcheck linting in ci 2023-03-26 14:28:15 +02:00
8611c9f6f7 build: add staticcheck, gosimple and stylecheck linting to the build pipeline - fixes #6273
These combined should correspond to the checks performed by the stand-alone
staticcheck tool, which is by default used for linting in Visual Studio Code
with the Go extension. One exception is the unused checks, which staticcheck
tool performs, but chose to not enabled here in rclone due to many reported
occurrences.

See #6387 for more information.
2023-03-26 14:28:15 +02:00
f6576237a4 fs: fix typos found by codespell 2023-03-25 12:51:04 +01:00
207b64865e fstest: fix typo found by codespell 2023-03-25 09:34:10 +01:00
9ee1b21ec2 vfs: fix typos found by codespell 2023-03-25 09:33:34 +01:00
55a12bd639 backend: fix repeated words typos 2023-03-25 09:31:36 +01:00
3b4a57dab9 build(deps): bump github.com/aws/aws-sdk-go from 1.44.227 to 1.44.228
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.227 to 1.44.228.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.227...v1.44.228)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-24 20:49:50 +00:00
afe158f878 docs: fix typos found by codespell 2023-03-24 20:49:00 +00:00
722a3f32cc cmdtest: fix typos found by codespell 2023-03-24 20:44:25 +00:00
9183618082 backend: fix typos found by codespell 2023-03-24 20:42:45 +00:00
18ebca3979 lib: fix typos found by codespell 2023-03-24 20:40:52 +00:00
e84d2c9e5f docs: add info about # of parallel checks for rclone check/cryptcheck
The original commit 7dbf1ab66f4ff45f put the changes in the auto
generated docs - this fixes that.
2023-03-24 12:43:45 +00:00
e98b61ceeb docs: update install with docker interactive use
* Install with docker: interactive use
* remove extra mount from command
* update listremotes
2023-03-24 11:42:58 +00:00