github-mirror/rclone
2
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-06-28 21:43:39 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
8,811 Commits 186 Branches 121 Tags
archive-backend
Commit Graph

361 Commits

Author SHA1 Message Date
Chris Lu
1f846c18d4 s3: Add SeaweedFS 2021-06-08 09:59:57 +01:00
Nick Craig-Wood
c0cda087a8 s3: don't check to see if remote is object if it ends with / 
Before this change, rclone would always check the root to see if it
was an object.

This change doesn't check to see if the root is an object if the path
ends with a /

This avoids a transaction where rclone HEADs the path to see if it
exists.

See #4990
 2021-05-17 16:43:34 +01:00
Tatsuya Noyori
4401d180aa s3: add --s3-no-head-object 
This stops rclone doing any HEAD requests on objects.
 2021-04-28 11:05:54 +01:00
albertony
2925e1384c Use binary prefixes for size and rate units 
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
 2021-04-27 02:25:52 +03:00
Nick Craig-Wood
e618ea83dd s3: remove WebIdentityRoleProvider to fix crash on auth #5255 
This code removes the code added in

15d19131bd6bc78e s3: use aws web identity role provider

This code no longer works because it doesn't initialise the
tokenFetcher - leading to a nil pointer crash.

The proper way to initialise this is with the
NewWebIdentityCredentials but it isn't clear where to get the other
parameters: roleARN, roleSessionName, path.

In the linked issue a user reports rclone working with EKS anyway, so
perhaps this code is no longer needed.

If it is needed, hopefully someone who knows AWS better will come
along and fix it!

See: https://forum.rclone.org/t/add-support-for-aws-sso/23569
 2021-04-26 16:55:50 +01:00
Nick Craig-Wood
b9a015e5b9 s3: fix --s3-profile which wasn't working - fixes #4757 2021-03-16 16:25:07 +00:00
Nick Craig-Wood
f2c0f82fc6 backends: Add context checking to remaining backends #4504 
This is a follow up to 4013bc4a4cd768e2 which missed some backends.

It adds a ctx parameter to shouldRetry and checks it.
 2021-03-16 16:17:22 +00:00
Nick Craig-Wood
f7e3115955 s3: fix Wasabi HEAD requests returning stale data by using only 1 transport 
In this commit

fc5b14b62046b240 s3: Added `--s3-disable-http2` to disable http/2

We created our own transport so we could disable http/2. However the
added function is called twice meaning that we create two HTTP
transports. This didn't happen with the original code because the
default transport is cached by fshttp.

Rclone normally does a PUT followed by a HEAD request to check an
upload has been successful.

With the two transports, the PUT and the HEAD were being done on
different HTTP transports. This means that it wasn't re-using the same
HTTP connection, so the HEAD request showed the previous object value.
This caused rclone to declare the upload was corrupted, delete the
object and try again.

This patch makes sure we only create one transport and use it for both
PUT and HEAD requests which fixes the problem with Wasabi.

See: https://forum.rclone.org/t/each-time-rclone-is-run-1-3-fails-2-3-succeeds/22545
 2021-03-05 15:34:56 +00:00
Nick Craig-Wood
b029fb591f s3: fix failed to create file system with folder level permissions policy 
Before this change, if folder level access permissions policy was in
use, with trailing `/` marking the folders then rclone would HEAD the
path without a trailing `/` to work out if it was a file or a folder.
This returned a permission denied error, which rclone returned to the
user.

    Failed to create file system for "s3:bucket/path/": Forbidden: Forbidden
        status code: 403, request id: XXXX, host id:

Previous to this change

53aa03cc44b2a425 s3: complete sse-c implementation

rclone would assume any errors when HEAD-ing the object implied it
didn't exist and this test would not fail.

This change reverts the functionality of the test to work as it did
before, meaning any errors on HEAD will make rclone assume the object
does not exist and the path is referring to a directory.

Fixes #4990
 2021-02-24 20:35:44 +00:00
Dmitry Chepurovskiy
1605f9e14d s3: Fix shared_credentials_file auth 
S3 backend shared_credentials_file option wasn't working neither from
config option nor from command line option. This was caused cause
shared_credentials_file_provider works as part of chain provider, but in
case user haven't specified access_token and access_key we had removed
(set nil) to credentials field, that may contain actual credentials got
from ChainProvider.

AWS_SHARED_CREDENTIALS_FILE env varible as far as i understood worked,
cause aws_sdk code handles it as one of default auth options, when
there's not configured credentials.
 2021-02-17 12:04:26 +00:00
Nick Craig-Wood
bbe791a886 swift: update github.com/ncw/swift to v2.0.0 
The update to v2 of the swift library introduces a context parameter
to each function. This required a lot of mostly mechanical changes
adding context parameters.

See: https://github.com/ncw/swift/issues/159
See: https://github.com/ncw/swift/issues/161
 2021-02-03 20:23:37 +00:00
Nick Craig-Wood
bcac8fdc83 Use http.NewRequestWithContext where possible after go1.13 minimum version 2021-02-03 17:41:27 +00:00
Nick Craig-Wood
8b41dfa50a s3: add --s3-no-head parameter to minimise transactions on upload 
See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
 2021-02-02 10:07:48 +00:00
Louis Koo
cdaea62932 s3: fix copy multipart with v2 auth failing with 'SignatureDoesNotMatch' 
Signed-off-by: zhuc <zhucan.k8s@gmail.com>
 2021-01-27 14:43:02 +00:00
Nick Craig-Wood
3877df4e62 s3: update help for --s3-no-check-bucket #4913 2021-01-10 17:54:19 +00:00
kelv
9e87f5090f s3: add requester pays option - fixes #301 2020-12-27 15:43:44 +00:00
Anagh Kumar Baranwal
8a429d12cf s3: Added error handling for error code 429 indicating too many requests 
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-12-01 18:13:31 +00:00
Nick Craig-Wood
9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
76ee3060d1 s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C 
Before this change, small objects uploaded with SSE-AWS/SSE-C would
not have MD5 sums.

This change adds metadata for these objects in the same way that the
metadata is stored for multipart uploaded objects.

See: #1824 #2827
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
4bb241c435 s3: store md5 in the Object rather than the ETag 
This enables us to set the md5 to cache it.

See: #1824 #2827
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
a06f4c2514 s3: fix hashes on small files with aws:kms and sse-c 
If rclone is configured for server side encryption - either aws:kms or
sse-c (but not sse-s3) then don't treat the ETags returned on objects
as MD5 hashes.

This fixes being able to upload small files.

Fixes #1824
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
53aa03cc44 s3: complete sse-c implementation 
This now can complete all operations with SSE-C enabled.

Fixes #2827
See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
8b96933e58 fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Josh Soref
0a6196716c docs: style: avoid double-nesting parens 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
a15f50254a docs: grammar: if, then 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
5d4f77a022 docs: grammar: Oxford comma 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
a089de0964 docs: grammar: uncountable: links 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
3068ae8447 docs: grammar: count agreement: files 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
67ff153b0c docs: grammar: article: a-file 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d4f38d45a5 docs: spelling: high-speed 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Anagh Kumar Baranwal
fc5b14b620 s3: Added --s3-disable-http2 to disable http/2 
Fixes #4673

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-10-13 17:11:22 +01:00
Anagh Kumar Baranwal
e3a5bb9b48 s3: Add missing regions for AWS 
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-10-06 16:54:42 +01:00
Christopher Stewart
f3cf6fcdd7 s3: fix spelling mistake 
Fix spelling mistake "patific" => "pacific"
 2020-09-18 12:03:13 +01:00
wjielai
22937e8982 docs: add Tencent COS to s3 provider list - fixes #4468 
* add Tencent COS to s3 provider list.

Co-authored-by: wjielai <wjielai@tencent.com>
 2020-09-08 16:34:25 +01:00
Nick Craig-Wood
725ae91387 s3: reduce the default --s3-copy-cutoff to < 5GB 
The maximum value for the --s3--copy-cutoff should be 5GiB as tested
with AWS S3.

However b2 have implemented this as 5GB rather than 5GiB so having the
default at 5 GiB makes the b2s3 server side copy of a large file by
default.

This patch sets the default to 4768 MiB which is slightly less than
5GB.

This should have very little effect on anything.

If in future rclone can lower this limit more if Copy can multithread.

See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/76
 2020-09-01 18:53:29 +01:00
Nick Craig-Wood
b7dd3ce608 s3: preserve metadata when doing multipart copy 
Before this change the s3 multipart server side copy was not
preserving the metadata of the object. This was most noticeable
because the modtime was not preserved.

This change fetches the metadata from the object before starting the
copy and overwrites it if requires.

It will also mean any other metadata is preserved.

See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
 2020-09-01 18:39:30 +01:00
Egor Margineanu
921e384c4d s3: update IBM COS endpoints - fixes #4522 2020-08-30 17:21:11 +01:00
Nick Craig-Wood
801a820c54 s3: fix detection of bucket existing 
This reverts part of

151f03378f s3: fix upload of single files into buckets without create permission

This erroneously assumed that a HEAD request on a non existent object
would return "NotFound" if the bucket was found. In fact it returns
"NotFound" when the bucket isn't found also.

This will break the fix for #4297 - however that can be made to work
using the new --s3-assume-bucket-exists flag
 2020-08-21 13:28:08 +01:00
Nick Craig-Wood
d5f4c74697 s3: implement cleanup and backend command to list & remove multipart uploads 
This implements `rclone cleanup` to remove multipart uploads over 24
hours old. It also implements the backend command
`list-multipart-uploads` to see which ones are available and `cleanup`
to delete them with a configurable expiry interval.

See #4302
 2020-07-28 11:37:46 +01:00
Nick Craig-Wood
2288a5c617 s3: implement profile and shared_credentials_file options 
It is impossible to use two different profiles at the same time -
these config vars enable that.

See: https://forum.rclone.org/t/s3-source-destination-named-profile/17417
 2020-07-28 11:32:32 +01:00
Nick Craig-Wood
f406dbbb4d s3: add --s3-no-check-bucket for minimising rclone transactions and perms 
Fixes #4449
 2020-07-27 17:49:40 +01:00
Nick Craig-Wood
80d2f38192 s3: fix bucket Region auto detection when Region unset in config #2915 
Previous to this fix if Region was not set and Endpoint was not set
then we set the endpoint to "https://s3.amazonaws.com/".

This is unecessary because if the Region alone isn't set then we set
it to "us-east-1" which has the same endpoint.

Having the endpoint set breaks the bucket region auto detection with
the error "Failed to update region for bucket: can't set region to
"xxx" as endpoint is set".

This fix removes that check.
 2020-07-10 17:16:59 +01:00
Nick Craig-Wood
c820576329 fs: define SlowModTime and SlowHash features in the relevant backends 2020-06-30 12:01:36 +01:00
David
9058ec32e1 s3: Use regional s3 us-east-1 endpoint 2020-06-26 16:25:52 +01:00
Nick Craig-Wood
fd7c63bc78 s3: add backend restore command to restore objects from GLACIER 
See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
 2020-06-25 21:33:23 +01:00