122 Commits

Author	SHA1	Message	Date
rkettelerij	87fa9f8e46	azureblob: Add support for custom upload headers	2022-11-14 15:12:28 +00:00
Roel Arents	e455940f71	azureblob: allow emulator account/key override	2022-11-08 20:24:06 +00:00
albertony	5d6b8141ec	Replace deprecated ioutil ... As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code.	2022-11-07 11:41:47 +00:00
albertony	555def2da7	build: add package comments to silence revive linter	2022-08-28 13:43:51 +02:00
Nick Craig-Wood	0501773db1	azureblob,b2,s3: fix chunksize calculations producing too many parts ... Before this fix, the chunksize calculator was using the previous size of the object, not the new size of the object to calculate the chunk sizes. This meant that uploading a replacement object which needed a new chunk size would fail, using too many parts. This fix fixes the calculator to take the size explicitly.	2022-08-09 12:57:38 +01:00
Nick Craig-Wood	6fd9e3d717	build: reformat comments to pass go1.19 vet ... See: https://go.dev/doc/go1.19#go-doc	2022-08-05 16:35:41 +01:00
Lorenzo Maiorfi	b5efffee9d	azureblob: allow remote emulator (azurite) - fixes #6290	2022-07-06 11:54:04 +01:00
albertony	7822df565e	staticcheck: unused func	2022-07-04 11:24:59 +02:00
albertony	fdd2f8e6d2	Error strings should not be capitalized ... Reported by staticcheck 2022.1.2 (v0.3.2) See: staticcheck.io	2022-06-23 23:26:02 +02:00
Nick Craig-Wood	bb6edb3c39	build: update dependencies ... Also: - azureblob: fix compile after API change in upstream library	2022-06-08 18:29:42 +01:00
Rob Pickerill	6d342a3c5b	azureblob: case insensitive access tier	2022-05-24 09:19:08 +01:00
Derek Battams	c0985e93b7	azureblob: use chunksize lib to determine chunksize dynamically	2022-05-13 09:25:48 +01:00
Leroy van Logchem	75455d4000	azureblob: Calculate Chunksize/blocksize to stay below maxUploadParts	2022-04-26 17:37:40 +01:00
Nick Craig-Wood	e4f5912294	azureblob: fix lint error with golangci-lint 1.45.0	2022-03-22 16:33:24 +00:00
Nick Craig-Wood	c2557cc432	azureblob: fix crash with SAS URL and no container - fixes #5820 ... Before this change attempting NewObject on a SAS URL's root would crash the Azure SDK. This change detects that using the code from this previous fix f7404f52e7654d12 azureblob: fix crash when listing outside a SAS URL's root - fixes #4851 And returns not object not found instead. It also prevents things being uploaded to the root of the SAS URL which also crashes the Azure SDK.	2021-11-27 16:18:18 +00:00
Nick Craig-Wood	df07964db3	azureblob: raise --azureblob-upload-concurrency to 16 by default ... After speed testing it was discovered that upload speed goes up pretty much linearly with upload concurrency. This patch changes the default from 4 to 16 which means that rclone will use 16 * 4M = 64M per transfer which is OK even for low memory devices. This adds a note that performance may be increased by increasing upload concurrency. See: https://forum.rclone.org/t/performance-of-rclone-vs-azcopy/27437/9	2021-11-18 16:09:02 +00:00
Nick Craig-Wood	fbc4c4ad9a	azureblob: remove 100MB upper limit on chunk_size as it is no longer needed	2021-11-18 16:09:02 +00:00
Nick Craig-Wood	4454b3e1ae	azureblob: implement --azureblob-upload-concurrency parameter to speed uploads ... See: https://forum.rclone.org/t/performance-of-rclone-vs-azcopy/27437	2021-11-18 16:08:57 +00:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
albertony	e2f47ecdeb	docs: punctuation cleanup ... See #5538	2021-10-20 22:56:19 +02:00
Nick Craig-Wood	eb0c8284f1	azureblob: fix incorrect size after --azureblob-no-head-object patch ... In 05f128868f47b1b5 azureblob: add --azureblob-no-head-object we incorrectly parsed the size of the object as the Content-Length of the returned header. This is incorrect in the presense of Range requests. This fixes the problem by parsing the Content-Range header if avaialble to read the correct length from if a Range request was issued. See: #5734	2021-10-19 20:12:17 +01:00
Ivan Andreev	24de896df2	build: apply gofmt from golang 1.17	2021-09-09 20:43:59 +03:00
Tatsuya Noyori	05f128868f	azureblob: add --azureblob-no-head-object	2021-09-06 10:41:54 +01:00
Nick Craig-Wood	2cefae51a1	build: make go1.14 the minimum supported Go for building	2021-08-20 19:04:29 +01:00
Dmitry Sitnikov	b439199578	azureblob: Fix typo in Azure Blob help ... Change the command to create RBAC file to the correct one `az ad sp create-for-rbac` Add the link to the command documentation https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest#az_ad_sp_create_for_rbac	2021-07-05 15:58:41 +01:00
albertony	2925e1384c	Use binary prefixes for size and rate units ... Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.	2021-04-27 02:25:52 +03:00
Manish Kumar	ff6868900d	azureblob: add container public access level support. Fixes #5045	2021-03-15 17:18:47 +00:00
Nick Craig-Wood	4013bc4a4c	Fix excessive retries missing --max-duration timeout - fixes #4504 ... This change checks the context whenever rclone might retry, and doesn't retry if the current context has an error. This fixes the pathological behaviour of `--max-duration` refusing to exit because all the context deadline exceeded errors were being retried. This unfortunately meant changing the shouldRetry logic in every backend and doing a lot of context propagation. See: https://forum.rclone.org/t/add-flag-to-exit-immediately-when-max-duration-reached/22723	2021-03-13 09:25:44 +00:00
Nick Craig-Wood	3b1122c888	azureblob: require go1.14+ to compile due to SDK changes	2021-01-30 18:01:12 +00:00
Nick Craig-Wood	189ef5f257	azureblob: fix memory usage by upgrading the SDK and implementing a TransferManager ... In the Azure SDK there was a bug which caused excessive memory use when doing repeated transfers: https://github.com/Azure/azure-storage-blob-go/issues/233 This patch updates the SDK to v0.13.0 which allowed us to implement a custom TransferManager which integrates with rclone's memory pool. This fixes the excessive memory consumption. See: https://forum.rclone.org/t/ask-for-settings-recommendation-for-azureblob/21505/	2021-01-30 17:26:59 +00:00
Denis Neuling	ec73d2fb9a	azure-blob-storage: utilize streaming capabilities - #1614	2021-01-10 17:02:42 +00:00
Nick Craig-Wood	cb97c2b0d3	azureblob: fix crash on startup ... This was introduced by accidental code deletion in 08b9ede2173fb34f azureblob: add support for managed identities	2020-12-31 18:39:09 +00:00
Mitsuo Heijo	9ea990d5a2	azureblob: update azure-storage-blob-go to v0.12.0 ... See https://github.com/Azure/azure-storage-blob-go/blob/master/ChangeLog.md#version-0120	2020-12-28 13:29:38 +00:00
Brad Ackerman	08b9ede217	azureblob: add support for managed identities ... Fixes #3213	2020-12-28 13:23:35 +00:00
Nick Craig-Wood	f347a198f7	azureblob: delete archive tier blobs before update if --azureblob-archive-tier-delete ... Before this change, attempting to update an archive tier blob failed with a 409 error message: 409 This operation is not permitted on an archived blob. This change detects if we are overwriting a blob and either generates the error (if `--azureblob-archive-tier-delete` is not set): can't update archive tier blob without --azureblob-archive-tier-delete Or deletes the blob first before uploading it again (if `--azureblob-archive-tier-delete` is set). Fixes #4819	2020-12-28 12:31:24 +00:00
Nick Craig-Wood	f7404f52e7	azureblob: fix crash when listing outside a SAS URL's root - fixes #4851 ... Before this change if you attempted to list a remote set up with a SAS URL outside its container then it would crash the Azure SDK. A check is done to make sure the root is inside the container when starting the backend which is usually enough, but when two SAS URL based remotes are mounted in a union, the union backend attempts to read paths outside the named container. This was causing a mysterious crash in the Azure SDK. This fixes the problem by checking to see if the container in the listing is the one in the SAS URL before listing the directory and returning directory not found if it isn't.	2020-12-27 15:55:00 +00:00
James Lim	2fd543c989	azure: add support for service principals - fixes #3230 ... Before: users can only connect to Azure blob containers using the access keys from the storage account. After: users can additionally choose connect to Azure blob containers using service principals. This uses OAuth2 under the hood to exchange a client ID and client secret for a short-lived access token. Ref: - https://github.com/rclone/rclone/issues/3230 - https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app?tabs=dotnet#well-known-values-for-authentication-with-azure-ad - https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization#available-authentication-types-and-methods - https://gist.github.com/ItalyPaleAle/ec6498bfa81a96f9ca27a2da6f60a770	2020-12-09 17:52:15 +00:00
Nick Craig-Wood	dfadd98969	azureblob,memory,pcloud: fix setting of mime types ... Before this change the backend was reading the mime type of the destination object instead of the source object when uploading. This changes fixes the problem and introduces an integration test for it. See: https://forum.rclone.org/t/is-there-a-way-to-get-rclone-copy-to-preserve-metadata/20682/2	2020-11-27 14:40:05 +00:00
Nick Craig-Wood	9d574c0d63	fshttp: read config from ctx not passed in ConfigInfo #4685	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	1973fc1ecc	azureblob: update lib from v0.10.0 to v0.11.0 and fix API breakage ... See: https://github.com/Azure/azure-storage-blob-go/issues/226	2020-10-29 13:34:39 +00:00
Josh Soref	e4a87f772f	docs: spelling: e.g. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Nick Craig-Wood	3a14b1d5a9	build: make rclone build with wasm ... Needed to drop - azureblob backend - cache backend - qingstor backend - cachestats command - ncdu command	2020-08-10 17:32:21 +01:00
Nick Craig-Wood	a2afa9aadd	fs: Add directory to optional Purge interface - fixes #1891 ... - add a directory to the optional Purge interface - fix up all the backends - add an additional integration test to test for the feature - use the new feature in operations.Purge Many of the backends had been prepared in advance for this so the change was trivial for them.	2020-07-31 17:43:17 +01:00
Nick Craig-Wood	2a3b377d34	azureblob: don't compile on < go1.13 after dependency update	2020-06-29 14:45:39 +01:00
Martin Michlmayr	041b201abd	doc: fix typos throughout docs and code	2020-05-25 11:23:58 +01:00