8675 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nick Craig-Wood	bbaa1d14f3	purge: exit with a fatal error if filters are set on rclone purge ... Fixes #8491	2025-04-08 21:12:36 +01:00
Nick Craig-Wood	9e62fa8c81	test logging - FIXME	2025-04-08 21:04:26 +01:00
Nick Craig-Wood	14fd2fd4b8	onedrive: fix metadata ordering in permissions ... Before this change, due to a quirk in Graph, User permissions could be lost when applying permissions. Fixes #8465	2025-04-08 21:04:16 +01:00
Nick Craig-Wood	385465bfa9	sync: implement --list-cutoff to allow on disk sorting for reduced memory use ... Before this change, rclone had to load an entire directory into RAM in order to sort it so it could be synced. With directories with millions of entries, this used too much memory. This fixes the probem by using an on disk sort when there are more than --list-cutoff entries in a directory. Fixes #7974	2025-04-08 18:02:24 +01:00
Nick Craig-Wood	0148bd4668	march: Implement callback based syncing ... This changes the syncing method to take callbacks for directory listings rather than being passed the entire directory listing at once. This will enable out of memory syncing.	2025-04-08 18:02:24 +01:00
Nick Craig-Wood	0f7ecf6f06	list: add ListDirSortedFn for callback oriented directory listing ... This will be used for the out of memory sync	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	08e81f8420	list: Implement Sorter to sort directory entries ... Later this will be extended to do out of memory sorts	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	0ac2d2f50f	cache: mark ListP as not supported yet	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	42fcb0a6fc	hasher: implement ListP interface	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	490dd14bc5	compress: implement ListP interface	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	943ea0acae	chunker: mark ListP as not supported yet	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	d64a97f973	union: mark ListP as not supported yet	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	5d8f1d4b88	crypt: implement ListP interface	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	b1d774c2e3	combine: implement ListP interface	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	fad579c4a2	s3: Implement paged listing interface ListP	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	37120ef7bd	list: add WithListP helper to implement List for ListP backends	2025-04-08 15:14:09 +01:00
Nick Craig-Wood	cba653d502	walk: move NewListRHelper into list.Helper to avoid circular dependency ... It turns out that the list helpers were at the wrong level and needed to be pushed down into the fs/list for future work.	2025-04-08 15:14:00 +01:00
Nick Craig-Wood	2a90de9502	fs: define ListP interface for paged listing #4788	2025-04-08 15:12:53 +01:00
Nick Craig-Wood	bff229713a	accounting: Add listed stat for number of directory entries listed	2025-04-08 15:12:53 +01:00
Nick Craig-Wood	117f583ebe	walk: factor Listing helpers into their own file and add tests	2025-04-08 15:12:53 +01:00
Nick Craig-Wood	205667143c	serve nfs: make metadata files have special file handles ... Metadata files have the file handle of their source file with 0x00000001 suffixed in big endian so we can look them up directly from their file handles.	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	fe84cbdc9d	serve nfs: change the format of --nfs-cache-type symlink file handles ... This is an backwards incompatible change which will invalidate the current handles. This change adds a 4 byte big endian length prefix to the handles so we can in future suffix extra info on the handles. This needed to be 4 bytes as Linux does not like File handles which aren't multiples of 4 bytes long.	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	533c6438f3	vfs: add --vfs-metadata-extension to expose metadata sidecar files ... This adds --vfs-metadata-extension which can be used to expose sidecar files with file metadata in. These files don't exist in the listings until they are accessed.	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	b587b094c9	docs: Add rcloneui.com as Silver Sponsor	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	525798e1a5	Add Klaas Freitag to contributors	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	ea63052d36	Add eccoisle to contributors	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	b5a99c5011	Add Fernando Fernández to contributors	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	56b7015675	Add alingse to contributors	2025-04-07 13:41:29 +01:00
Nick Craig-Wood	4ff970ebab	Add Jörn Friedrich Dreyer to contributors	2025-04-07 13:41:29 +01:00
eccoisle	dccb5144c3	docs: replace option --auto-filename-header with --header-filename	2025-04-06 14:28:34 +02:00
dependabot[bot]	33b087171a	build: update github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 to fix CVE-2025-30204 ... Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2. - [Release notes](https://github.com/golang-jwt/jwt/releases) - [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md) - [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2) See: https://github.com/golang-jwt/jwt/security/advisories/GHSA-mh63-6h87-95cp See: https://www.cve.org/CVERecord?id=CVE-2025-30204 Signed-off-by: dependabot[bot] <support@github.com>	2025-04-06 11:46:13 +01:00
Fernando Fernández	58d9ae1c60	docs/googlephotos: fix typos	2025-04-06 10:49:02 +02:00
dependabot[bot]	20302ab6b9	build: bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2 ... Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.1 to 4.5.2. - [Release notes](https://github.com/golang-jwt/jwt/releases) - [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md) - [Commits](https://github.com/golang-jwt/jwt/compare/v4.5.1...v4.5.2) --- updated-dependencies: - dependency-name: github.com/golang-jwt/jwt/v4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>	2025-04-04 17:26:44 +02:00
alingse	6fb0de62a4	operations: fix call fmt.Errorf with wrong err	2025-04-04 16:21:45 +02:00
Jörn Friedrich Dreyer	839eef0db2	webdav: retry propfind on 425 status ... This retries propfind on 425 status In ownCloud Infinite Scale, files might be in that state if postprocessing is still ongoing. All metadata are available anyway Allow item status 425 "too early" for items when changing metadata Fixes the upload behavior with ownCloud Infinite Scale Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> Co-authored-by: Klaas Freitag <kraft@freisturz.de>	2025-03-26 12:51:04 +00:00
Nick Craig-Wood	267eebe5c9	Add --max-connections to control maximum backend concurrency	2025-03-25 15:49:27 +00:00
Nick Craig-Wood	755d72a591	rc: fix debug/* commands not being available over unix sockets ... This was caused by an incorrect handler URL which was passing the debug/* commands to the debug/pprof handler by accident. This only happened when using unix sockets.	2025-03-25 15:30:49 +00:00
Dan McArdle	4d38424e6c	cmd/gitannex: Prevent tests from hanging when assertion fails ... This fixes another way that the gitannex tests can hang. The issue is that our test harness explicitly called `wg.Done()` at the end of each test case, but when assertions checked with [require] fail, they halt test execution and prevent `wg.Done()` from happening. A second issue is that we were incorrectly calling [require] functions in the goroutine that runs the gitannex server. I found that [require] calls [testing.T.FailNow] under the hood, which says "FailNow must be called from the goroutine running the test or benchmark function, not from other goroutines created during the test." [1] This commit fixes both issues by replacing the explicit synchronization with a `chan error`. This enables us to run the gitannex server in a goroutine, interact with the server in the test's goroutine, and then at then end use [require] on the test-associated goroutine to ensure the server's error/nil value matches expectations. [1]: https://pkg.go.dev/testing#T.FailNow	2025-03-18 12:38:04 +00:00
Dan McArdle	53624222c9	cmd/gitannex: Add explicit timeout for mock stdout reads in tests ... It seems like (*testState).readLine() hangs indefinitely when it's waiting for a line that will never be written [1]. This commit adds an explicit 30-second timeout when reading from the internal mock stdout. Given that we integrate with fstest, this timeout needs to be sufficiently long that it accommodates slow-but-successful operations on real remotes. [1]: https://github.com/rclone/rclone/pull/8423#issuecomment-2701601290	2025-03-18 12:38:04 +00:00
nielash	44e83d77d7	http: correct root if definitely pointing to a file - fixes #8428 ... This was formalized in c69eb84573 But it appears that we forgot to update `http`, and the `FsRoot` test didn't catch it because we don't currently have an http integration test.	2025-03-17 18:05:23 +00:00
Nick Craig-Wood	19aa366d88	pool: add --max-buffer-memory to limit total buffer memory usage	2025-03-17 18:01:15 +00:00
Nick Craig-Wood	3fb4164d87	filter: Add --hash-filter to deterministically select a subset of files ... Fixes #8400	2025-03-17 17:25:59 +00:00
dependabot[bot]	4e2b78f65d	build: update golang.org/x/net to 0.36.0. to fix CVE-2025-22869 ... SSH servers which implement file transfer protocols are vulnerable to a denial of service attack from clients which complete the key exchange slowly, or not at all, causing pending content to be read into memory, but never transmitted. This updates golang.org/x/net to fix the problem. See: https://pkg.go.dev/vuln/GO-2025-3487 See: https://www.cve.org/CVERecord?id=CVE-2025-22869 Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2025-03-17 17:25:12 +00:00
Nick Craig-Wood	e47f59e1f9	rc: add add short parameter to core/stats to not return transferring and checking	2025-03-17 13:44:37 +00:00
Nick Craig-Wood	63c4fef27a	fs: fix corruption of SizeSuffix with "B" suffix in config (eg --min-size) ... Before this change, the config system round tripped fs.SizeSuffix values through strings like this, corrupting them in the process. "2B" -> 2 -> "2" -> 2048 This caused `--min-size 2B` to be interpreted as `--min-size 2k`. This fix makes sure SizeSuffix values have a "B" suffix when turned into a string where necessary, so it becomes "2B" -> 2 -> "2B" -> 2 In rclone v2 we should probably declare unsuffixed SizeSuffix values are in bytes not kBytes (done for rsync compatibility) but this would be a backwards incompatible change which we don't want for v1. Fixes #8437 Fixes #8212 Fixes #5169	2025-03-13 09:56:20 +00:00
Nick Craig-Wood	a7a7c1d592	filters: show --min-size and --max-size in --dump filters	2025-03-12 12:32:21 +00:00
Nick Craig-Wood	6a7e68aaf2	build: check docs for edits of autogenerated sections ... This adds a lint step which checks the top commit for edits to autogenerated doc sections.	2025-03-10 22:07:19 +00:00
Nick Craig-Wood	6e7a3795f1	Add jack to contributors	2025-03-10 22:07:19 +00:00
jack	177337686a	docs: fix incorrect mentions of vfs-cache-min-free-size	2025-03-09 01:23:42 +01:00
Nick Craig-Wood	ccef29bbff	fs/object: fix memory object out of bounds Seek	2025-03-06 11:31:52 +00:00