aaa1370a36
Add directory parameter to Rmdir and Mkdir #100 #831
...
This will enable rclone to manage directories properly in the future.
2016-11-26 12:02:53 +00:00
441951a93b
Stop removing failed upload to cloud storage remotes - fixes #559
...
We do remove a partially written file on local so we don't have
corrupted files lying around.
2016-11-04 21:34:25 +00:00
aef2ac5c04
Add options for Open and implement Range for all remotes
2016-10-05 21:03:56 +01:00
945f49ab5e
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-10-03 20:02:04 +01:00
0cb9bb3b54
Redo http Transport code
...
* Insert User-Agent in Transport - fixes #199
* Update timeouts to use Context
* Modernise transport
2016-09-12 17:50:19 +01:00
346d4c587c
swift: don't read metadata for directory marker objects - fixes #703
2016-09-08 16:44:11 +01:00
e6a0521ca2
Make it possible to test Fs multiple times and use this with crypt
...
We test both the filename encryption modes for crypt.
2016-08-23 17:45:37 +01:00
c2f6decb9c
swift: note that tenant isn't optional for > v1 auth - fixes #563
2016-07-15 18:25:59 +01:00
02a3bbaa3d
swift: add support for non-default project domain.
...
With Keystone V3 both users and projects (a.k.a. tenants) can belong
to different domains. This change allow specifying different domains
for the user and the project.
2016-07-11 15:16:58 +01:00
3ad8fb8634
Make DeleteFile and DeleteFiles return errors
2016-06-28 08:51:57 +01:00
ab43005422
Make NewObject return an error
...
* make it return an error
* make a canonical error fs.ErrorNotFound
* make a test for it
* remove logs/debugs of error
2016-06-28 08:51:57 +01:00
b1f131964e
Rename NewFsObject to NewObject
2016-06-28 08:51:57 +01:00
1a87b69376
Get rid of LimitedFs - FIXME needs docs on copying single files
...
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.
Use this to remove the LimitedFs and just add this file to the
--files-from list.
This means that server side operations can be used also.
Fixes #518
Fixes #545
2016-06-28 08:51:43 +01:00
2cfb3834f2
Log errors with %v
2016-06-18 09:36:47 +01:00
df1092ef33
Change Fs.Put so that it must cope with existing files
...
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.
None of the other Fses needed adjusting.
Fixes #483
2016-06-13 19:29:10 +01:00
4c5b2833b3
Convert to using github.com/pkg/errors everywhere
2016-06-13 17:43:03 +01:00
1fce83b936
swift: add auth version parameter - fixes #407
2016-06-03 17:52:24 +01:00
c2d0e86431
Add more tests for List() and fix resulting problems
2016-05-07 14:50:35 +01:00
68ec6a9f5b
Add a directory parameter to Fs.List()
2016-05-06 16:52:34 +01:00
753b0717be
Refactor the List and ListDir interface
...
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.
Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.
This commit contains some code originally by Klaus Post.
Fixes #316
2016-05-06 16:52:34 +01:00
640d7bd365
Add domain option for openstack (v3 auth)
2016-04-22 21:00:54 +01:00
02c11dd4a7
Don't de-reference swift connection
...
The connection object contains a mutex, so it is good practice not to dereference it to a value.
Reported by Go tip "go vet".
2016-03-23 17:09:05 +00:00
414b35ea56
Change the interface of SetModTime to return an error - #348
2016-03-22 15:56:44 +00:00
bd27473762
swift: Don't return an MD5SUM for static large objects - #392
...
* rename isManifest to isDynamicLargeObject for clarity
2016-03-17 17:36:20 +00:00
9c263e3e2b
Commit missing tests
2016-02-28 20:25:51 +00:00
e6f340d245
swift: Fix uploading of chunked files with non ascii characters - fixes #350
2016-02-27 18:59:16 +00:00
e08e35984c
Add help to remote chooser in rclone config - fixes #43
2016-02-21 13:40:16 +00:00
ef06371c93
Create separate interface for object information.
...
Take out read-only information about a Fs in a separate struct to limit access.
See discussion at #282 .
2016-02-19 13:31:09 +00:00
c2e8f06bfa
Swift storageUrl overloading fixes #167
2016-02-09 22:17:13 +00:00
c3a0c0c451
swift: Fix upload from unprivileged user - fixes #273
2016-01-23 20:32:53 +00:00
ccba859812
Test all available hashes for each remote
2016-01-23 09:10:36 +00:00
78c3a5ccfa
Add support for multiple hash types.
...
Add support for multiple hash types with negotiation of common hash types for comparison.
Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
e7b7432079
OVH Swift authentication enpoint
2016-01-08 08:30:13 +00:00
8369b5209f
swift: Make sure we read the size for 0 length files - Fixes #237
...
This was causing a problem with sync for chunked files. The directory
listing would read their size back as 0 and see that the size had
changed and immediately resync it.
2015-12-17 13:30:58 +00:00
fcea3777c0
Implement Hubic storage system - fixes #200
2015-11-14 08:08:52 +00:00
e8ba43c479
swift: Use ContentType from Object to avoid lookups in listings - fixes #208
2015-11-11 17:19:57 +00:00
5bb76cc35c
Stop SetModTime losing metadata (eg X-Object-Manifest) - fixes #203
2015-11-11 17:00:08 +00:00
a795d93bc3
swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172
2015-11-07 15:32:40 +00:00
5df04cb763
swift: ignore directory marker objects where appropriate - fixes #190
...
* When creating a LimitedFs
* When calling List() to list files
* In the Storable() method
* Add a Purge() method to delete the directory marker objects too
This is a partial fix for #172
2015-11-07 15:32:11 +00:00
b257de4aba
Be more constistent with naming in remotes
...
* External objects are called Fs and Object
* Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
95fd79faf9
swift: use Content-Length on uploads - fixes #125
2015-10-03 16:04:51 +01:00
b79dc01016
swift: stop chunked operations logging "Failed to read info: Object Not Found"
2015-10-03 16:04:51 +01:00
4a0a42c2f1
swift: large file upload fixes
...
* Read metadata in file listing for 0 length files to fix syncs
* Ignore non-existent files in isManifestFile to fix errors on copy
* remove nsToSwiftFloatString - experiments with the swift program
indicate that it puts a variable number of points after the
decimal, so might as well use the one in the swift library.
* Make sure segments get deleted properly when move from segmented
to non segmented and vice versa
* Use internal list routine to detect errors on listing
* Remove the _segments container if possible
* Remove manifest first when deleting
2015-09-26 17:58:04 +01:00
cc7b9af50e
swift: support files > 5GB - fixes #46
...
* Write segments to ..._segments container
* Choice of container and segment names compatible with swift tool
* See http://docs.openstack.org/developer/swift/overview_large_objects.html
* Controlled by command line flag --swift-chunk-size
* Segments removed on delete
2015-09-26 13:03:58 +01:00
68fef49c55
swift: fetch headers as the only source of metadata
2015-09-26 13:03:58 +01:00
e9c915e6fe
Fix golint warnings
2015-09-22 18:47:16 +01:00
e2717a031e
Implement Mover and DirMover interfaces fixes #115
...
* unit tests
* local
* drive
* dropbox
2015-09-01 21:49:13 +01:00
92745aa950
Add Root() to Fs interface
2015-09-01 21:49:13 +01:00
a96b522958
Implement server side copies if possible - fixes #99
...
Add optional fs.Copier interface
Implemented for
* swift
* s3
* drive
* dropbox
* google cloud storage
2015-08-23 21:18:38 +01:00
fedf81c2b7
Add Name() to Fs interface to return name as passed to NewFs
2015-08-23 13:36:38 +01:00
