142 Commits

Author	SHA1	Message	Date
albertony	00aafc957e	sftp: add rclone to list of supported md5sum/sha1sum commands to look for ... See #5781	2021-11-11 15:16:45 +01:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
albertony	f589dbc077	docs: don't show fictional example values for options as multiple choice items ... See #5538	2021-10-20 22:56:19 +02:00
albertony	e2f47ecdeb	docs: punctuation cleanup ... See #5538	2021-10-20 22:56:19 +02:00
Nick Craig-Wood	96e14bf456	sftp: fix initialization bug introduced by fs.ErrorIsDir return ... 3fbaa4c0b0e8076b backends: make NewObject return fs.ErrorIsDir if possible	2021-10-16 10:38:24 +01:00
Nick Craig-Wood	93d85015af	sftp: fix timeout when doing MD5SUM of large file ... Before this change we were timing out MD5SUMs after 1 minute because rclone was closing the SSH session when there were sessions still aftive. This change counts sessions active for all SSH sessions now (Upload, Download, Hashes and running commands). See: https://forum.rclone.org/t/while-rclone-copying-large-files-md5sum-failed-with-exit-status/26845/	2021-10-15 15:19:22 +01:00
Nick Craig-Wood	3fbaa4c0b0	backends: make NewObject return fs.ErrorIsDir if possible ... This changes the interface to NewObject so that if NewObject is called on a directory then it should return fs.ErrorIsDir if possible without doing any extra work, otherwise fs.ErrorObjectNotFound. Tested on integration test server with: go run integration-test.go -tests backend -run TestIntegration/FsMkdir/FsPutFiles/FsNewObjectDir -branch fix-stat -maxtries 1	2021-10-14 17:15:50 +01:00
Ivan Andreev	24de896df2	build: apply gofmt from golang 1.17	2021-09-09 20:43:59 +03:00
Nick Craig-Wood	6545755758	sftp: remove spurious error message on --sftp-disable-concurrent-reads	2021-07-31 11:04:45 +01:00
albertony	ce83228cb2	sftp: expand tilde and environment variables in configured known_hosts_file (#5322) ... Fixes #5220	2021-05-11 19:58:26 +02:00
Nick Craig-Wood	0537791d14	sftp: Fix performance regression by re-enabling concurrent writes #5197 ... Betweeen rclone v1.54 and v1.55 there was an approx 3x performance regression when transferring to distant SFTP servers (in particular rsync.net). This turned out to be due to the library github.com/pkg/sftp rclone uses. Concurrent writes used to be enabled in this library by default (for v1.12.0 as used in rclone v1.54) but they are no longer enabled (for v1.13.0 as used in rclone v1.55) for safety reasons and it is necessary to enable them specifically. The safety concerns are due to the uncertainty as to whether writes come in order and whether a half completed file might have holes in it. This isn't a problem for rclone since a) it doesn't restart uploads and b) it has a post-transfer checksum test. This change introduces a new flag `--sftp-disable-concurrent-writes` to control the feature which defaults to false, meaning that concurrent writes are enabled as in v1.54. However this isn't quite enough to fix the problem as the sftp library needs to be able to sniff the size of the stream from the reader passed in, so this also adds a `Size` interface to the reader to enable this. This involved a patch to the library. The library was reverted to v1.12.0 for v1.55.1 - this patch installs v1.13.0+master to fix the Size interface problem. See: https://github.com/pkg/sftp/issues/426	2021-04-26 09:24:28 +01:00
Nick Craig-Wood	7dc7c021db	sftp: fix Update ReadFrom failed: failed to send packet: EOF errors ... In a3fcadddc871e646 sftp: close idle connections after --sftp-idle-timeout (1m by default) Idle SFTP connections were closed after 1 minute. However due to the way SSH multiplexes connections over a single SSH connection this meant that if uploads or downloads went on for more than one minute they failed with "EOF errors" as their underlying connection was closed. This fixes the problem by not clearing idle connections if there are any transfers in progress. Fixes #5197	2021-04-06 10:01:49 +01:00
Nick Craig-Wood	fe1aa13069	sftp: revert sftp library to v1.12.0 from v1.13.0 to fix performance regression #5197 ... This reverts the library update done in this commit. 713f8f357dc5eb51 sftp: fix "file not found" errors for read once servers Reverting this commit triples the performance to a far away sftp server. See: https://github.com/pkg/sftp/issues/426	2021-04-06 10:01:49 +01:00
Nick Craig-Wood	9ee3ad70e9	sftp: fix SetModTime stat failed: object not found with --sftp-set-modtime=false ... Some sftp servers don't allow the user to access the file after upload. In this case the error message indicates that using --sftp-set-modtime=false would fix the problem. However it doesn't because SetModTime does a stat call which can't be disabled. Update SetModTime failed: SetModTime stat failed: object not found After upload this patch checks for an `object not found` error if set_modtime == false and ignores it, returning the expected size of the object instead. It also makes SetModTime do nothing if set_modtime = false https://forum.rclone.org/t/sftp-update-setmodtime-failed/22873	2021-03-18 16:31:51 +00:00
Nick Craig-Wood	713f8f357d	sftp: fix "file not found" errors for read once servers - fixes #5077 ... It introduces a new flag --sftp-disable-concurrent-reads to stop the problematic behaviour in the SFTP library for read-once servers. This upgrades the sftp library to v1.13.0 which has the fix.	2021-03-13 15:38:38 +00:00
Nick Craig-Wood	a3fcadddc8	sftp: close idle connections after --sftp-idle-timeout (1m by default) ... This fixes a problem where sftp backends live on forever when using the rc and use more and more connections. Fixes #4883	2021-02-16 12:39:05 +00:00
Nick Craig-Wood	e6a9f005d6	sftp: implement --sftp-use-fstat ... See: https://forum.rclone.org/t/sftp-fails-to-sync-to-local-failed-to-copy-file-does-not-exist/21759	2021-01-28 16:07:26 +00:00
Nick Craig-Wood	e775328523	ftp,sftp: Make --tpslimit apply - fixes #4906	2021-01-08 10:29:57 +00:00
Nick Craig-Wood	ba51409c3c	sftp: implement keyboard interactive authentication - fixes #4177 ... Some ssh servers are set up with keyboard interactive authentication which previously the sftp backkend was ignoring.	2020-12-29 19:48:09 +00:00
Nick Craig-Wood	bcbe393af3	sftp: implement Shutdown method	2020-11-27 17:35:01 +00:00
Nick Craig-Wood	9d574c0d63	fshttp: read config from ctx not passed in ConfigInfo #4685	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Josh Soref	3e1cb8302a	docs: spelling: etc. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Stephen Harris	bbddadbd04	sftp: remember entered password in AskPass mode ... As reported in https://github.com/rclone/rclone/issues/4660#issuecomment-705502792 After switching to a password callback function, if the ssh connection aborts and needs to be reconnected then the user is-reprompted for their password. Instead we now remember the password they entered and just give that back. We do lose the ability for them to correct mistakes, but that's the situation from before switching to callbacks. We keep the benefits of not asking for passwords until the SSH connection succeeds (right known_hosts entry, for example). This required a small refactor of how `f := &Fs{}` was built, so we can store the saved password in the Fs object	2020-10-13 16:53:11 +01:00
Stephen Harris	9e925becb6	sftp: defer asking for user passwords until the SSH connection succeeds ... Issue: 4660 https://github.com/rclone/rclone/issues/4660 Unexpected side effect: a wrong password allows for the user to retry!	2020-10-07 12:01:17 +01:00
Stephen Harris	6dc28ef50a	sftp: Allow user to optionally check server hosts key to add security ... Based on Issue 4087 https://github.com/rclone/rclone/issues/4087 Current behaviour is insecure. If the user specifies this value then we switch to validating the server hostkey and so can detect server changes or MITM-type attacks.	2020-10-06 16:27:42 +01:00
Nick Craig-Wood	41ec712aa9	ftp,sftp: fix docs for usernames ... - factor env.CurrentUser out of backend/sftp - Use env.CurrentUser in ftp and sftp - fix docs to have correct username	2020-09-27 11:44:05 +01:00
Stephen Harris	17acae2b00	sftp: allow cert based auth via optional pubkey ... Discussion at https://forum.rclone.org/t/ssh-certificate-based-authentication-does-not-work/19222 Basically we allow the user to specify their own public key cert rather than letting the SSH client extract the pubkey from the private key. This allows certificate based authentication to work.	2020-09-27 11:10:13 +01:00
buengese	60cc2cba1f	sftp: always convert the checksum to lower case - fixes #4518	2020-09-22 03:15:09 +02:00
aus	b6d3cad70e	sftp: add options for subsystem and server_command - fixes #1801	2020-08-25 21:38:13 +01:00
Nick Craig-Wood	fefcbf60fa	sftp: use the absolute path instead of the relative path ... Before this change rclone used the relative path from the current working directory. It appears that WS FTP doesn't like this and the openssh sftp tool also uses absolute paths which is a good reason for switching to absolute paths. This change reads the current working directory at startup and bases all file requests from there. See: https://forum.rclone.org/t/sftp-ssh-fx-failure-directory-not-found/17436	2020-06-30 16:07:23 +01:00
Nick Craig-Wood	c820576329	fs: define SlowModTime and SlowHash features in the relevant backends	2020-06-30 12:01:36 +01:00
Zac Rubin	f9306218f8	sftp: Fix SSH key PEM loading ... For SSH authentication, `key_pem` should both override `key_file` and not require other SSH authentication methods to be set. Prior to this fix, rclone would attempt to use an ssh-agent when `key_pem` was the only SSH authentication method set. Fixes #4240	2020-06-12 22:46:33 +01:00
Nick Craig-Wood	973e3d6a7b	backends: make sure backends expand ~ and environment vars in file names they use ... See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693	2020-06-03 17:39:08 +01:00
Martin Michlmayr	041b201abd	doc: fix typos throughout docs and code	2020-05-25 11:23:58 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
calisro	bcbfad1482	sft[: added --sftp-pem-key to support inline key files	2020-05-19 11:55:38 +01:00
Nick Craig-Wood	e4f1e19127	sftp: fix post transfer copies failing with 0 size when using set_modtime=false ... Before this change we early exited the SetModTime call which means we skipped reading the info about the file. This change reads info about the file in the SetModTime call even if we are skipping setting the modtime. See: https://forum.rclone.org/t/sftp-and-set-modtime-false-error/16362	2020-05-14 17:30:01 +01:00
Nick Craig-Wood	db39adeb3e	sftp: open files for update write only to fix AWS SFTP interop - fixes #3776	2020-01-18 11:46:56 +00:00
Nick Craig-Wood	5f822f2660	sftp: fix "failed to parse private key file: ssh: not an encrypted key" error ... This error started happening after updating golang/x/crypto which was done as a side effect of: 3801b8109 vendor: update termbox-go to fix ncdu command on FreeBSD This turned out to be a deliberate policy of making ssh.ParsePrivateKeyWithPassphrase fail if the passphrase was empty. See: https://go-review.googlesource.com/c/crypto/+/207599 This fix calls ssh.ParsePrivateKey if the passphrase is empty and ssh.ParsePrivateKeyWithPassphrase otherwise which fixes the problem.	2020-01-13 11:05:16 +00:00
Nick Craig-Wood	a7d65bd519	sftp: add --sftp-skip-links to skip symlinks and non regular files - fixes #3716 ... This also corrects the symlink detection logic to only check symlink files. Previous to this it was checking all directories too which was making it do more stat calls than was necessary.	2019-11-24 16:10:53 +00:00
Sebastian Brandt	f158a398f3	sftp: Retry Creation of Connection - fixes #3656 ... Removes the existing rate limiter because it is implicitly included in the pacer.	2019-11-14 12:50:01 +00:00
Carlos Ferreyra	9cb549a227	sftp: include more ciphers with use_insecure_cipher	2019-10-17 14:58:31 +01:00
Nick Craig-Wood	b90e4a8769	sftp: fix hashes of files with backslashes	2019-09-30 22:00:24 +01:00
Nick Craig-Wood	3cff258577	sftp: fix --sftp-ask-password trying to contact the ssh agent ... See: https://forum.rclone.org/t/rclone-command-line/11766	2019-09-16 11:16:27 +01:00
Nick Craig-Wood	fa539b9d9b	sftp: save the md5/sha1 command in use to the config file	2019-08-14 20:54:08 +01:00