New Port of the Week after 3 years! I never thought it was so long
since last blog post about slrn.
This post is about the awesome rclone program, written in Go and
available on most popular platforms (including OpenBSD!). I will
explain how to configure it from the interactive command, from file
and what you can do with rclone.
rclone can be see as a rsync on steroids which supports lot of
Cloud backend and also support creating an encrypted data repository
over any backend (local file, ftp, sftp, webdav, Dropbox, AWS S3,
etc…).
It’s not a automatic synchronization tool or a backup
software. It can copy files from A to B, synchronize two places
(can be harmful if you don’t pay attention).
Let’s see how to use it with an ssh server on which we will
create an encrypted repository to store important data.
Official documentation
Installation
Most of the time, run your package manager to install rclone.
It’s a single binary.
Interactive configuration
You can skip this LONG section if you want to learn what rclone
can do and how to configure it in a 10 lines files.
There is a parameter to have a question / answer interface to
configure your repository, using rclone config.
I’ll make a full walkthrough to enable an encrypted repository
because I struggled to understand the logic behind rclone when I
started using it.
Let’s start. I’ll create an encrypted destination on my local NAS
which doesn’t have full disk encryption, so anyone who access the
system won’t be able to read my data. First, this will require to
set up an sftp repository and then an encrypted repository using the
previous one as a backend.
Let’s create a new config named home_nas.
$ rclone config
2020/10/27 21:30:48 NOTICE: Config file "/home/solene/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> home_nas
We want the storage type 29, “SSH/SFTP” (I removed all 50+ others
storages for readability).
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
29 / SSH/SFTP Connection
\ "sftp"
[...]
Storage> 29
My host is 192.168.1.200
** See help for sftp backend at: https://rclone.org/sftp/ **
SSH host to connect to
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Connect to example.com
\ "example.com"
host> 192.168.1.200
I will connect with the username solene.
SSH username, leave blank for current username, solene
Enter a string value. Press Enter for the default ("").
user> solene
Standard port 22, which is the default
SSH port, leave blank to use default (22)
Enter a string value. Press Enter for the default ("").
port>
I answer n because I want rclone to use ssh agent, this could
be the ssh password to the remote user, but I highly discourage
everyone from using password authentication on SSH!
SSH password, leave blank to use ssh-agent.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> n
Leave this except if you want to provide a private key.
Raw PEM-encoded private key, If specified, will override key_file parameter.
Enter a string value. Press Enter for the default ("").
key_pem>
Leave this except if you want to provide a PEM-encoded private key.
Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a string value. Press Enter for the default ("").
key_file>
Leave this except if you need to use a password to unlock your
private key. I use ssh agent so I don’t need it.
The passphrase to decrypt the PEM-encoded private key file.
Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keys
in the new OpenSSH format can't be used.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> n
If your user agent manage multiples keys, you should enter the
correct value here, I only have one key so I leave it empty.
When set forces the usage of the ssh-agent.
When key-file is also set, the ".pub" file of the specified key-file is read and only the associated key is
requested from the ssh-agent. This allows to avoid `Too many authentication failures for *username*` errors
when the ssh-agent contains many keys.
Enter a boolean value (true or false). Press Enter for the default ("false").
key_use_agent>
This is a question about crypto, accept the default except if you
have to connect to old servers.
Enable the use of insecure ciphers and key exchange methods.
This enables the use of the following insecure ciphers and key exchange methods:
- aes128-cbc
- aes192-cbc
- aes256-cbc
- 3des-cbc
- diffie-hellman-group-exchange-sha256
- diffie-hellman-group-exchange-sha1
Those algorithms are insecure and may allow plaintext data to be recovered by an attacker.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
1 / Use default Cipher list.
\ "false"
2 / Enables the use of the aes128-cbc cipher and diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1 key exchange.
\ "true"
use_insecure_cipher>
We want to keep hashcheck feature so just skip the answer to keep
the default value.
Disable the execution of SSH commands to determine if remote file hashing is available.
Leave blank or set to false to enable hashing (recommended), set to true to disable hashing.
Enter a boolean value (true or false). Press Enter for the default ("false").
disable_hashcheck>
We are at the end of the configuration, we are proposed to change
more parameters but we don’t need to.
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Now we can see the output of the configuration file of rclone in
regards to my home_nas destination. I agree with the configuration
to continue.
Remote config
--------------------
[home_nas]
type = sftp
host = 192.168.1.200
user = solene
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Here is a summary of the configuration, we have only one remote
here.
Current remotes:
Name Type
==== ====
home_nas sftp
In the menu, I will choose to add another remote. Let’s name it
home_nas_encrypted
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> home_nas_encrypted
We will choose the special storage crypt which work on an existing
backend.
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
10 / Encrypt/Decrypt a remote
\ "crypt"
Storage> 10
To this question, we will define we want the data stored to
home_nas_encrypted being saved in home_nas remote in the
encrypted_repo directory.
** See help for crypt backend at: https://rclone.org/crypt/ **
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> home_nas:encrypted_repo
Depending on the level of obfuscation your choice may vary. The
simple filename obfuscation is fine for me.
How to encrypt the filenames.
Enter a string value. Press Enter for the default ("standard").
Choose a number from below, or type in your own value
1 / Encrypt the filenames see the docs for the details.
\ "standard"
2 / Very simple filename obfuscation.
\ "obfuscate"
3 / Don't encrypt the file names. Adds a ".bin" extension only.
\ "off"
filename_encryption> 2
As for the directory names obfuscation, I recommend to enable it,
otherwise that leave the whole directory tree readable!
Option to either encrypt directory names or leave them intact.
NB If filename_encryption is "off" then this option will do nothing.
Enter a boolean value (true or false). Press Enter for the default ("true").
Choose a number from below, or type in your own value
1 / Encrypt directory names.
\ "true"
2 / Don't encrypt directory names, leave them intact.
\ "false"
directory_name_encryption> 1
Type the password that will be used to encrypt the data.
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:
You can add a salt to the passphrase, I choose not too.
Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n>
No need to change advanced parameters.
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Here is a summary of the configuration of this remote backend.
I’m fine with it.
Remote config
--------------------
[home_nas_encrypted]
type = crypt
remote = home_nas:encrypted_repo
directory_name_encryption = true
password = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
We see we have now two remote backends, one with the crypt type.
Current remotes:
Name Type
==== ====
home_nas sftp
home_nas_encrypted crypt
Quit rclone, the configuration is done.
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
Configuration file
The previous configuration process only produced this short
configuration file, so you may copy/paste from it and adapt to add
more backends if you want, instead of doing the tedious config
process.
Here is my file ~/.config/rclone/rclone.conf on my desktop.
[home_nas]
type = sftp
host = 192.168.1.200
user = solene
[home_nas_encrypted]
type = crypt
remote = home_nas:encrypted_repo
directory_name_encryption = true
password = GDS9B1B1LrBa3ltQrSbLf1Vq5C6VbaA1AJVlSZ8
First usage
Now we defined our configuration, we need to create the remote
directory that will be used as a backend, this is important to avoid
errors when using rclone, this is a simple step required only once.
$ rclone mkdir home_nas_encrypted:
On the remote server, I can see a /home/solene/encryted_repo
directory. It’s now ready to use!
A few commands
rclone has a LOT of commands available, I will present a few
of them.
Copying files to/from backend
Let’s say I want to copy files to the encrypted repository. There
is a copy command.
$ rclone copy /home/solene/log/templates home_nas_encrypted:blog_template
There are no output by default when the program runs fine. You can
use -v flag to have some verbose output (I prefer it).
List files on a remote backend
Now, we want to see if the files were copied correctly, we will use
the ls command.
$ rclone ls home_nas_encrypted:
299 blog_template/article.tpl
700 blog_template/gopher_head.tpl
2505 blog_template/layout.tpl
295 blog_template/more.tpl
236 blog_template/navigation.tpl
57 blog_template/one-tag.tpl
34 blog_template/page.tpl
189 blog_template/rss-item.tpl
326 blog_template/rss.tpl
We can also use ncdu to mimic the ncdu program displaying a
curses interfaces to visualize disk usage in a nice browsing tree.
$ rclone ncdu home_nas_encrypted
-- home_nas_encrypted: ------------------
6.379k [##########] /blog_template
The sync command
Files and directories can also be copied with the sync command,
but this must be used with care because it makes sure the destination
matches exactly the origin when you use it. It’s the equivalent of
rsync -a --delete origin/ destination/, so any extra files will
be removed! Note that you can use --dry-run to see what will
happen.
Filters
When you copy files using the various available method, instead of
using a path, you can provide a filter file or a list of paths to
transfers. This can be very efficient when you want to recover
specifics data.
The documentation about
filtering is available here
Parameters
rclone supports a lot of parameters, like to limit upload
bandwidth, copy multiples files at once, enable an interactive mode
in case of file deletion/overwriting.
Mount
On Linux, FreeBSD and MacOS, rclone can use a FUSE filesystem
to mount the remote repository on the filesystem, making its uses
totally transparent.
This is extremely useful, avoiding the tediousness of the get/put
paradigm of rclone.
This can even be used to make an encrypted repository on the local
filesystem! :)
Create a webdav/sftp/ftp server
rclone has the capability of act as a server and expose a
configured remote backend on various network protocol like webdav,
sftp, ftp, s3 (minio) !
The
serv document is available here
Example running a simple webdav server with hardcoded login/password:
$ rclone serv webdav --user solene --password ANicePassword home_nas_encrypted: