About me: My name is Solène Rapenne. I like learning and sharing knowledge. Hobbies: '(BSD OpenBSD h+ Lisp cmdline gaming internet-stuff). I love percent and lambda characters. OpenBSD developer solene@.

Contact me: solene on Freenode, solene+www at dataswamp dot org or solene@bsd.network (mastodon). If for some reason you want to give me some money, I accept paypal at the address donate@perso.pw.

Port of the week: rclone

Written by Solène, on 28 October 2020.
Tags: #portoftheweek

Comments on Mastodon

New Port of the Week after 3 years! I never thought it was so long since last blog post about slrn.

This post is about the awesome rclone program, written in Go and available on most popular platforms (including OpenBSD!). I will explain how to configure it from the interactive command, from file and what you can do with rclone.

rclone can be see as a rsync on steroids which supports lot of Cloud backend and also support creating an encrypted data repository over any backend (local file, ftp, sftp, webdav, Dropbox, AWS S3, etc…).

It’s not a automatic synchronization tool or a backup software. It can copy files from A to B, synchronize two places (can be harmful if you don’t pay attention).

Let’s see how to use it with an ssh server on which we will create an encrypted repository to store important data.

Official documentation

Installation

Most of the time, run your package manager to install rclone. It’s a single binary.

Interactive configuration

You can skip this LONG section if you want to learn what rclone can do and how to configure it in a 10 lines files.

There is a parameter to have a question / answer interface to configure your repository, using rclone config.

I’ll make a full walkthrough to enable an encrypted repository because I struggled to understand the logic behind rclone when I started using it.

Let’s start. I’ll create an encrypted destination on my local NAS which doesn’t have full disk encryption, so anyone who access the system won’t be able to read my data. First, this will require to set up an sftp repository and then an encrypted repository using the previous one as a backend.

Let’s create a new config named home_nas.

$ rclone config
2020/10/27 21:30:48 NOTICE: Config file "/home/solene/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> home_nas

We want the storage type 29, “SSH/SFTP” (I removed all 50+ others storages for readability).

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
29 / SSH/SFTP Connection
   \ "sftp"
[...]
Storage> 29

My host is 192.168.1.200

** See help for sftp backend at: https://rclone.org/sftp/ **

SSH host to connect to
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Connect to example.com
   \ "example.com"
host> 192.168.1.200

I will connect with the username solene.

SSH username, leave blank for current username, solene
Enter a string value. Press Enter for the default ("").
user> solene

Standard port 22, which is the default

SSH port, leave blank to use default (22)
Enter a string value. Press Enter for the default ("").
port> 

I answer n because I want rclone to use ssh agent, this could be the ssh password to the remote user, but I highly discourage everyone from using password authentication on SSH!

SSH password, leave blank to use ssh-agent.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> n

Leave this except if you want to provide a private key.

Raw PEM-encoded private key, If specified, will override key_file parameter.
Enter a string value. Press Enter for the default ("").
key_pem> 

Leave this except if you want to provide a PEM-encoded private key.

Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").
key_file> 

Leave this except if you need to use a password to unlock your private key. I use ssh agent so I don’t need it.

The passphrase to decrypt the PEM-encoded private key file.

Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keys
in the new OpenSSH format can't be used.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> n

If your user agent manage multiples keys, you should enter the correct value here, I only have one key so I leave it empty.

When set forces the usage of the ssh-agent.

When key-file is also set, the ".pub" file of the specified key-file is read and only the associated key is
requested from the ssh-agent. This allows to avoid `Too many authentication failures for *username*` errors
when the ssh-agent contains many keys.
Enter a boolean value (true or false). Press Enter for the default ("false").
key_use_agent> 

This is a question about crypto, accept the default except if you have to connect to old servers.

Enable the use of insecure ciphers and key exchange methods. 

This enables the use of the following insecure ciphers and key exchange methods:

- aes128-cbc
- aes192-cbc
- aes256-cbc
- 3des-cbc
- diffie-hellman-group-exchange-sha256
- diffie-hellman-group-exchange-sha1

Those algorithms are insecure and may allow plaintext data to be recovered by an attacker.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Use default Cipher list.
   \ "false"
 2 / Enables the use of the aes128-cbc cipher and diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1 key exchange.
   \ "true"
use_insecure_cipher> 

We want to keep hashcheck feature so just skip the answer to keep the default value.

Disable the execution of SSH commands to determine if remote file hashing is available.
Leave blank or set to false to enable hashing (recommended), set to true to disable hashing.
Enter a boolean value (true or false). Press Enter for the default ("false").
disable_hashcheck> 

We are at the end of the configuration, we are proposed to change more parameters but we don’t need to.

Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n

Now we can see the output of the configuration file of rclone in regards to my home_nas destination. I agree with the configuration to continue.

Remote config
--------------------
[home_nas]
type = sftp
host = 192.168.1.200
user = solene
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Here is a summary of the configuration, we have only one remote here.

Current remotes:

Name                 Type
====                 ====
home_nas             sftp

In the menu, I will choose to add another remote. Let’s name it home_nas_encrypted

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> home_nas_encrypted

We will choose the special storage crypt which work on an existing backend.

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
10 / Encrypt/Decrypt a remote
   \ "crypt"
Storage> 10

To this question, we will define we want the data stored to home_nas_encrypted being saved in home_nas remote in the encrypted_repo directory.

** See help for crypt backend at: https://rclone.org/crypt/ **

Remote to encrypt/decrypt.
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> home_nas:encrypted_repo

Depending on the level of obfuscation your choice may vary. The simple filename obfuscation is fine for me.

How to encrypt the filenames.
Enter a string value. Press Enter for the default ("standard").
Choose a number from below, or type in your own value
 1 / Encrypt the filenames see the docs for the details.
   \ "standard"
 2 / Very simple filename obfuscation.
   \ "obfuscate"
 3 / Don't encrypt the file names.  Adds a ".bin" extension only.
   \ "off"
filename_encryption> 2

As for the directory names obfuscation, I recommend to enable it, otherwise that leave the whole directory tree readable!

Option to either encrypt directory names or leave them intact.

NB If filename_encryption is "off" then this option will do nothing.
Enter a boolean value (true or false). Press Enter for the default ("true").
Choose a number from below, or type in your own value
 1 / Encrypt directory names.
   \ "true"
 2 / Don't encrypt directory names, leave them intact.
   \ "false"
directory_name_encryption> 1

Type the password that will be used to encrypt the data.

Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:

You can add a salt to the passphrase, I choose not too.

Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> 

No need to change advanced parameters.

Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n

Here is a summary of the configuration of this remote backend. I’m fine with it.

Remote config
--------------------
[home_nas_encrypted]
type = crypt
remote = home_nas:encrypted_repo
directory_name_encryption = true
password = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

We see we have now two remote backends, one with the crypt type.

Current remotes:

Name                 Type
====                 ====
home_nas             sftp
home_nas_encrypted   crypt

Quit rclone, the configuration is done.

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Configuration file

The previous configuration process only produced this short configuration file, so you may copy/paste from it and adapt to add more backends if you want, instead of doing the tedious config process.

Here is my file ~/.config/rclone/rclone.conf on my desktop.

[home_nas]
type = sftp
host = 192.168.1.200
user = solene

[home_nas_encrypted]
type = crypt
remote = home_nas:encrypted_repo
directory_name_encryption = true
password = GDS9B1B1LrBa3ltQrSbLf1Vq5C6VbaA1AJVlSZ8

First usage

Now we defined our configuration, we need to create the remote directory that will be used as a backend, this is important to avoid errors when using rclone, this is a simple step required only once.

$ rclone mkdir home_nas_encrypted:

On the remote server, I can see a /home/solene/encryted_repo directory. It’s now ready to use!

A few commands

rclone has a LOT of commands available, I will present a few of them.

Copying files to/from backend

Let’s say I want to copy files to the encrypted repository. There is a copy command.

$ rclone copy /home/solene/log/templates home_nas_encrypted:blog_template  

There are no output by default when the program runs fine. You can use -v flag to have some verbose output (I prefer it).

List files on a remote backend

Now, we want to see if the files were copied correctly, we will use the ls command.

$ rclone ls home_nas_encrypted:
      299 blog_template/article.tpl
      700 blog_template/gopher_head.tpl
     2505 blog_template/layout.tpl
      295 blog_template/more.tpl
      236 blog_template/navigation.tpl
       57 blog_template/one-tag.tpl
       34 blog_template/page.tpl
      189 blog_template/rss-item.tpl
      326 blog_template/rss.tpl

We can also use ncdu to mimic the ncdu program displaying a curses interfaces to visualize disk usage in a nice browsing tree.

$ rclone ncdu home_nas_encrypted
-- home_nas_encrypted: ------------------
  6.379k [##########] /blog_template

The sync command

Files and directories can also be copied with the sync command, but this must be used with care because it makes sure the destination matches exactly the origin when you use it. It’s the equivalent of rsync -a --delete origin/ destination/, so any extra files will be removed! Note that you can use --dry-run to see what will happen.

Filters

When you copy files using the various available method, instead of using a path, you can provide a filter file or a list of paths to transfers. This can be very efficient when you want to recover specifics data.

The documentation about filtering is available here

Parameters

rclone supports a lot of parameters, like to limit upload bandwidth, copy multiples files at once, enable an interactive mode in case of file deletion/overwriting.

Mount

On Linux, FreeBSD and MacOS, rclone can use a FUSE filesystem to mount the remote repository on the filesystem, making its uses totally transparent.

This is extremely useful, avoiding the tediousness of the get/put paradigm of rclone.

This can even be used to make an encrypted repository on the local filesystem! :)

Create a webdav/sftp/ftp server

rclone has the capability of act as a server and expose a configured remote backend on various network protocol like webdav, sftp, ftp, s3 (minio) !

The serv document is available here

Example running a simple webdav server with hardcoded login/password:

$ rclone serv webdav --user solene --password ANicePassword home_nas_encrypted: