MINIO + RCLONE

For OpenSim, it may not matter too much that by default MINIO isn't encrypted and is plain http. After all, in the majority of cases OpenSim itself is not encrypted either - so does it matter that commands to your asset storage are not encrypted? I would say yes, but even more so if you are planning to use object storage for anything else. Myself, I will be using it for backups of my virtual machines from another site. I definitely want those to be encrypted in transit.

Looking at the documentation, this should not be too hard. Put the certs in a .minio/certs folder of the user that runs minio and restart minio. Now it will require the addresses are accessed via https://. Brilliant.

After having a few issues getting this working with a TLS cert - in particular a very helpful "Input/Output" error when trying to read mounted object storage, I thought I would write it down in case it helps anyone else - and also to look back on myself!

Problem number one: I go to my admin interface and go to log in. Oh... big red error at the top of the screen. Basically cant continue as the IP address isnt in the SAN of the certificate. While you CAN add IPs to the SAN of a cert - this isnt currently possible with lets encrypt. It shouldn't be needed anyway - I mean why is it even USING the IP?

Answer is in the /etc/default/minio file:

# Volume to be used for MinIO server.

MINIO_VOLUMES="/storage/minio/"

# Use if you want to run MinIO on a custom port.

MINIO_OPTS="--address :9199 --console-address :9001"

# Root user for the server.

MINIO_ROOT_USER=minio-admin-user

# Root secret for the server.

MINIO_ROOT_PASSWORD=minio-admin-password

# set this for MinIO to reload entries with 'mc admin service restart'

MINIO_CONFIG_ENV_FILE=/etc/default/minio

The issue here seem to be the --address lines - without specifying the hostname they use an IP address it seems. Change this then to something like:

MINIO_OPTS="--address your.fully.qualified.domain.name:9199 --console-address your.fully.qualified.domain.name:9001"

assuming you have installed minio as a systemd process, restart minio with

systemctl restart minio

Then I tried to log back in to my admin interface, and it worked.

Thinking this was now working, I moved onto my remove machines and noticed that I was getting errors accessing the mounted storage. Ok, that made sense. I had mounted them before they were https, so I unmounted them and went to edit the ./config/rclone/rclone.conf file:

[minioalias]

type = s3

env_auth = false

access_key_id = username

secret_access_key = password

region = us-east-1

endpoint = http://192.168.0.222:9199

location_constraint =

server_side_encryption =

Clearly here the http needs changing - but also the up address needs to be changed to the fully qualified domain name.

endpoint = https://your.fully.qualified.domain.name:9199

NOW I tried to mount these again with:

rclone mount minioalias:bucket /mnt/point/ --daemon --allow-other --vfs-read-chunk-size off --transfers=80 --fast-list --allow-non-empty

Finally this managed to mount and could access without error - all encrypted with TLS. Note that the --allow-non-empty at the end of the rclone is not often needed - but when this is mounted by ProxMox, it tends to put its directory structure in the mounted storage before it rclone mounts the storage, so it has empty directories like dump, snippets etc before you mount and that will give errors unless you speficy --allow-non-empty.

I am not 100% sure if this was needed - as I got so many errors I was a little tangled up - but SHOULD be done regardless imo - so the last thing was using mcli on the minio server to create the alias. Note that most of the documentation calls this command "mc" However a lot of people already HAVE a command called "mc" (Midnight Commander) which I use extensively. It seems that this command is now called mcli which makes sense. You can use this to see the current aliases set up:

mcli alias list

minioalias

URL : http://127.0.0.1:9001

AccessKey : username

SecretKey : password

API : s3v4

Path : auto

The access key/secret key need to have been set up as a user in MINIO, with permissions to read/write (probably) your object store. Also notice the URL is an IP assress and http:// Change this as:

URL : https://your.fully.qualified.domain.name:9001

I now have a remote object storage to back up thinks like my virtual machines to from my main ProxMox cluster, and unlike my "CrashPlan Unlimited" it doesn't take a 3 days to upload each day's backups.

I will also be able to move my fsassets to this storage too. I need to do some more testing to see what happens when I break it - and if I can "put humpy back together again" if I do!