S3QL

In this page you will find documentation on how to mount SWIFT object storage as a normal file system through s3ql in user space.

S3QL has features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival.

Using s3ql can be mounted as a filesystem using swift’s own native protocol and S3 on linux systems.

Note

Important: Since s3ql only supports keystone V2 authentication, this will only work for users having a local keystone account when mounting using the swift protocol. Such a limitation does not exist using S3.

Installation

Documentation on how to install s3ql can be found at: https://bitbucket.org/nikratio/s3ql/wiki/Home. On linux, your distro often has the s3ql package in their repos, see: https://bitbucket.org/nikratio/s3ql/wiki/Installation. There is also a user guide: http://www.rath.org/s3ql-docs/index.html.

Configuration

  1. Create a file ~/.s3ql/authinfo2 with the following content:

For swift:

[swift]
backend-login: <project name>:<user name>
backend-password: <password>
storage-url: swiftks://proxy.swift.surfsara.nl:5000/RegionOne:<container name>
fs-passphrase: <password>

For S3:

[s3]
backend-login: <access key>
backend-password: <secret keys>
storage-url: s3c://proxy.swift.surfsara.nl:443/<bucket>
fs-passphrase: <password>

By default, the data will be encrypted. The fs-passphrase is the password used to encrypt the randomly generated master key. This makes sure that s3ql encrypts data before it uploads the data to SWIFT.

The <container name> and <bucket> are the names of the container that is to be mounted. They are the same.

Don’t forget:

chmod 600 ~/.s3ql/authinfo2

Mounting the file system

  1. If you haven’t mounted the container before, you first have to create a file system. You can do this, for example, in the following way:
mkfs.s3ql swiftks://proxy.swift.surfsara.nl:5000/RegionOne:<container name>

If you don’t want to encrypt the file system you can use the commandline flag –plain, as in:

mkfs.s3ql --plain swiftks://proxy.swift.surfsara.nl:5000/RegionOne:<container name>
  1. Create a directory that is to be mounted
mkdir /path/to/mount
  1. Mount the SWIFT object storage

For SWIFT:

mount.s3ql swiftks://proxy.swift.surfsara.nl:5000/RegionOne:<container name> /path/to/mount

For S3:

mount.s3ql s3c://proxy.swift.surfsara.nl:443/<bucket> /path/to/mount

By default, the data will be compressed. If you do not want this then you have to use the commandline flag *–compress none* when mounting the storage.

Unmounting the file system

Unmounting the file system is done by:

umount.s3ql /path/to/mount