Contents

Installation BACK TO TOP

To make all of this work requires a few components:

The installation tutorial, up next, will take you through the steps required to set up a single server backup solution. This is the easiest to set up.

Overview BACK TO TOP

As mentioned above, this tutorial will show you how to set up a single server backup solution. The backup server can be placed in the corporate network or in Amazon EC2.

We will do the installation in two stages. First we will set up the rsync backup server and make sure it works. Then we will modify the configuration for lift-and-shift.

Installation consists of the following broad steps:

We'll start the installation next...

Create the rsync backup server BACK TO TOP

In this section we will cover the first bullet point, creating the rsync backup server.

Create a server and install Obdi BACK TO TOP

Creating a server and installing Obdi is already documented on on the obdi.io main page.

So, head over to the Obdi Installation section, complete the instructions there, then continue from here. The link should open in a new window so that you won't lose your place.

Set up zfs etc. BACK TO TOP

NOTE: A new volume should be created that will be mounted on /backup. It could be an LVM Volume, a physical disk or an Amazon Volume depending on the platform it is running on. This volume will probably be quite large and will be set up with the zfs filesystem.

Steps for Centos 6/7:

Install zfs from The Official ZFS pages.

The following example creates a partition table on /dev/xvdb - a 3TB Amazon volume created and attached by the administrator - then creates the zfs filesystem:

# Become root
sudo su

# Load the module that was built earlier
modprobe zfs

# Make the root of the backup tree
mkdir /backup

# Create a partition on /dev/xvdb1 using parted
echo -e "mktable gpt\nmkpart\n\n\n0%\n100%\nprint\nq" | parted /dev/xvdb

# Create a zpool
zpool create backup /dev/xvdb1

# Switch on deduplication and compression.
# Note: dedup needs a large amount of memory.
zfs create -o dedup=on -o compression=gzip backup/servers-zfs

# Disable atime
zfs set atime=off backup/servers-zfs

# List and check settings
zfs list
zpool list
zfs get dedup
zfs get compression
zfs get atime

Configure the system BACK TO TOP

This section could contain lots of screen shots with instructions telling you what to click on and what fields to type in using the Obdi Admin interface but we won't do that! Instead we'll go straight to using the REST api to demonstrate the versatility gained by using REST for everything. Anything that can be done using the Obdi User Interface can always be done using REST.

The following graphic shows the scenario that we will implement:

screenshot

With this scenario:

Note: You may have installed Obdi in a machine on the corporate network instead. This is fine, and will not affect the configuration, but it will have different performance characteristics (and you might not need the VPN).

Ssh into the machine you made and configuration starts next...

Log in to Obdi

Using the default user name and password log into Obdi. By default, logins timeout after 10 minutes of inactivity, so if that happens just log in again and get a new token.

proto="https"
opts="-k -s" # don't check ssl cert, silent
ipport="127.0.0.1:443"
token=`curl $opts -d '{"Login":"admin","Password":"admin"}' \
    $proto://$ipport/api/login | grep -o "[a-z0-9][^\"]*"`

# View the token (should not be empty!)
echo "token=$token"

Create users

We will create a user to login with, nomen.nescio, and a user that will be used by the worker daemon.

curl $opts -d '{
    "login":"worker",
    "passHash":"w0rker",
    "forename":"Worker",
    "surname":"Daemon",
    "email":"worker@invalid",
    "MultiLogin":true,
    "enabled":true}' "$proto://$ipport/api/admin/$token/users"

curl $opts -d '{
    "login":"nomen.nescio",
    "passHash":"p4ssw0rd",
    "forename":"Nomen",
    "surname":"Nescio",
    "email":"nomen@invalid",
    "enabled":true}' "$proto://$ipport/api/admin/$token/users"

Create a datacentre and environment

Create a datacentre named 'AWS':

curl $opts -d '{
    "SysName":"aws",
    "DispName":"AWS"
    }' "$proto://$ipport/api/admin/$token/dcs"

Create an environment named 'perf' in the AWS datacentre:

# Get the datacentre ID
dcid=`curl $opts "$proto://$ipport/api/admin/$token/dcs?sys_name=aws" \
      | grep Id | grep -o "[0-9]"`

# Create the env
if [[ -z $dcid ]]; then
    echo "dcid should not be empty"
else
  curl $opts -d '{
      "SysName":"perf",
      "DispName":"Performance Servers",
      "DcId":'"$dcid"',
      "WorkerUrl":"https://127.0.0.1:4443/",
      "WorkerKey":"changeme"
  }' "$proto://$ipport/api/admin/$token/envs"
fi

Change the key setting in /etc/obdi-worker/obdi-worker.conf to whatever you set for the WorkerKey property above. Use an editor or use sed. For example, to change the key to 'changeme' using sed:

sed -ri 's/(^key\s*=\s*).*/\1"changeme"/' /etc/obdi-worker/obdi-worker.conf

Assign permissions

Give the 'Enabled' and 'Writeable' permissions to the user, nomen.nescio.

A bash function, 'edit_perm', is defined then used to edit the permission:

# Add the bash function, edit_perm:

edit_perm()
# Parameters:
# $1 - login (text)
# $2 - data centre (text)
# $3 - environment (text)
# $4 - Enabled (true|false)
# $5 - Writeable (true|false)
{
    userid=`curl $opts "$proto://$ipport/api/admin/$token/users?login=$1" | grep Id | grep -o "[0-9]"`
    dcid=`curl $opts "$proto://$ipport/api/admin/$token/dcs?sys_name=$2" | grep Id | grep -o "[0-9]"`
    envid=`curl $opts "$proto://$ipport/api/admin/$token/envs?sys_name=$3&dc_id=$dcid" | grep -w Id | grep -o "[0-9]"`

    echo
    echo "$1 $2 $3 $4 $5"
    echo curl $opts -d '{
        "UserId":'"$userid"',
        "EnvId":'"$envid"',
        "Enabled":true,
        "Writeable":true
    }' "$proto://$ipport/api/admin/$token/perms"

    curl $opts -d '{
        "UserId":'"$userid"',
        "EnvId":'"$envid"',
        "Enabled":'"$4"',
        "Writeable":'"$5"'
    }' "$proto://$ipport/api/admin/$token/perms"
}

# Use edit_perm to change permissions

edit_perm nomen.nescio aws perf true true

Install Obdi plugins: BACK TO TOP

Add the plugin repositories and install the plugins. We'll install the Job Viewer, nettools, and rsyncbackup.

# Obdi core repo

curl $opts -d '{
    "Url":"https://github.com/mclarkson/obdi-core-repository.git"
}' "$proto://$ipport/api/admin/$token/repos"

# systemjobs plugin

curl $opts -d '{
    "Name":"systemjobs"
}' "$proto://$ipport/api/admin/$token/repoplugins"

# nettools repo

curl $opts -d '{
    "Url":"https://github.com/mclarkson/obdi-nettools-repository.git"
}' "$proto://$ipport/api/admin/$token/repos"

# nettools plugin

curl $opts -d '{
    "Name":"nettools"
}' "$proto://$ipport/api/admin/$token/repoplugins"

# rsyncbackup plugin
# This could take quite some time since it compiles 10 Go source files.

curl $opts -d '{
    "Name":"rsyncbackup"
}' "$proto://$ipport/api/admin/$token/repoplugins"

Test the User Interface BACK TO TOP

Connect to the Obdi Admin inteface using a Web Browser to 'https://OBDI_IP/manager/admin' and take a look around, but don't change anything just yet!

Connect to the Obdi Run interface next using, 'https://OBDI_IP/manager/run', and you will see the plugins that have been installed, great! You will also encounter an error message as shown below:

smallscreenshot

The error message is: "Server said: Plugin endpoint 'rsyncbackup/tasks' does not exist. Compile failed for '/usr/lib/obdi/plugins/rsyncbackup/tasks'. System said 'exit status 2'. STDOUT: STDERR: go: cannot find GOROOT directory: /usr/local/go".

The Go (golang) plugins are compiled when they are needed and the error message is due to a misconfiguration. To fix it, the 'go_root' variable needs correcting. The following command will fix this:

# Modify /etc/obdi/obdi.conf

sed -ri 's#(^go_root\s*=\s*).*#\1"'"`go env GOROOT`"'"#' /etc/obdi/obdi.conf

# Restart obdi

service obdi restart 
service obdi-worker restart

Now you will be able to click the 'Add Backup Task' button and start configuring the backup tasks, but first, there is just one more problem to fix, which shows itself when the 'View Files and Snapshots' is clicked, shown next.

smallscreenshot

The error message is: "Server said: Could no send job to worker. ('Login from worker to Manager failed ('User or password error.').')".

We set up a user named 'worker' earlier with a password, 'w0rker'. Naturally all passwords on this page should be changed! This user has no permissions to access any environment and is used to accept status information from one or more workers.

To fix the error just shown we need to edit the /etc/obdi-worker/obdi-worker.conf configuration file. Just the 'man_password' variable needs changing to 'w0rker'. The commands would be:

sed -ri 's/(^man_password\s*=\s*).*/\1"w0rker"/' /etc/obdi-worker/obdi-worker.conf

service obdi-worker restart

Configure the target host and backup task BACK TO TOP

Setting up the Servers that will be Backed Up From

On every server enable rsyncd by ensuring xinetd has disable set to 'no' in /etc/xinetd.d/rsync. For example:

service rsync
{
        disable = no
        flags           = IPv6
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += USERID
}

Also supply an rsyncd configuration. For example, use the following configuration in /etc/rsyncd.conf to be able to do a full system backup:

[backup]
    path = /
    comment = Full system backup
    read only = true
    uid = 0
    gid = 0
    exclude = dev/** proc/** media/** mnt/** selinux/** sys/** tmp/**

That's all that is required on each target server to make sure they're ready for being backed up. Just ensure that the xinetd daemon is installed and running and rsync is installed.

Setting up Obdi Rsyncbackup

From the configuration so far we can use the following settings for a new Backup Task:

Those settings can be entered using the Graphical User Interface (GUI) or by using REST calls. Using REST is shown below, but it is absolutely fine if you use the GUI.

The following commands will create a new Backup Task, 'BACKUP TASK #1', and some settings for that task:

# Login as a normal user, not admin!

proto="https"
opts="-k -s" # don't check ssl cert, silent
ipport="127.0.0.1:443"
token=`curl $opts -d '{"Login":"nomen.nescio","Password":"p4ssw0rd"}' \
    $proto://$ipport/api/login | grep -o "[a-z0-9][^\"]*"`

# View the token (should not be empty!)
echo "token=$token"

# Get the environment ID for our 'perf' environment

t="/tmp/deleteme_$$"

curl $opts "https://$ipport/api/nomen.nescio/$token/envs?sys_name=perf" | tee $t

envid=`grep \"Id\" $t | grep -Eo "[0-9]+"`

# View the envid (should not be empty!)
echo "envid=$envid"

# Create a new backup task

curl $opts -d '{
 "Id":0,
 "TaskDesc":"BACKUP TASK #1",
 "CapTag":"RSYNCBACKUP_WORKER_1"}' \
 "$proto://$ipport/api/nomen.nescio/$token/rsyncbackup/tasks?env_id=$envid" | tee $t

taskid=$(grep -Eo '\\"Id\\":[0-9]+' $t | grep -Eo "[0-9]+")

# View the taskid (should not be empty!)
echo "taskid=$taskid"

# Create a new set of settings

curl $opts -d '{
 "BaseDir":"/backup/servers-zfs/",
 "Id":0,
 "KnownHosts":"",
 "NumPeriods":1,
 "Pre":"create_zfs_snapshot",
 "Protocol":"rsyncd",
 "Repeat":false,
 "RepeatPost":"",
 "RepeatPre":"",
 "RsyncOpts":"--sparse",
 "SshKeyFile":"",
 "SshNotProcs":"",
 "SshSudo":"",
 "SshUid":"",
 "TaskId":1,
 "Timeout":0,
 "Verbose":false}' \
 "$proto://$ipport/api/nomen.nescio/$token/rsyncbackup/settings?env_id=$envid&task_id=$taskid"

All that's left is to create the Includes and Excludes and then we can do a test backup.

Includes and Excludes BACK TO TOP

So, you've got this far - well done! Now it's time to start adding hosts, each with their own includes and excludes (because this is how rsync works).

In this section you'll see how using the REST API will probably save you loads of time, especially if you have a bunch of identically configured servers.

You will find two example scripts in the GitHub repository:

We will use those scripts to backup a single server, then you will be able to add more lines to add more hosts.

So long as the rsyncbackup plugin is installed (which it should be if you followed the instructions thus far), then you can copy the files with the following commands on the Obdi server:

# Go to your home directory:
cd

# Copy the file:
cp -a /usr/share/obdi/static/plugins/rsyncbackup/doc/example_rest_scripts/ .

# The files are now in the 'example_rest_scripts' directory in your home directory.
# You can edit them and run them from there.

Starting with the script, rsync_add_hosts.sh, you will see a few variables defined near the top, they are:

Armed with this information, edit the file, rsync_add_hosts.sh, and set the variables TASKID= and ENVID=. Make sure the user name and password are set correctly for guid=. Finally set the INCL= variable to one or more hosts on your network.

Make the file executable and run it:

chmod +x rsync_add_hosts.sh

./rsync_add_hosts.sh

To check the configuration, log into the Run interface GUI and view the Includes and Excludes.

To test, select one item then click the green 'Backup Selected Now' button.

The job is started and you can view it's progress by clicking on the link that will appear after clicking the button, or, at any other time you can click on System Jobs in the left menu to see the job running, view it's output, or kill the job and all child processes.

The initial rsync will take quite some time. I did mine in small groups until all 80 servers were backed up, then I set up a cron job to run the backup very early each day. Example cron jobs can be found on the main GitHub page . I also set a cron job to only keep the last 30 days worth of daily snapshots.

If you followed the installation using the command line only then you will now be able to put all the commands into one file and set up another server from scratch. This is great for configuration management tools such as Salt Stack, Ansible or Puppet.

In the next section we will configure and set up Lift and Shift...

Enable Lift and Shift BACK TO TOP

Install extra Obdi plugins

We need the GUI plugin, aws-p2ec2, and the Amazon EC2 Obdi Library, aws-ec2lib.

# Log in as the admin user for system changes

proto="https"
opts="-k -s" # don't check ssl cert, silent
ipport="127.0.0.1:443"
token=`curl $opts -d '{"Login":"admin","Password":"admin"}' \
    $proto://$ipport/api/login | grep -o "[a-z0-9][^\"]*"`

# View the token (should not be empty!)
echo "token=$token"

# Add the AWS Tools repository

curl $opts -d '{
    "Url":"https://github.com/mclarkson/obdi-awstools-repository.git"
}' "$proto://$ipport/api/admin/$token/repos"

# Install the aws-ec2lib plugin

curl $opts -d '{
    "Name":"aws-ec2lib"
}' "$proto://$ipport/api/admin/$token/repoplugins"

# Install the aws-p2ec2 plugin

curl $opts -d '{
    "Name":"aws-p2ec2"
}' "$proto://$ipport/api/admin/$token/repoplugins"

Configure the system

When the two extra plugins were installed, they added two capabilities to the system, HAS_AWS_EC2LIB and HAS_AWS_P2EC2. We need to assign those capabilities to our environment so that other plugins know that they are now available.

Once the capabilities have been added, the green button, 'Create Amazon EC2 Instance', will appear in the Directory Browser when the directory looks like a root filesystem.

# Add the capabilities to our environment

# Get the environment ID for our 'perf' environment

t="/tmp/deleteme_$$"

curl $opts "https://$ipport/api/admin/$token/envs?sys_name=perf" | tee $t

envid=`grep \"Id\" $t | grep -Eo "[0-9]+"`

# View the envid (should not be empty!)
echo "envid=$envid"

# Get the environment capability id for HAS_AWS_EC2LIB

envcapid=`curl $opts "https://$ipport/api/admin/$token/envcaps?code=HAS_AWS_EC2LIB" \
      | grep Id | grep -o "[0-9]"`

# Create the environment capability mapping

if [[ -z $envcapid ]]; then
    echo "envcapid should not be empty"
else
  curl $opts -d "{
      \"EnvId\":$envid,
      \"EnvCapId\":$envcapid
  }" "$proto://$ipport/api/admin/$token/envcapmaps"
fi

# Get the environment capability id for HAS_AWS_P2EC2

envcapid=`curl $opts "https://$ipport/api/admin/$token/envcaps?code=HAS_AWS_P2EC2" \
      | grep Id | grep -o "[0-9]"`

# Create the environment capability mapping

if [[ -z $envcapid ]]; then
    echo "envcapid should not be empty"
else
  curl $opts -d "{
      \"EnvId\":$envid,
      \"EnvCapId\":$envcapid
  }" "$proto://$ipport/api/admin/$token/envcapmaps"
fi

If you look at the logs you will see:

smallscreenshot

So we need the user, 'sduser' (the secret data user). Lets add it:

curl $opts -d '{
    "login":"sduser",
    "passHash":"webfkaalwiugfasf",
    "forename":"Secret Data",
    "surname":"User",
    "email":"sduser@invalid",
    "MultiLogin":false,
    "enabled":true}' "$proto://$ipport/api/admin/$token/users"

And finally we need to set the JSON object capability, AWS_ACCESS_KEY_ID_1. The secret data user is required for plugins to access this object securely.

Setting AWS_ACCESS_KEY_ID_1

The JSON object, AWS_ACCESS_KEY_ID_1, has quite a few fields to complete. See the aws-p2ec2 plugin page for a full list and description of all the parameters that need setting.

Copy the following text into a text editor and get ready to enter the correct details in plalce of 'EMPTY' for the parameters. The items already entered should be correct for a single server installation.

{
    \"aws_access_key_id\":\"EMPTY\",
    \"aws_secret_access_key\":\"EMPTY\",
    \"aws_keyname\":\"EMPTY\",
    \"aws_obdi_worker_instance_id\":\"EMPTY\",
    \"aws_obdi_worker_region\":\"EMPTY\",
    \"aws_obdi_worker_url\":\"https://127.0.0.1:4443/\",
    \"aws_obdi_worker_key\":\"changeme\",
    \"aws_obdi_worker_ip\":\"EMPTY\",
    \"aws_dnsserver_ip\":\"EMPTY\",
    \"aws_dnsdomain\":\"EMPTY\",
    \"aws_gateway\":\"EMPTY\",
    \"aws_shell_user_name\":\"EMPTY\",
    \"aws_filter\":\"key-name=EMPTY\",
    \"aws_shell_ssh_key_b64\":\"EMPTY\",
    \"aws_securitygroups\":[\"EMPTY\"]
}

The prior json object text has the speech marks escaped with a backslash for inserting at the command line. If the object is going to be pasted into the Obdi Admin Web interface then the speech marks should not be escaped (omit all of the backslashes).

Quick summary of where to get parameter details:

    aws_access_key_id            <-- from Amazon
    aws_secret_access_key        <-- from Amazon
    aws_keyname                  <-- from Amazon (you made the key probably)
    aws_obdi_worker_instance_id  <-- From Amazon Management Console
    aws_obdi_worker_region       <-- From Amazon Management Console
    aws_obdi_worker_url          <-- From obdi-worker.conf 'listen_address'
    aws_obdi_worker_key          <-- From obdi-worker.conf 'key'
    aws_obdi_worker_ip           <-- From Amazon Management Console
    aws_dnsserver_ip             <-- From the network admin
    aws_dnsdomain                <-- From the network admin
    aws_gateway                  <-- From the network admin
    aws_shell_user_name          <-- Username used to log into the shell
    aws_filter                   <-- You choose
    aws_shell_ssh_key_b64        <-- The ssh private key encoded as base 64 text
    aws_securitygroups           <-- The security groups to apply to instances

Use the summary, above, and the aws-p2ec2 plugin page to work out what to put in the above json object.

The following code will create the AWS_ACCESS_KEY_ID_1 capability:

# Get the environment ID for our 'perf' environment

t="/tmp/deleteme_$$"

curl $opts "https://$ipport/api/admin/$token/envs?sys_name=perf" | tee $t

envid=`grep \"Id\" $t | grep -Eo "[0-9]+"`

# View the envid (should not be empty!)
echo "envid=$envid"

# Get the environment capability id for AWS_ACCESS_KEY_ID_1

envcapid=`curl $opts "https://$ipport/api/admin/$token/envcaps?code=AWS_ACCESS_KEY_ID_1" \
      | grep Id | grep -o "[0-9]"`

# View the envcapid (should not be empty!)
echo "envcapid=$envcapid"

# Create the environment capability mapping

if [[ -z $envcapid ]]; then
    echo "envcapid should not be empty"
else
  curl $opts -d "{
      \"EnvId\":$envid,
      \"EnvCapId\":$envcapid
  }" "$proto://$ipport/api/admin/$token/envcapmaps"
fi

The capability has been created but the JSON Object has not.

Edit the json object text so that it is all on one line then create the object. For example:

# Create the JSON Object

curl -k -d '{
    "EnvId":'"$envid"',
    "EnvCapId":'"$envcapid"',
    "Json":"{ \"aws_access_key_id\":\"EMPTY\", \"aws_secret_access_key\":\"EMPTY\", \"aws_obdi_worker_instance_id\":\"EMPTY\", \"aws_obdi_worker_region\":\"EMPTY\", \"aws_obdi_worker_url\":\"https://127.0.0.1:4443/\", \"aws_obdi_worker_key\":\"changeme\", \"aws_obdi_worker_ip\":\"EMPTY\", \"aws_dnsserver_ip\":\"EMPTY\", \"aws_dnsdomain\":\"EMPTY\", \"aws_gateway\":\"EMPTY\", \"aws_shell_user_name\":\"EMPTY\", \"aws_filter\":\"key-name=EMPTY\", \"aws_shell_ssh_key_b64\":\"EMPTY\", \"aws_securitygroups\":[\"EMPTY\"] }"
}' "https://$ipport/api/admin/$token/jsonobjects"

Now you can test creating Amazon Volumes, Snapshots and Instances.

The installation is complete.

Troubleshooting BACK TO TOP

Problem: sudo: sorry, you must have a tty to run sudo

smallscreenshot

The Obdi server needs: 'Defaults !requiretty'. There's a nice blog about it here.