17. Disaster recovery¶
This document describes how to recover from a total crash of the LinOTP server by installing a new (virtual) Appliance and using a former backup file created as described in Backup and restore. If you don’t have such a backup please contact our support in order to (maybe) find a solution to retrieve the important data manually from your dysfunctional machine.
The special case of a redundant setup of LinOTP replacing one of its nodes and the additionally steps needed on top of the normal disaster recovery are covered in this manual as well. So follow the guide - if there is something special to regard or any difference to the normal procedure it will be mentioned.
17.1. Create and install a new Appliance¶
Create a virtual server (or use a bare metal one if you like) with at least 1 GB of main memory, 4 GB of hard drive space and a network card capable of accessing the internet. (VM-Ware: e1000, Hyper-V: “legacy network adapter”). Assign minimal 2 GB RAM and 10GB disk space if the LinOTP database will be stored on the same machine.
Integrate the LinOTP Virtual Appliance ISO image as a CD drive in your virtualization and start it from there.
After a short boot process, the installation menu will appear. You can choose whether the network settings should be conducted manually (M) or whether they should be received from a DHCP server (D). If you would like to cancel the installation, please select “E”.
Manual Network Configuration:
Fill in the required information and then confirm with “y”.
Automatic Network Configuration
The required information will be automatically filled in and saved.
Serial Number
Type “s” and confirm with [Enter] to enter the serial number of the LinOTP Virtual Appliance. You received the serial number together with the license file when you made your purchase. The serial number will be verified. After the verification, you can continue by pressing any key.
If you receive the error message “arxes-tolina Backend has no support for this Appliance”, answer the question “Retry?” with “n”. Check the serial number and correct it if necessary. If the installation process is interrupted despite having the correct serial number, please contact your system provider.
Select the desired keyboard layout for the further installation process.
Agree to start the installation:
Check one last time the installation data and proceed by pressing any key:
The installation of LinOTP Virtual Appliance begins after the necessary components have been downloaded from the arxes-tolina server. Please note that all data saved on this (virtual) system will be lost.
The remaining installation process does not require any additional information from the user.
The system is restarted after the conclusion of the installation. Please ensure that the following boot process does NOT take place from the ISO image, but instead from the hard drive of the virtual server. You can access LinOTP from the web interface as soon as the login prompt appears in the console of the virtual machine.
17.2. Basic configuration of the new Appliance¶
Launch your browser and enter https://[IP_address_of_your_LinOTP]:8443 in the address line in order to access the web-based configuration interface of the Appliance.
- TIP:
If your Appliance received its IP address via DHCP you can figure out the address by directly log in to the console of your virtual system using username “root” and password “eBai6Lait9” and run the command ifconfig.
When opening the configuration interface of the Appliance, a window will appear with a certificate warning that varies based on the browser used. However, there is no risk in this case. The warning notification appears because LinOTP Virtual Appliance presents the browser with a self-signed certificate when opened. Therefore, please ignore the certificate warning and confirm the access to the IP address requested (your configuration interface of the Appliance).
You will be asked for a user name and password on the login screen that then opens. Enter “root” here as the user name and the initial root password “eBai6Lait9”
After the successfull login you should see the following welcome screen:
This is the setup wizard. You have to go through all steps in order to get a minimal configured working LinOTP Appliance which you can later use to restore your backup.
1. License Agreement
Please read the license agreement carefully. To advance to the next step, check the box following “I accept the license agreement” and click on “Next”. By doing so, you declare your acceptance of the license agreement. In the next window choose the support and subscription license file you received for your LinOTP Appliance. After loading the file please check the details of the displayed information about your subscription.
2. Basic Network Configuration
This page contains all necessary information to connect the Appliance to your network.
Network Interface:
Enter the IP address and netmask via which the LinOTP Appliance should be accessed. This should probably be the same IP your crashed LinOTP machine used. Enter the default gateway under “gateway”.
Hostname and DNS:
Enter the hostname of the LinOTP server here. You can freely select this hostname, however, it should not already be used by another device in the network. Please enter the name of the domain in which the LinOTP Appliance is located separately in the next field. A list of domains can be entered in the “Search” field that will also be searched. Normally you will enter the domain names there again.
3. Setting the Time and Date
In a third step, you can change the date, time or time zone, if necessary. An NTP server can be entered from which the time is automatically received.
4. User Roles and Passwords
Even if you later restore the Appliance users and passwords from the backup you now have to configure them in order to interact with the Appliance for the restoration procedure.
The LinOTP Appliance generally recognizes three different roles:
LinOTP Administrator – the LinOTP Admin manages the LinOTP. They do not have any rights on the level of the Appliance operating system or on the network level.
Root Administrator – the Root Admin has the most rights. They can do everything except manage the LinOTP.
Appliance Administrator – the Appliance Admin may only change the Appliance functions and provide access rights. He may change passwords (including the root password if he is able to provide the previous root Password).
Now provide and/or change the passwords for these three roles. The “Old root password” requested for the Root Admin role is the initial root password “eBai6Lait9”.
Note regarding password complexity
For security reasons, be certain to select sufficiently complex passwords when changing the passwords. Passwords are deemed to be sufficiently complex by today’s standards when they meet the following criteria:
Password length of at least 10 characters
Combination of uppercase and lowercase letters, number and special characters
Once you have entered and/or changed all of the passwords, go ahead to the next step by clicking the “Next” button. Please note that the input mask depicted can only be exited when all fields have been completed and the newly entered passwords are highlighted in green. Scrolling backwards using the “Previous” button is also not possible until the fields have been fully completed.
5. Definition of the RADIUS Clients
The RADIUS configuration is (hopefully) in your backup - so check the “no configuration” box here.
6. Key Generation and Database Passwords
In the last step, you decide whether the factory installed keys for SSH, SSL and to encrypt the files in the database and the internal database passwords should be generated again. This is absolutely recommended when running the wizard for the first time in order to prevent unauthorized third parties with knowledge of the factory installed keys and passwords from accessing the system or the files. Some of these keys are stored in your backupfile and can be imported during the restore operation later.
In addition, you can specify in this mask whether the Appliance should be shut down after the completion of the configuration. If you have changed the IP address the Appliance will only be accessible via the new IP address after the activation of the configuration and can no longer be administrated and/or turned off via the existing browser session.
Click the “Activate” button to save all entries, activate and - if selected - shutdown the Appliance. All further configurations can be made from any computer within the network.
17.3. Restore the backup¶
Now we are going to restore the old settings of your crashed LinOTP Appliance to the new machine.
- Restoring a LinOTP node - deactivate redundancy during restoring procedure:
In order to make sure, which node is the primary for the synchronization process, you have to deactivate the redundancy on the working node before restoring the backup on the replacement machine. Afterwards we will freshly initialize the two node setup. So log in to the console of the reference system and execute:
appliance_configure.py -c reset_redundancy
Open the management webinterface of your new installed Appliance using the credentials set during the installation procedure:
https://IP_OF_YOUR_LINOTP:8443
Navigate to System -> Restore
Choose the backup file and enter the passphrase that was used to encrypt this file:
If the passphrase is correct, you will see the following popup:
Now choose the datas you would like to apply to your new LinOTP Appliance from the backup:
The restorable components in depth:
- Appliance configuration /etc
Contains important configuration details of the machine itself, like:
network configuration (IP, gateway, nameserver, NTP)
RADIUS server configuration
scheduled backups
update behavior
- Database encryption key
Is needed to access the token database. You can store the key and the database in independent backup files to improve security. Anyhow - the key has to be the right one for the restored database…
- SSL certificate and key
The SSL infrastructure of you LinOTP Appliance. The keys are used when someone accesses one of the webinterfaces and for the authentication SSL-API of LinOTP.
- Database
The token database itself, containing the seeds, UserIdResolver configuration, realms, policies, so essentially the whole user-token management system.
If in doubt, choose all components for restoration.
Please be aware: log files and data files of Flatfile UserIdResolvers are
not included in the backup. The log files can be downloaded using the
support file in System -> Support. The data files for Flatfile
UserIdResolvers must be backed up manually.
Restoring a LinOTP node:
It should be sufficient to choose only Appliance configuration /etc (and if wanted SSL certificate and key), because the database and the encryption key will be replicated if the redundancy setup is executed later.
Be aware:
Don’t close your browser until you see the messages of success at the bottom of the website. This can take some time… be patient!
Restoring a LinOTP node - reactivate redundancy setup:
To be on the safe site we are going to deactivate the redundancy first and then initialize the two node setup from the old working node to make sure, the synchronization is done in the right direction: to the recovered node.
Log in to the NEW machine and execute:
appliance_configure.py -c reset_redundancy
Log in to the REFERENCE machine and execute:
appliance_configure.py -c reset_redundancy
appliance_configure.py -c setup_redundancy -p IP_OF_THE_NEW_SYSTEM
For security reasons you have to make sure to connect to the right machine by comparing the SSH host key displayed during the setup procedure to the one of the new machine. You will find the SSH host key in the Appliance management webinterface https://IP_OF_YOUR_NEW_SYSTEM:8443 in System -> Advanced -> SSH configuration:
The complete redundancy procedure looks like:
root@linotpappliance:~# appliance_configure.py -c setup_redundancy -p
192.168.122.161
Welcome to the redundancy wizard.
Please make sure that timer_entropyd is running.
Do you want to setup IPSec-based encryption between the machines? [y/n] y
You will have to verify the partner hostkey fingerprint and give the partner's
root password in order to start the setup process.
Since the SSH command only waits for input for a certain time, please make sure
that:
* the partner hostkey fingerprint
* the partner root password
are close at hand.
You can find the partner hostkey fingerprint at https://192.168.122.161:8443/
at System-Advanced-Redundancy.
Please hit RETURN to continue:
The authenticity of host '192.168.122.161 (192.168.122.161)' can't be
established.
RSA key fingerprint is af:e7:6a:19:87:a1:3b:b7:d4:5d:57:9e:f4:94:8d:c6.
Are you sure you want to continue connecting (yes/no)? yes
root@192.168.122.161's password:
bind: Cannot assign requested address
Success!
Terminating the SSH tunnel ... (this is expected!)
17.4. Check the new LinOTP Appliance¶
Review the success of the restore procedure. Log in to the Smart Virtual Appliance management webinterface (consider that the IP might have been changed due to the recovery):
https://IP_OF_YOUR_LINOTP:8443
- In case you can not connect to the webinterface:
If something went wrong and you are unable to reach the Appliance e.g. because of a wrong IP you can log into the console of your virtual system using username “root” and the appropiate password as it was set in the backup. Enter the command unsupported. Now you have full administrational access and you can fix the wrong IP with ifconfig eth0 IP_YOU_WANT. Now you should be able to connect to the webinterface to check and fix configuration there.
Note
If you have changed the IP in unsupported mode in order to reach again the web interfaces of LinOTP please do not forget to set the desired IP in the appliance dashboard for persistent configuration.
Check configuration:
Have a look at least at:
network/routing/hostname/DNS settings
correct time configuration
If used: the RADIUS setup - are all client configurations correctly restored?
Be aware:
Not all configuration details are backuped and restored by LinOTP Appliance. So if applicable please review after the recovery procedure special settings you remember like:
deactivated/activated services, e.g. entropyd or sshd
SSH server- and clientkeys & SSH server configuration itself
Finally connect to the token- and user management webinterface:
https://IP_OF_YOUR_LINOTP/manage
Are the UserIdResolver working correctly and are the users displayed in User View sorted by their realms?
Do you see tokens in Token View and are they assign to users?
Are the policies restored?
Node recovery:
Rollout an example token on one of the nodes and assign it to a test user. Switch to the other machine and check if you can see the token and the assignment there as well.
Test if Appliance is working properly:
If you possess a test account, please try to authenticate by accessing
https://IP_OF_YOUR_LINOTP/validate?user=TESTUSER&pass=PINOTP
Where the possible parameters are:
- user
the name of the user that should be authenticated
- pass
in most cases a combination of the PIN and the OTP of the user
- realm (optional)
the realm of the user; must be specified, if the user is not member of the default realm
In case of success you should see something like this:
If RADIUS is used you should additionally check whether a user can authenticate via this infrastructure as well.
- Be aware:
Your backup data might be a bit older, i.e. the HMAC counter of the tokens might be out of sync with the data in the database. So you probably have to ask your users to go to the Selfservice Portal and resynchronize their tokens. You can provide the necessary ability for your users by configuring the selfservice policy resync as described in Selfservice policies.