Migrating to LinOTP 3

3.9. Migrating to LinOTP 3#

If you are not using appliance you need to migrate LinOTP and RADIUS service. Follow this section. For upgrading appliance based LinOTP read here Appliance uprade to Version 3

The jump to version 3 of LinOTP brings several breaking changes. To ensure a smooth migration, please read the following sections.

3.9.1. Configuration#

The .ini files used formerly have been replaced with .cfg files, with most configuration parameters renamed (due to restrictions imposed by the Flask web server). Configuration parameters can now also be set via the process environment, for increased convenience in container-based setups (e.g., Docker). This is done by prefixing their names with LINOTP_.

For example, the database configuration URL previously found in the .ini file as sqlalchemy.url can now be found in the DATABASE_URI setting, and the respective environment variable is LINOTP_DATABASE_URI.

Configuration directory#

The settings directory /etc/linotp2 is renamed to /etc/linotp.

Configuration file#

Configuration settings will now be loaded from multiple locations. These locations are determined by the value of the LINOTP_CFG environment variable, which may contain a colon-separated list of directory names and file glob expressions. LinOTP checks these names and file glob expressions in the order they occur in LINOTP_CFG, and reads any matching files. File names resulting from the expansion of a glob expression are processed in lexicographic order. Directory names have *.cfg appended to them implicitly. Relative directory names and file glob expressions are taken relative to the application root directory (where the app.py file is found).

If LINOTP_CFG is not set and a file called linotp-cfg-default exists in the application root directory, the content of that file is used as the default value for LINOTP_CFG (this feature is generally used to establish a distribution-specific search order for configuration files). If the file does not exist, linotp.cfg is assumed as the default value.

Configuration file locations for Debian packages (as per linotp-cfg-default)#

File path

Description

/usr/share/linotp/linotp.cfg

Distribution defaults - these settings are supplied by the packaging and represent settings that are appropriate for the distribution and packaging format used.

/etc/linotp/linotp.cfg

Central configuration settings.

/etc/linotp/conf.d/*

Configuration snippets can be placed in this directory. The Debian packaging places the database configuration in 01-debian-database.cfg

The order of the locations in the above table matters. Settings in later files in the search order will override settings previously found. For example, if LOGFILE_DIR is defined in the following places:

LOGFILE_DIR=/srv/linotp-logs in the file /etc/linotp/linotp.cfg and

LOGFILE_DIR=/var/log/linotp in the file /usr/share/linotp/linotp.cfg,

then the value that is used is /srv/linotp-logs, because that was the last one encountered.

New configuration settings#

Configuration settings have been renamed in LinOTP 3. The existing configuration entries in a LinOTP instance are migrated during the upgrade from version 2 to version 3. You can find an overview of the changes in the following table:

Old .ini name

New .cfg name

Details

sqlalchemy.url

DATABASE_URI

If you are using the Debian packages this setting will be automatically migrated during upgrade

linotpSecretFile

SECRET_FILE

linotpAudit.sql.url

AUDIT_DATABASE_URI

When set to the new default value of SHARED, linotp will write the audit log to DATABASE_URI

linotpAudit.sql.pool_recycle

AUDIT_POOL_RECYCLE

linotpAudit.key.private

AUDIT_PRIVATE_KEY_FILE

linotpAudit.key.public

AUDIT_PUBLIC_KEY_FILE

linotpAudit.error_on_truncation

AUDIT_ERROR_ON_TRUNCATION

linotpadmin.username

ADMIN_USERNAME

linotpadmin.password

ADMIN_PASSWORD

linotpGetotp.active

GETOTP_ENABLED

linotpHelp.url

HELP_URL

linotpTokenModules

TOKEN_MODULES

cache_dir

CACHE_DIR

DATA_DIR/template-cache

Generated Mako templates previously found under cache_dir are now under the new DATA_DIR directory

custom_templates

CUSTOM_TEMPLATES_DIR

sqlalchemy.echo

logging

logger_*

handler_*

formatter_*

LOGGING

See the new logging section of the documentation

service.*

CONTROLLERS

maintenance_verify_client_env_var

MAINTENANCE_VERIFY_CLIENT_ENV_VAR

radius.nas_identifier

RADIUS_NAS_IDENTIFIER

debug

DEBUG (not completely equivalent)

See the developer documentation for details of how this setting has changed

Configuration settings whose names end in _FILE or _DIR contain file or directory names, respectively. These names can be either absolute (starting with a slash) or relative, in which case they are evaluated (at the time of use, not when the configuration is being read) starting from the directory named by the ROOT_DIR configuration setting. This setting must contain an absolute directory name.

Removed settings#

The following settings have been removed from the configuration file and are no longer supported. Many of them are only meaningful in the context of the old Pylons-based implementation and have no equivalent in the new Flask-based implementation.

In the default section:

  • unitTest

  • profile

  • smtp_server

  • email_error_from

  • radius.dictfile

  • linotpAudit.sql.highwatermark

  • linotpAudit.sql.lowwatermark

  • linotpAudit.sql.table_prefix

  • linotpNoSessionCheck

  • linotp.uencode_data

  • linotp.DefaultSyncWindow

  • linotp.DefaultOtpLen

  • linotp.DefaultCountWindow

  • linotp.DefaultMaxFailCount

  • linotp.FailCounterIncOnFalsePin

  • linotp.PrependPin

  • linotp.DefaultResetFailCount

  • linotp.splitAtSign

In the server section:

  • use

  • host

  • port

  • ssl_pem

The host, port, and ssl_pem settings are obsoleted by the fact that LinOTP is run productively as a WSGI application, i.e., these settings should be configured in the web server (e.g., Apache) or WSGI application server (e.g., uWSGI). During development they can be set using the Flask development server’s CLI.

In the app:main section:

  • use

  • full_stack

  • static_files

  • who.config_file

  • who.log_file

  • who.log_level

  • beaker.session.key

  • beaker.session.secret

  • licfile

3.9.2. Database encoding migration#

Starting with LinOTP2 there has been support for mysql. Mysql in its early version did not support the today standard utf8 encoding to store non ascii data. Instead mysql did use the ISO latin encoding on top of a python 2 which as well did not fully support utf8. During the years the support for utf8 increased and mysql and the python 2 db driver layer changed the standard encoding to utf8.

Thus there are cases when you migrate your LinOTP server from an early version to the 3.2 version of linotp & database & database driver, you will see problems with the encoding eg. when looking at the manage ui or in your linotp log files.

To support you with this situation there is a linotp command available which is:

$ linotp admin fix-db-encoding

which will convert the mysql data and database schema from the ISO-8859-1 encoding to the utf8 standard encoding. The command:

$ linotp admin fix-db-encoding --help

will describe which data in your database will be migrated.

But be careful with this command and apply this command only if you have some log entries which show up hints for a conversion. And as always: do a backup of your data upfront before doing any data changing command.

3.9.3. Administrator authentication#

LinOTP no longer needs to rely on the authentication mechanisms of web servers (such as Apache’s htaccess) for securing access to the management interface and management API.

There is a dedicated realm containing a dedicated resolver for defining administrator accounts. You can add an account to this resolver by executing linotp local-admins add <username> and linotp local-admins password <username>. Further documentation on the linotp local-admins command can be found in the respective man page. You can also add other resolvers to the administrator realm.

The management API authentication is done by calling the http://<linotp-server>/admin/login API endpoint with username and password in the body (see LinOTP’s API documentation). The response returns a JSON Web Token (JWT) which has to be used for all management API calls.

Management API endpoints which modify data or the server state can only be accessed via HTTP POST request (see the full list below). In addition to the JWT for authentication, these endpoints require a X-CSRF-TOKEN field in the header. Its value is provided by the csrf_access_token cookie.

3.9.4. API endpoints requiring POST requests#

Following LinOTP API endpoints are now accessible via HTTP POST requests only. This is to prevent data or server state modification via cross-site request forgery (CSRF).

  • admin/assign

  • admin/copyTokenPin

  • admin/copyTokenUser

  • admin/disable

  • admin/enable

  • admin/init

  • admin/loadtokens

  • admin/losttoken

  • admin/remove

  • admin/reset

  • admin/resync

  • admin/set

  • admin/setPin

  • admin/setValidity

  • admin/tokenrealm

  • admin/unassign

  • admin/unpair

  • maintenance/setLogLevel

  • migrate/backup

  • migrate/restore

  • reporting/delete_all

  • reporting/delete_before

  • system/delConfig

  • system/delPolicy

  • system/delProvider

  • system/delRealm

  • system/delResolver

  • system/importPolicy

  • system/setConfig

  • system/setDefault

  • system/setDefaultProvider

  • system/setDefaultRealm

  • system/setPolicy

  • system/setProvider

  • system/setRealm

  • system/setResolver

  • system/setSupport

  • system/setupSecurityModule

  • tools/import_users

  • tools/migrate_resolver

  • tools/setPassword

  • userservice/activateocratoken

  • userservice/assign

  • userservice/delete

  • userservice/disable

  • userservice/enable

  • userservice/enroll

  • userservice/finishocra2token

  • userservice/reset

  • userservice/resync

  • userservice/setdescription

  • userservice/setmpin

  • userservice/setpin

  • userservice/unassign

  • userservice/webprovision

For detailed information on each endpoint please have a look at the LinOTP API documentation.

3.9.5. User interface#

Audit trail tab of ManageUI#

In the Audit trail, the times are expressed in UTC. The Column header expresses this clearly by “Date (UTC)” so we make sure we are all on the same page. If you have any script that uses the column/header to read data from this table, you might need to consider readjusting it.

3.9.6. Howto migrating from LinOTP 2 to LinOTP 3#

In an environment based on the SVA, the restorewizard can be used. Please follow the documentation at https://linotp.org/doc/latest/part-appliance/upgrade.html

If no SVA is available you can work with the command line of LinOTP.

A short description of the procedure follows.

  • First a backup is created on the productive system. You will be asked for a passphrase to encrypt the encKey with. The same password is required for unpacking during recovery on a LinOTP 2. For LinOTP 3, we will transfer the encKey directly to the new machine.

# linotp-backup
--------- LSE LinOTP 2 backup -----------------------
----- Backing up encryption Key ---------------------
Think of a good passphrase you can remember well...

----- Backing up token database ---------------------
We will now backup the database <<LinOTP2>> on the host <<192.168.100.5>>.
This will be done with the user <<linotp2>>.
The database will be written to <<linotp_backup_2212061351.sql>>.
...done.

This saves the initial data for the migration. Of course, the backup should be renewed directly before the actual migration in order to migrate the most current data possible. The access data to the LinOTP2 database are read from the configuration and used.

Hint

Note that the files created here contain important data and are not intended for third parties.

  • Prepare the LinOTP 3. Install the new virtual machine and run the configuration of the required services, use https://linotp.org/download.html. Especially check the function of the NTP client.

  • Set up a local admin to manage LinOTP.

linotp local-admins --help
# Add User 'admin' for /manage
linotp local-admins add admin
# Set Password for User 'admin'
linotp local-admins password admin

  • Now the user admin can log in to the management GUI /manage.

    https://linotp-fqdn/manage

  • After successful login LinOTP can be configured. To check the function it is best to set up a UserIdResolver according to the example of your LinOTP 2. Roll out a token as an example and test it with the test page

    https://linotp-fqdn/auth/index.

In case of problems with login or rolling out the token, carefully read the logfile in

/var/log/apache2/error.log
/var/log/linotp/linotp.log

  • Now the actual migration takes place.

    Stop the Apache web server and renew the LinOTP backup from the old server.

    Copy the current data to the new LinOTP 3.

# A fresh backup contains the last state of the data.
linotp-backup
# The encKey is needed on the new machine to be able to read the database completely.
cp /etc/linotp2/encKey .
tar -czf linotp2.backup.tar.gz encKey linotp_backup_....sql
# The tar can optionally be additionally encrypted with gpg.
# Copy the tar directly to the new LinOTP 3
scp linotp2.backup.tar.gz <user>@linotp3-fqdn:
ssh <user>@linotp3-fqdn
# Unpacking the tar for the next steps.
tar -cf linotp2.backup.tar.gz
# The encKey allows reading in the new database after the restore , the current database becomes unusable.
cp encKey /etc/linotp
chown linotp:www-data /etc/linotp/encKey
chmod 400 /etc/linotp/encKey

  • Use the linotp tool to perform the restore on the LinOTP 3.

linotp backup restore --file linotp_backup_2212061351.sql

‘linotp backup restore’ reads the access data for the database from the LinOTP configuration and writes the data from the SQL dump via this path into the open database.

Hint

In the backup of LinOTP 2 there are no local administrators for /manage, the corresponding table remains from the previous configuration and the admins can log in, After the passwords are reset. All other tables are emptied and refilled.

  • After the restore the database is in the version level of LinOTP 2, it is necessary to raise it to LinOTP 3. This is done by the command

# This request is listed in the log file or output by the linotp tool.
linotp init database
# This prompt will appear only if necessary and should be executed only then.
linotp admin fix-db-encoding

  • Now Apache must be restarted. After that /manage is available with the new data. Please note messages from the log files.

  • Resetting the admin passwords.

linotp local-admins --help
# Add User 'admin' for /manage
linotp local-admins list
# Set Password for User 'admin'
linotp local-admins password admin
# You can also add new admins

Hint

If administrators were allowed from an LDAP group in the previous version of LinOTP, they can be re-entered and reactivated in LinOTP. To do this, follow this section: LinOTP administrators.