Commit c7c058e3 authored by Andreas Mann's avatar Andreas Mann
Browse files

initial

parents
GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
This version of the GNU Lesser General Public License incorporates
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.
0. Additional Definitions.
As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
General Public License.
"The Library" refers to a covered work governed by this License,
other than an Application or a Combined Work as defined below.
An "Application" is any work that makes use of an interface provided
by the Library, but which is not otherwise based on the Library.
Defining a subclass of a class defined by the Library is deemed a mode
of using an interface provided by the Library.
A "Combined Work" is a work produced by combining or linking an
Application with the Library. The particular version of the Library
with which the Combined Work was made is also called the "Linked
Version".
The "Minimal Corresponding Source" for a Combined Work means the
Corresponding Source for the Combined Work, excluding any source code
for portions of the Combined Work that, considered in isolation, are
based on the Application, and not on the Linked Version.
The "Corresponding Application Code" for a Combined Work means the
object code and/or source code for the Application, including any data
and utility programs needed for reproducing the Combined Work from the
Application, but excluding the System Libraries of the Combined Work.
1. Exception to Section 3 of the GNU GPL.
You may convey a covered work under sections 3 and 4 of this License
without being bound by section 3 of the GNU GPL.
2. Conveying Modified Versions.
If you modify a copy of the Library, and, in your modifications, a
facility refers to a function or data to be supplied by an Application
that uses the facility (other than as an argument passed when the
facility is invoked), then you may convey a copy of the modified
version:
a) under this License, provided that you make a good faith effort to
ensure that, in the event an Application does not supply the
function or data, the facility still operates, and performs
whatever part of its purpose remains meaningful, or
b) under the GNU GPL, with none of the additional permissions of
this License applicable to that copy.
3. Object Code Incorporating Material from Library Header Files.
The object code form of an Application may incorporate material from
a header file that is part of the Library. You may convey such object
code under terms of your choice, provided that, if the incorporated
material is not limited to numerical parameters, data structure
layouts and accessors, or small macros, inline functions and templates
(ten or fewer lines in length), you do both of the following:
a) Give prominent notice with each copy of the object code that the
Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the object code with a copy of the GNU GPL and this license
document.
4. Combined Works.
You may convey a Combined Work under terms of your choice that,
taken together, effectively do not restrict modification of the
portions of the Library contained in the Combined Work and reverse
engineering for debugging such modifications, if you also do each of
the following:
a) Give prominent notice with each copy of the Combined Work that
the Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the Combined Work with a copy of the GNU GPL and this license
document.
c) For a Combined Work that displays copyright notices during
execution, include the copyright notice for the Library among
these notices, as well as a reference directing the user to the
copies of the GNU GPL and this license document.
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to
recombine or relink the Application with a modified version of
the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying
Corresponding Source.
1) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (a) uses at run time
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.
e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
GNU GPL, and only to the extent that such information is
necessary to install and execute a modified version of the
Combined Work produced by recombining or relinking the
Application with a modified version of the Linked Version. (If
you use option 4d0, the Installation Information must accompany
the Minimal Corresponding Source and Corresponding Application
Code. If you use option 4d1, you must provide the Installation
Information in the manner specified by section 6 of the GNU GPL
for conveying Corresponding Source.)
5. Combined Libraries.
You may place library facilities that are a work based on the
Library side by side in a single library together with other library
facilities that are not Applications and are not covered by this
License, and convey such a combined library under terms of your
choice, if you do both of the following:
a) Accompany the combined library with a copy of the same work based
on the Library, uncombined with any other library facilities,
conveyed under the terms of this License.
b) Give prominent notice with the combined library that part of it
is a work based on the Library, and explaining where to find the
accompanying uncombined form of the same work.
6. Revised Versions of the GNU Lesser General Public License.
The Free Software Foundation may publish revised and/or new versions
of the GNU Lesser General Public License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the
Library as you received it specifies that a certain numbered version
of the GNU Lesser General Public License "or any later version"
applies to it, you have the option of following the terms and
conditions either of that published version or of any later version
published by the Free Software Foundation. If the Library as you
received it does not specify a version number of the GNU Lesser
General Public License, you may choose any version of the GNU Lesser
General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide
whether future versions of the GNU Lesser General Public License shall
apply, that proxy's public statement of acceptance of any version is
permanent authorization for you to choose that version for the
Library.
# iReceptor Turnkey scripts for podman
The provided scripts in scriptspod let the turnkey service (https://github.com/sfu-ireceptor/turnkey-service-php) run with podman instead of docker on operating systems where docker is not available like Centos 8.
The web\_stats scripts and test\_performance.sh have not yet been adjusted.
## Installation of the scripts
Clone branch production-v3 from https://github.com/sfu-ireceptor/turnkey-service-php to a location where you have enough storage to host the amount of data you want to support.
Copy scriptspod directory to turnkey-service installation directory.
## Edit file podman.conf.sh.EDIT
Configure the following variables then remove .EDIT from the file.
Absolute path to your backup directory, the mongo db container will have this path inside its filesystem at /mnt/bkup
```
PATH_BKUP_DIR="/var/data/bkup"
```
You can run several installations of turnkey on same machine. In that case add a suffix to container names, e. g. dev, when running the latest container versions.
```
DATABASE_CONT_NAME_SUFFIX=""
API_CONT_NAME_SUFFIX=""
DATALOADING_CONT_NAME_SUFFIX=""
PERFORMANCE_TESTING_CONT_NAME_SUFFIX=""
```
Container tags are configured for production installation, for dev installations use second value.
```
# prod:turnkey-v3 dev:master
DATABASE_TAG="turnkey-v3"
# prod:turnkey-v3 dev:latest
API_TAG="turnkey-v3"
# prod:turnkey-v3-dataloading dev:turnkey-dataloading-latest
DATALOADING_TAG="turnkey-v3-dataloading"
# prod:turnkey-v3-performance dev:turnkey-performance-latest
PERFORMANCE_TESTING_TAG="turnkey-v3-performance"
```
Port on the host where the API endpoint is accessible. Open this port on the host firewall if required.
```
POD_EX_PORT="8443"
```
Port of the mongo database container on the host. Normally this port should not be accessible from outside the host.
```
POD_EX_PORT_DB="27017"
```
Name for pod, e.g. use turnkey-service and turnkey-service-dev for latest versions of containers.
```
POD_NAME_SVC="turnkey-service"
```
User id for running the database container.
```
POD_USER_ID="1000"
```
## Install turnkey service
Change to turnkey installation folder, then run
```
scriptspod/install_turnkey.sh
```
## Start/Stop the turnkey service
The install script will create a service pod-*POD\_NAME\_SVC*.service where *POD\_NAME\_SVC* is the name of the pod service.
```
systemctl start pod-POD_NAME_SVC.service --user
systemctl stop pod-POD_NAME_SVC.service --user
```
## Load data to turnkey
Follow the documentation on the turnkey service repository, just call the same scripts in the scriptspod folder.
## Backup database
This is different from the turnkey with docker installation. Normally schedule backups from a cron job in case data has been loaded to the turnkey service. To run an unscheduled backup, create a file BKUP\_DB.FLAG in the backup directory for this turnkey instance, then run
```
scriptspod/backup_database.sh
```
This will create a file in the incoming folder in PATH\_BKUP\_DIR/POD\_NAME\_SVC
mongodb\_SRVCENTOS01\_turnkey-service\_20210416T020201.gzip.dump
e.g. SRVCENTOS01 is your hostname, turnkey-service is the value of POD\_NAME\_SVC
## Restore a database
To restore a database move the database dump to the restore folder in PATH\_BKUP\_DIR/POD\_NAME\_SVC
e.g. move file mongodb\_SRVCENTOS01\_turnkey-service\_20210415T135049.gzip.dump from /var/data/bkup/turnkey-service/backup.daily/2021-04-15 to /var/data/bkup/turnkey-service/restore.
Then cd to turnkey installation folder and run scriptspod/restore\_database.sh mongodb\_SRVCENTOS01\_turnkey-service\_20210415T135049.gzip.dump. Do not specifiy a path, just give the filname.
The container will see the file in its own location and restore the database.
## cronjob for databse backups
Schedule the backup process when the data has changed. The data load scripts will trigger a database backup, this can be detected with a cron job.
```
# daily bkup for turnkey-service-php
2 2 * * * /var/data/turnkey-service-php/scriptspod/backup_database.sh >/dev/null 2>&1
4 2 * * * /var/data/turnkey-service-php/scriptspod/bkup_file_rotation.sh >/dev/null 2>&1
```
Adjust this process (specifically the file rotation script) according to your needs.
## Contact
Send questions and comments to andreas.mann@dkfz-heidelberg.de
# See "Using the turnkey developement version" at:
# https://github.com/sfu-ireceptor/turnkey-service-php/blob/master/doc/dev.md
DATABASE_TAG=dev
API_TAG=latest
DATALOADING_TAG=turnkey-dataloading-latest
PERFORMANCE_TESTING_TAG=turnkey-performance-latest
210427
- README changes
210421
- install script corrected unshare command
210420
- bkup_file_rotation.sh dont create empty folders
- update readme files
210417
- new configuration option DO_BKUP_FLAG absolute path to location of file \
which will trigger db backup, is checked in backup_database.sh
- created by load_ and update_scripts
# Backup and Restore with turnkey and podman
The install script will create folders below PATH_BKUP_DIR, see podman.conf.sh.
incoming backup_database.sh will create mongodb dump file here when running backup_database.sh
backup.daily file rotation will put dumped file here on a regular day
backup.weekly on saturday file will be placed here
backup.monthly on first day of month
restore move any file to restore to this folder and use filename as argument in restore_database.sh
#!/bin/sh
restore_abs_file_path=$1
parent_dir="$(dirname "$restore_abs_file_path")"
base_name="$(basename "$restore_abs_file_path")"
SCRIPT_DIR=`dirname "$0"`
SCRIPT_DIR_FULL="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
POD_CONF_FILE="podman.conf.sh"
# read configuration for turnkey pod
. $SCRIPT_DIR_FULL/$POD_CONF_FILE
#echo ${DO_BKUP_FLAG}
if [ -f "${DO_BKUP_FLAG}" ]; then
log "DO_BKUP_FLAG exists...";
rm ${DO_BKUP_FLAG}
else
exit;
fi
#bkup_file_name=mongodb_${host_name}_${POD_NAME_SVC}_$(date +%s).dump
bkup_file_name=mongodb_${host_name}_${POD_NAME_SVC}_$(date +%Y%m%dT%H%M%S).gzip.dump
#sudo docker-compose --file ${SCRIPT_DIR}/docker-compose.yml --project-name turnkey-service exec -T ireceptor-database \
# sh -c 'mongodump --archive'
podman exec -e MONGO_INITDB_DATABASE="ireceptor" \
-e bkup_file_name=$bkup_file_name \
ireceptor-database$DATABASE_CONT_NAME_SUFFIX \
sh -c 'mongodump --archive=/mnt/bkup/incoming/$bkup_file_name --gzip'
log "backup db: ${bkup_file_name}"
#!/bin/bash
# Julius Zaromskis
# Backup rotation
# https://nicaw.wordpress.com/2013/04/18/bash-backup-rotation-script/
SCRIPT_DIR=`dirname "$0"`
SCRIPT_DIR_FULL="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
POD_CONF_FILE="podman.conf.sh"
# read configuration for turnkey pod
. $SCRIPT_DIR_FULL/$POD_CONF_FILE
# Storage folder where to move backup files
# Must contain backup.monthly backup.weekly backup.daily folders
# storage=/home/backups/your_website_name
storage=${PATH_BKUP_DIR}/${POD_NAME_SVC}
# Source folder where files are backed
source=$storage/incoming
# Destination file names %d-%m-%Y"
date_daily=`date +"%Y-%m-%d"`
#date_weekly=`date +"%V sav. %m-%Y"`
#date_monthly=`date +"%m-%Y"`
# Get current month and week day number
month_day=`date +"%d"`
week_day=`date +"%u"`
# Optional check if source files exist. Email if failed.
#if [ ! -f $source/archive.tgz ]; then
# ls -l $source/ | mail $sysadmin_email \
# -s "[${host_name}-bkup script] Daily backup failed! Please check for missing files."
#fi
if [ "$(ls -A $source)" ]; then
echo "File found..."
else
exit;
fi
# It is logical to run this script daily. We take files from source folder and move them to
# appropriate destination folder
# On first month day do
if [ "$month_day" -eq 1 ] ; then
destination=${storage}/backup.monthly/$date_daily
else
# On saturdays do
if [ "$week_day" -eq 6 ] ; then
destination=${storage}/backup.weekly/$date_daily
else
# On any regular day do
destination=${storage}/backup.daily/$date_daily
fi
fi
# Move the files
mkdir $destination
mv -v $source/* $destination
# daily - keep for 14 days
find $storage/backup.daily/ -maxdepth 1 -mtime +14 -type d -exec rm -rv {} \;
# weekly - keep for 60 days
find $storage/backup.weekly/ -maxdepth 1 -mtime +60 -type d -exec rm -rv {} \;
# monthly - keep for 300 days
find $storage/backup.monthly/ -maxdepth 1 -mtime +300 -type d -exec rm -rv {} \;
echo "done"
#!/bin/bash
#SCRIPT_DIR_FULL="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
POD_CONF_FILE="podman.conf.sh"
. scriptspod/$POD_CONF_FILE
# stop and disable turnkey service
systemctl --user stop pod-${POD_NAME_SVC}.service
systemctl --user disable pod-${POD_NAME_SVC}.service
# stop and remove turnkey-service pod
podman pod stop $POD_NAME_SVC
podman pod rm $POD_NAME_SVC
# stop and remove awstats container
#podman stop awstats
#podman rm awstats
# remove database folder
sudo rm -r .mongodb_data
# check where this file is coming from
rm 0
# remove log
rm $LOGFILE_NAME
# neede for adc-api-test dataset loading
rm -rf ${PATH_BKUP_DIR}/${POD_NAME_SVC}/restore
rm -rf ${PATH_BKUP_DIR}/${POD_NAME_SVC}/incoming
#export MONGO_DBDIR=/var/data/turnkey-service-php/.mongodb_data
#!/bin/sh
SCRIPT_DIR=`dirname "$0"`
SCRIPT_DIR_FULL="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
POD_CONF_FILE="podman.conf.sh"
# read configuration for turnkey pod
. $SCRIPT_DIR_FULL/$POD_CONF_FILE
echo -n "Starting $0: "
date
# create indexes
#sudo docker-compose --file ${SCRIPT_DIR}/docker-compose.yml --project-name turnkey-service exec -T ireceptor-database \
# sh -c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/create_indexes.js'
podman exec -e MONGO_INITDB_DATABASE="ireceptor" \
ireceptor-database${DATABASE_CONT_NAME_SUFFIX} sh \
-c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/create_indexes.js'
# create query plans
${SCRIPT_DIR}/create_database_queryplans.sh
echo -n "Done $0: "
date
#!/bin/sh
SCRIPT_DIR=`dirname "$0"`
SCRIPT_DIR_FULL="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
POD_CONF_FILE="podman.conf.sh"
# read configuration for turnkey pod
. $SCRIPT_DIR_FULL/$POD_CONF_FILE
# create query plans
#sudo docker-compose --file ${SCRIPT_DIR}/docker-compose.yml --project-name turnkey-service exec -T ireceptor-database \
# sh -c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/create_query_plans.js'
podman exec -e MONGO_INITDB_DATABASE="ireceptor" ireceptor-database${DATABASE_CONT_NAME_SUFFIX} sh \
-c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/create_query_plans.js'
[Unit]
Description=Disable Transparent Huge Pages (THP)
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=mongod.service
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo never | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null'
[Install]
WantedBy=basic.target
#!/bin/bash
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: mongod mongodb-mms-automation-agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description: Disable Linux transparent huge pages, to improve
# database performance.
### END INIT INFO
case $1 in
start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fi
echo 'never' > ${thp_path}/enabled
echo 'never' > ${thp_path}/defrag
re='^[0-1]+$'
if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
then
# RHEL 7
echo 0 > ${thp_path}/khugepaged/defrag
else
# RHEL 6
echo 'no' > ${thp_path}/khugepaged/defrag
fi
unset re
unset thp_path
;;
esac
version: '3'
services:
ireceptor-database:
image: ireceptor/repository-mongodb:${DATABASE_TAG:-turnkey-v3}
volumes:
- ../.mongodb_data:/data/db
environment:
MONGO_INITDB_DATABASE: ireceptor
ireceptor-api:
depends_on:
- ireceptor-database
image: ireceptor/service-php-mongodb:${API_TAG:-turnkey-v3}
environment:
DB_HOST: ireceptor-database
DB_DATABASE: ireceptor
DB_SAMPLES_COLLECTION: sample
DB_SEQUENCES_COLLECTION: sequence
ports:
- ${API_PORT:-80}:80
ireceptor-dataloading:
depends_on:
- ireceptor-database
image: ireceptor/dataloading-mongo:${DATALOADING_TAG:-turnkey-v3-dataloading}
environment:
DB_HOST: ireceptor-database
DB_DATABASE: ireceptor
volumes:
- ${FILE_FOLDER:-/tmp}:/scratch
ireceptor-performance-testing:
image: ireceptor/dataloading-mongo:${PERFORMANCE_TESTING_TAG:-turnkey-v3-performance}
environment:
DB_HOST: ireceptor-database
DB_DATABASE: ireceptor
DB_SAMPLES_COLLECTION: sample
DB_SEQUENCES_COLLECTION: sequence
#!/bin/sh
SCRIPT_DIR=`dirname "$0"`
#sudo docker-compose --file ${SCRIPT_DIR}/docker-compose.yml --project-name turnkey-service exec -T ireceptor-database \
# sh -c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/drop_indexes.js'
SCRIPT_DIR_FULL="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
POD_CONF_FILE="podman.conf.sh"
# read configuration for turnkey pod
. $SCRIPT_DIR_FULL/$POD_CONF_FILE
podman exec -e MONGO_INITDB_DATABASE="ireceptor" \
ireceptor-database${DATABASE_CONT_NAME_SUFFIX} sh \
-c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/drop_indexes.js'
#!/bin/sh
SCRIPT_DIR=`dirname "$0"`
#sudo docker-compose --file ${SCRIPT_DIR}/docker-compose.yml --project-name turnkey-service exec -T ireceptor-database \
# sh -c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/drop_indexes_dataloading.js'
SCRIPT_DIR_FULL="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
POD_CONF_FILE="podman.conf.sh"
# read configuration for turnkey pod
. $SCRIPT_DIR_FULL/$POD_CONF_FILE
podman exec -e MONGO_INITDB_DATABASE="ireceptor" \
ireceptor-database${DATABASE_CONT_NAME_SUFFIX} \
sh -c 'cd /app && mongo --quiet $MONGO_INITDB_DATABASE /app/scripts/drop_indexes_dataloading.js'
#!/bin/sh
set -e
# This script is meant for quick & easy install via:
# $ curl -fsSL https://get.docker.com -o get-docker.sh