mirror of https://github.com/pditzel/dbb.git
| INSTALL.md | ●●●●● patch | view | raw | blame | history | |
| README.md | ●●●●● patch | view | raw | blame | history | |
| TODO.md | ●●●●● patch | view | raw | blame | history | |
| config/dbb.cfg | ●●●●● patch | view | raw | blame | history | |
| config/mariadb.cfg | ●●●●● patch | view | raw | blame | history | |
| config/mongodb.cfg | ●●●●● patch | view | raw | blame | history | |
| config/postgresql.cfg | ●●●●● patch | view | raw | blame | history | |
| dbb | ●●●●● patch | view | raw | blame | history | |
| dbb-modules/mariadb | ●●●●● patch | view | raw | blame | history | |
| dbb-modules/mongodb | ●●●●● patch | view | raw | blame | history | |
| dbb-modules/postgresql | ●●●●● patch | view | raw | blame | history | |
| install.sh | ●●●●● patch | view | raw | blame | history |
INSTALL.md
New file @@ -0,0 +1,13 @@ # Installation of the script 1.) Run the install.sh - script ```bash ./install.sh ``` 2.) Edit the configfile that it fits to your environment. 3.) Make changes to the skript and some pull requests ;-).
New file @@ -0,0 +1,23 @@ # Databasebackup (dbbackup) This is a bash-script to backup your databases with the most common standard tools. ## Requirenments Important: This Script is developed with debian GNU/Linux so it should work on debian and the most derived distributions. For the backup of your databases you need a propper client to connect to your databaseserver. This script aims to backup many different databasetype eg. postgres, mysql or mongodb in one single program. So if you have to backup postgres you have to install the psql-client package. If you want to backup MySQL databases install the mysql-client package. ## Features * Runs interactive and noninteractive * Make Postgrsbackups (localy and remote) * Make MySQL/MariaDB-Backups (coming soon) * Make MongoDB-Backups (coming soon) * Optinal keep a filehistory of backups * Optinal logging to syslog * Reads configuration for postgresbackups from ~/.pgpass More Information here: [https://www.central-computer.de/datenbanken-backup/]
New file @@ -0,0 +1,26 @@ # ToDo-List Ideas for the future ## Database Options ### MariaDB / MySQL - [ ] Backup MariaDB / MySQL on the local machine - [ ] Backup MariaDB / MySQL on remote machines ### MongoDB - [ ] Backup MongoDB on the local host - [ ] Backup MongoDB on remote machines ### Oracle - [ ] Check if it possible to dump OracleDatabases - [ ] Backup the local host - [ ] Backup remote hosts ## General Functionality - [ ] Run single backuptasks in background (using "&"" and "wait") - [ ] Run script in interactive mode and tell it what db on what host with which user an pwd (<- only interactive) (using a script alias and do interactiv or noninteractive depending on the calling name) - [x] Adding an optional filehistory of the backuped files/dumps (specify an timerange or a number how many dumps should be saved) config/dbb.cfg
New file @@ -0,0 +1,40 @@ ################################################################################ # Configure the einvironment: ################################################################################ # ################################################################################ # Should dbbackup log to syslog (TRUE | FALSE) # Default is TURE ENABLE_SYSLOG=TRUE ################################################################################ # Enable debugout # Default is FALSE ENABLE_DEBUG=TRUE ################################################################################ # Where to backup datafiles? BACKUP_DIR=/home/backupuser/database_backups ################################################################################ # We need a tmpdir TMP_DIR=/tmp/databases ################################################################################ # Which user is running dbbackup? BACKUPUSER=backupuser ################################################################################ # Should a history of backuped files kept? KEEP_BACKUP_FILES=TRUE # How many days should these files kept? BACKUP_FILES_DAYS=14
New file @@ -0,0 +1,11 @@ ################################################################################ # ToDo MYSQL_BACKUP_LOCAL=FALSE ################################################################################ # ToDo MYSQL_BACKUP_REMOTE=FALSE ################################################################################ config/mongodb.cfg
New file @@ -0,0 +1,11 @@ ################################################################################ # ToDo MONGODB_BACKUP_LOCAL=FALSE ################################################################################ # ToDo MONGODB_BACKUP_REMOTE=FALSE ################################################################################ config/postgresql.cfg
New file @@ -0,0 +1,19 @@ ################################################################################ # Backup Postgres @ localhost as user postres (Debian) POSTGRES_BACKUP_LOCAL=TRUE # Should every database backuped in a single file? # Works together with POSTGRES_BACKUP_LOCAL_DUMP_ALL POSTGRES_BACKUP_LOCAL_SINGLE=TRUE # Should all databases backuped in one large file? # Works together with POSTGRES_BACKUP_LOCAL_SINGLE POSTGRES_BACKUP_LOCAL_DUMP_ALL=TRUE ################################################################################ ## Enable backup of remote postgres databases specified in ~/.pgpass POSTGRES_BACKUP_REMOTE=TRUE dbb
New file @@ -0,0 +1,234 @@ #!/bin/bash ################################################################################ # # # Author: Patrick Ditzel (patrick@central-computer.de) # # Lizenz: GNU GENERAL PUBLIC LICENSE v3 # # # ################################################################################ ################################################################################ # Set up the backupenvironment and "global" functions ################################################################################ INSTALLATION_PATH_PREFIX=/usr/local function set_config { if [ -r /etc/dbb.cfg ]; then # Check if configuration is found in /etc. If yes set configvar BACKUPCFG=/etc/dbb.cfg elif [ -r ~/.dbb.cfg ]; then # If config is found in the backupuser home directory set it into the configvar BACKUPCFG=~/.dbb.cfg else echo "No configuration file is found, please create one" | /usr/bin/logger -s -i -t databasebackup fi } function load_dbmodul_dfg { if [ -d $BACKUPCFG.d ]; then if [ ! "$(ls -A $BACKUPCFG.d)" ]; then echo "Configurationdirectory for modules exist but it is empty" | /usr/bin/logger -s -i -t databasebackup else for MODULCFG in $(ls $BACKUPCFG.d); do . $BACKUPCFG.d/MODULCFG; done fi else echo "Configurationdirectory does nort exists, can't load any configurationfile" | /usr/bin/logger -s -i -t databasebackup fi } function check_backup_env { # Check if the configuration exists and is not empty if [ -r $BACKUPCFG ] && [ -s $BACKUPCFG ]; then # If true then read it . $BACKUPCFG load_dbmodul_dfg else # If not throw an errormessage echo "The configfile does not exists or is empty" | /usr/bin/logger -s -i -t databasebackup echo "Please create the $BACKUPCFG or write your settings into" | /usr/bin/logger -s -i -t databasebackup exit 1 fi if [ "$ENABLE_DEBUG" = "TRUE" ]; then # If debugoutput is enabled show the configurationfile without comments echo "################################################################################" | /usr/bin/logger -s -i -t databasebackup while read -r configline; do echo "$configline" | grep -v '^$' | grep -v '^#' | /usr/bin/logger -s -i -t databasebackup done <<< "$(cat $BACKUPCFG)" echo "################################################################################" | /usr/bin/logger -s -i -t databasebackup fi # Check if the target directory for the backupfiles exists if [ ! -d "$BACKUP_DIR" ]; then # If not create it mkdir -p "$BACKUP_DIR" fi # Check if the directory for tempfiles exists if [ ! -d "$TMP_DIR" ]; then # If not create it mkdir -p "$TMP_DIR" fi } function set_logger { # Check if log to syslog is enabled if [ "$ENABLE_SYSLOG" = "TRUE" ]; then # If true then define the logger LOGGER="/usr/bin/logger -s -i -t databasebackup" else # If not cat it out on stdout LOGGER="/bin/cat" fi } function debug { DEBUGMSG=$1 if [ "$ENABLE_DEBUG" = "TRUE" ]; then echo "$DEBUGMSG" | $LOGGER fi } function load_dbmodules { if [ -d $BACKUPCFG.d ] && [ -d $INSTALLATION_PATH_PREFIX/lib/dbb-modules ]; then if [ ! "$(ls -A $BACKUPCFG.d)" ] && [ ! "$(ls -A $INSTALLATION_PATH_PREFIX/lib/dbb-modules)" ]; then debug "Configurationdirectory or directory for modules exist but it is empty" else for MODUL in $(ls $BACKUPCFG.d | cut -d "." -f1); do . $INSTALLATION_PATH_PREFIX/lib/dbb-modules/$MODUL debug "Load module: $INSTALLATION_PATH_PREFIX/lib/dbb-modules/$MODUL" check_$MODUL_deps done fi else debug "Configurationdirectory does nort exists, can't load any configurationfile" fi } function is_interactive { SCRPT=$(basename "$0") debug "$SCRPT" if [ "$SCRPT" = "dbbi" ]; then # If the script is called for interactive use we have to chenge the calls of the functions # Setting the $LOGGER für interactive use to /bin/cat LOGGER="/bin/cat" # Here we set teh environment variable for interactive use debug "dbbi (DataBase Interactive Backup) is called" RADIOACTIVE=TRUE elif [ "$SCRPT" = "dbb" ]; then # Set the $LOGGER LOGGER="/usr/bin/logger -s -i -t databasebackup" # If the script is used noninteractive we have also to set the environmet variable with this information debug "dbb (DataBase Backup) is called" RADIOACTIVE=FALSE # unset the $LOGGER because this will be set later within the informatione in the configfile unset LOGGER else # If the switch between interactive and noninteractive does not work: tell it but before set the $LOGGER LOGGER="/usr/bin/logger -s -i -t databasebackup" debug "An error occured - don't know if to use interactive or noninteractive" exit 1 fi } function backup_file_handler { # translate the vars to make it more readable BFH_TMP_DIR=$1 BFH_BACKUPDIR_DIR=$2 BFH_FILE=$3 # If enabled put out some debug infos debug "BFH_TMP_DIR: $BFH_TMP_DIR" debug "BFH_BACKUPDIR_DIR: $BFH_BACKUPDIR_DIR" debug "FILE: $BFH_FILE" # Check if the script should keep a filehistorie if [ "$KEEP_BACKUP_FILES" = "TRUE" ]; then debug "Keep history" # Set some vars to manage the files BACKUP_DAY=$(date +%x) REMOVE_NUMBER=$(( $BACKUP_FILES_DAYS + 1 )) BFH_FILE_PREFIX_NAME_TO_REMOVE=$(date -d "$REMOVE_NUMBER days ago" "+%x") # ... and if it is turned on give some debig info debug "BACKUP_DAY: $BACKUP_DAY" debug "REMOVE_NUMBER: $REMOVE_NUMBER" debug "FILE_PREFIX_NAME_TO_REMOVE: $BFH_FILE_PREFIX_NAME_TO_REMOVE-$BFH_FILE" # Check if there is an backupfile from the current day if [ -f "$BFH_BACKUPDIR_DIR"/"$BACKUP_DAY"-"$BFH_FILE" ]; then # If yes append miniutes and seconds to the date-profix of the filename debug "File $BFH_BACKUPDIR_DIR/$BACKUP_DAY-$BFH_FILE already exists. Rename the new one." DATE_TIME_SUFFIX=$(date +%H%M%S) # ... and move it into the targetdir mv "$BFH_TMP_DIR"/"$BFH_FILE" "$BFH_BACKUPDIR_DIR"/"$BACKUP_DAY"-"$DATE_TIME_SUFFIX"-"$BFH_FILE" else # If there is no backupfile of the current day move it to the backupfolder mv "$BFH_TMP_DIR"/"$BFH_FILE" "$BFH_BACKUPDIR_DIR"/"$BACKUP_DAY"-"$BFH_FILE" fi # Check if there are files older then the days to keep set in the config if [ -f "$BFH_BACKUPDIR_DIR"/"$BFH_FILE_PREFIX_NAME_TO_REMOVE"-"$BFH_FILE" ]; then # if yes remove it rm "$BFH_BACKUPDIR_DIR"/"$BFH_FILE_PREFIX_NAME_TO_REMOVE"-"$BFH_FILE" # Also remove the files with the extended prefix in the name # If there is ab file with the extende prefix then there has to be a file with tne normal prefix rm "$BFH_BACKUPDIR_DIR"/"$BFH_FILE_PREFIX_NAME_TO_REMOVE"?????-"$BFH_FILE" else # If no file exists do nothing but some debuginfo debug "File $BFH_BACKUPDIR_DIR/$BFH_FILE_PREFIX_NAME_TO_REMOVE-$BFH_FILE does not exists, so can not remove it." fi else # If we do not keep a filehistory do the following # Check if the targefile exists if [ -f "$BFH_BACKUPDIR_DIR"/"$BFH_FILE" ]; then debug "$BFH_FILE exists ... make a diff" # Check if there are differences betwenn last backup and the actual one diff "$BFH_TMP_DIR"/"$BFH_FILE" "$BFH_BACKUPDIR_DIR"/"$BFH_FILE" > /dev/null 2>&1 if [ $? -ne 0 ]; then # If yes then move it to the backupfolder debug "Differences found between old and new $BFH_FILE -> moving to BACKUP_DIR" mv "$BFH_TMP_DIR"/"$BFH_FILE" "$BFH_BACKUPDIR_DIR"/"$BFH_FILE" else # If not do nothing debug "No differences found between old an new $BFH_FILE" fi else # If there is a new databasedumpfile move it to the backupfolder debug "New Backupfile $BFH_FILE -> moving to $BFH_BACKUPDIR_DIR" mv "$BFH_TMP_DIR"/"$BFH_FILE" "$BFH_BACKUPDIR_DIR"/"$BFH_FILE" fi fi } function check_global_deps { # Check the dependencys if [ ! -e /usr/bin/sudo ]; then echo "It seems that you dont have sudo installed. Please install sudo and restart" | /usr/bin/logger -s -i -t databasebackup exit 1 fi } ################################################################################ # The mainfunktion function main { check_global_deps is_interactive if [ "$RADIOACTIVE" = "TRUE" ]; then debug "Unsing dbbi (dbb interactive = dbbi) is for future use" fi if [ "$RADIOACTIVE" = "FALSE" ]; then debug "running noninteractive" # Set up the configuration for the noninteractive mode set_config # Configure logging (from configurationfil)e set_logger # Check if the backupenvironment is setup properly check_backup_env # Run modul-main-functions for MODULMAIN in $(ls $BACKUPCFG.d | cut -d "." -f1); do $MODULMAIN_main done # The final action: remove the dumps rm -rf "$TMP_DIR" fi } main dbb-modules/mariadb
New file @@ -0,0 +1,19 @@ #!/bin/bash # MariaBD/MySQL remote # # The Information what MySQL/MariaDB - databases should be backuped are defined in ~/.my.cnf function run_mysql_backups { debug "Dump remote database $MYSQLDB from $MYSQLDBHOST" /usr/bin/mysqldump --skip-dump-date -h "$MYSQLDBHOST" -u "$MYSQLDBUSER" -p"$MYSQLPASSWD" "$MYSQLDB" > "$TMP_DIR"/"$MYSQLDBHOST"_"$MYSQLDB".mysql debug "Diff MySQLDump $MYSQLDB" diff "$TMP_DIR"/"$MYSQLDBHOST"_"$MYSQLDB".mysql "$BACKUP_DIR"/"$MYSQLDBHOST"_"$MYSQLDB".mysql > /dev/null 2>&1 if [ $? -ne 0 ]; then debug "Differences found -> moving to BACKUP_DIR" mv "$TMP_DIR"/"$MYSQLDBHOST"_"$MYSQLDB".mysql "$BACKUP_DIR"/"$MYSQLDBHOST"_"$MYSQLDB".mysql else debug "No differences found an Database $MYSQLDB" fi echo "" } dbb-modules/mongodb
New file @@ -0,0 +1,5 @@ #!/bin/bash function run_mongodb_backup { echo "" } dbb-modules/postgresql
New file @@ -0,0 +1,138 @@ #!/bin/bash # dependencies function check_postgresql_deps { if [ ! -e /usr/bin/psql ]; then debug "It seems that you dont have psql installed. You may have problems to backup remote databases" fi numberRegex='^[0-9]+$' if ! [[ $(id -u postgres 2>&1) =~ $numberRegex ]] ; then debug "It seems that you dont have the user postgres on your computer. You may have problems to backup local postgres databases" fi } # PostgreSQL local function pg_local_all { # Debugoutput id DEBUG is enabled debug "Dump all from Postgres local" # Dump all databases in one file as user postgres (Debian) sudo -H -i -u postgres pg_dumpall > "$TMP_DIR"/local_dumpall.pgsql debug "Diff alldumps from Postgres local" # Check if there are differences between the actual dump and the last dump backup_file_handler "$TMP_DIR $BACKUP_DIR" local_dumpall.pgsql } function pg_local_single { # Get a list with all databases on localhost. Do it as user postgres (Debian) # parallelize the following for DB in $(sudo -H -i -u postgres /usr/bin/psql -At -c "SELECT datname FROM pg_database WHERE NOT datistemplate AND datallowconn ORDER BY datname;" postgres) do debug "Dumping local database $DB" # Dump each database in a single file sudo -H -i -u postgres /usr/bin/pg_dump -Fp "$DB" > "$TMP_DIR"/local_"$DB".pgsql backup_file_handler "$TMP_DIR" "$BACKUP_DIR" local_"$DB".pgsql echo "" done } # Do the local backupjobs function run_local_pg_backups { # Test if the current backupuser has access via sudo to postgres # This part has to be improved! sudo -l -U "$BACKUPUSER" | grep postgres if [ $? -eq 0 ];then # If the user has access vi sudo to postgres and a full dump is configured, do it. if [ "$POSTGRES_BACKUP_LOCAL_DUMP_ALL" == "TRUE" ]; then pg_local_all fi # If the user has access vi sudo to postgres and dumps for each database is configured , do it. if [ "$POSTGRES_BACKUP_LOCAL_SINGLE" == "TRUE" ]; then pg_local_single fi else # If the user is not permitted to acces the postgresdatabases vi sudo throw an errormessage echo "" | $LOGGER echo "The backupuser does not the permission to act as user postgres" | $LOGGER echo "Please add the following line to your /etc/sudoers:" | $LOGGER echo "backupuser ALL=(postgres) NOPASSWD: /usr/bin/psql,/usr/bin/pg_dump" | $LOGGER echo "" | $LOGGER fi } ################################################################################ # Postgres remote # # All databases on remotehosts defined in the ~/.pgpass file will be backuped. # So the ~/.pgpass is the configurationfile for this part! function dump_remote_pgdb { # Translate params ;-) TRGTHOST=$1 TRGTPORT=$2 TRGTDB=$3 TRGTBDUSER=$4 # If debug is enabled, check the translated params debug "PostgreSQL:" debug " Host: $TRGTHOST" debug " Port: $TRGTPORT" debug " Database: $TRGTDB" debug " User: $TRGTBDUSER" debug "Testing TLS-Connection" # Check if the connection to the postgres-server are encryptet (here we force with sslmode=require) psql -U "$TRGTBDUSER" postgresql://"$TRGTHOST":"$TRGTPORT"/"$TRGTDB"?sslmode=require -c '\conninfo' | grep TLS > /dev/null 2>&1 if [ "$?" -eq "0" ]; then debug "Dumping remote database $TRGTHOST-$TRGTDB" # If we successfuly testet the encrypted connection to the postgres-server we try to force the sslmode # I don't know if the following statement really effect to pg_dump :-( export PGSSLMODE=require # Dump the databases which are defined in the params /usr/bin/pg_dump -U "$TRGTBDUSER" -h "$TRGTHOST" -p "$TRGTPORT" "$TRGTDB" > "$TMP_DIR"/"$TRGTHOST"-"$TRGTDB".pgql backup_file_handler "$TMP_DIR" "$BACKUP_DIR" "$TRGTHOST"-"$TRGTDB".pgql else # If no encrypted connection to the postgres-server can be established throw an errormessage echo "" | $LOGGER echo "Could not establish a TLS encrypted connection the the databasehost." | $LOGGER echo "Please configure the connections with hostssl in pg_hba.conf." | $LOGGER echo "" | $LOGGER fi echo "" } function run_remote_pg_backups { # Check if the remoebackup for postgres is configured if [ "$POSTGRES_BACKUP_REMOTE" = "TRUE" ]; then # If yes the check for the ~/.pgpass-file. Here are the remotedatabases specified if [ -r ~/.pgpass ]; then # parallelize the following while read -r LINE; do # For each entry do the backup debug "run dump with params $LINE" # CAUTION: No doublequotes in the following line. The var $LINE has to be splittet! dump_remote_pgdb $LINE # To get the params for the function the .pgpass-file is striped from the comments, # the ":" are replaces against whitespaces and only the first four coloums are used # so we give "host port database user" to the function done <<< "$(cat ~/.pgpass | grep -v '#' | tr ":" " " | cut -d " " -f1,2,3,4)" else # If the ~/.pgpass-file is missig, throw an errormessage echo "" | $LOGGER echo "The ~/.pgpass file is missing, no remote postgres databases will be backuped." | $LOGGER echo "If you want do backup postgres reomte databases, please create a ~/.pgpass file in the homedirectory of your backupuser (https://wiki.postgresql.org/wiki/Pgpass)." | $LOGGER echo "" | $LOGGER fi fi echo "" } function postgres_main { if [ "$POSTGRES_BACKUP_LOCAL" = "TRUE" ]; then run_local_pg_backups fi if [ "$POSTGRES_BACKUP_REMOTE" = "TRUE" ]; then run_remote_pg_backups fi } install.sh
New file @@ -0,0 +1,11 @@ #/bin/bash # ToDo # # 1. Set installationpath (e.g. /usr/ or /usr/local/) => INSTPATH # 2. Generate directories # - $INSTPATH/lib/dbb # 3. Check if it is an local user installation odr a global system installation # 4. If it is a global installation mkdit /etc/dbb.cfg.d an cop dbb.cf to /etc # If it is an local user installation create ~/.dbb.cfg.d an cop dbb.cfg to ~/.ddb.cfg # 5. Copy dbb to $INSTALLPATH/bin/dbb and create symlink from $INSTALLPATH/bin/dbb to $INSTALLPATH/bin/dbbi
