Example of configuring scheduled backup
Purpose of customization
Create a script that will regularly run the console client, archive and move important data to object storage.
What you need to customize
- console client (in the example S3cmd with crontab automation tool);
- cloud or dedicated server with Ubuntu version 18.04 or higher installed;
- user with access to object storage.
Result of customization
The script will create a backup of the file or directory using tar and upload the backup to the object store using s3cmd.
Steps to customize
- Create script.
- Transfer files to object storage.
- Configure flow-control.
- Check script.
- Optional: automate backups via crontab or Cyberduck.
1. Create a script
-
Open the home directory on your server:
cd ~
-
Use the nano editor to create an empty file (for example, named
bkupscript
):nano bkupscript.sh
-
Start writing a backup script in a text editor with shebang:
#!/bin/bash
Shebang is an interpreter directive that allows scripts or data files to be run as commands and looks like a sequence of two characters:
#
and!
. Thanks to the shebang at the beginning of the script, the shell runs the file's commands in bash. -
Under shebang at the top of the text file, add variables to the script:
#!/bin/bash
DATETIME=`date +%y%m%d-%H_%M_%S`
SRC=$1
DST=$2
GIVENNAME=$3Here:
DATETIME
— the timestamp to attach to the received file name. Each file backed up in the space will have a unique name. This timestamp is created by calling thedate
command and formatting the output to display the last two digits of the year (% y
), the last two digits of the month (% m
), the last two digits of the day (% d
), the hour (% H
), the minute (% M
), and the second (% S
);SRC
— the source path for the file or folder to which the backup is being made.$1
indicates that the value is taken from the first parameter passed to the script;DST
is the destination of the file. In the example, this is the name of the space to which the backup is loaded. This name will be obtained from the second parameter passed to the script, as indicated by$2
;GIVENNAME
— user-selected name for the destination file. The resulting file name will start withGIVENNAME
, andDATETIME
will be appended to it. This name comes from the third parameter passed to the$3
script.
-
Add a
showhelp
function to the backup script to display messages if the script fails:#!/bin/bash
DATETIME=`date +%y%m%d-%H_%M_%S`
SRC=$1
DST=$2
GIVENNAME=$3
showhelp(
echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
} -
Gather the files you need and combine them into a single package that can be downloaded:
#!/bin/bash
DATETIME=`date +%y%m%d-%H_%M_%S`
SRC=$1
DST=$2
GIVENNAME=$3
showhelp(
echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
){
}
tarandzip(){
echo "\n##### Gathering files #####\n"
tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC
}When the
if
instruction is called, the script executes thetar
instruction and waits for the result. If the command is successful, the lines after thethen
statement will be executed:- output a message that the
tar
process has successfully terminated; - return error code
0
so that the part of the code calling this function knows that everything is working fine.
The
else
part of the script will only be executed if thetar
command detects an execution error:- output a message that the
tar
command was not executed; - returning error code 1, indicating that something has gone wrong.
- output a message that the
-
End the
if/then/else
script with thefi
phrase. View of the completedtarandzip
function:tarandzip(){
echo "\n##### Gathering files #####\n"
if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
echo "\n##### Done gathering files #####\n"
return 0
else
echo "\n##### Failed to gather files #####\n"
return 1
fi
}
2. Transfer files to object storage
-
Add a function to the backup script to transfer the
movetoSpace
file to the selected space. The previously declared variables are used to create a command that will place the backup files in the selected space:movetoSpace(){
/bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
}Here:
/bin/s3cmd
— calls s3cmd, a command line tool used to manage object store segments;put
— used by s3cmd to load data into the container;$ GIVENNAME- $ DATETIME.tar.gz
is the name of the backup that will be loaded into the space. It consists of the fourth and first declared variables followed by.tar.gz
, and is created by thetarandzip
function;s3: // $ DST ;
— the location where the file will be uploaded;s3: //
is a schema of type URI used to describe where objects are stored on the network, and$DST;
is the third variable declared earlier.
-
Add a notification that the file transfer process has started:
movetoSpace(){
echo "\n##### MOVING TO SPACE #####\n"
/bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
} -
Customize the output of the message about the result of command execution:
if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
echo "\n##### Done moving files to s3://"$DST" #####\n"
return 0
else
echo "\n##### Failed to move files to the space #####\n"
return 1
fi
3. Customize flow control
If the script is configured correctly, when it starts, it should read the input command, assign values from it to each variable, execute the tarandzip
function, and then execute movetoSpace
.
If the script fails at any step, it should print the output of the showhelp
function to help troubleshoot the problem.
At the end of the file, add a conditional instruction if / then / else
:
if [ ! ! -z "$GIVENNAME" ]; then
if tarandzip; then
movetoSpace
else
showhelp
fi
else
showhelp
fi
4. Check the script
-
Check the script:
#!/bin/bash
DATETIME=`date +%y%m%d-%H_%M_%S`
SRC=$1
DST=$2
GIVENNAME=$3
showhelp(){
echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
}
tarandzip(){
echo "\n##### Gathering files #####\n"
if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
echo "\n##### Done gathering files #####\n"
return 0
else
echo "\n##### Failed to gather files #####\n"
return 1
fi
}
movetoSpace(){
echo "\n##### MOVING TO SPACE #####\n"
if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
echo "\n##### Done moving files to s3://"$DST" #####\n"
return 0
else
echo "\n##### Failed to move files to the Spac #####\n"
return 1
fi
}
if [ ! -z "$GIVENNAME" ]; then
if tarandzip; then
movetoSpace
else
showhelp
fi
else
showhelp
fi -
Before exiting nano, close the file (Ctrl+X) and save your changes (Y+Enter).
5. Optional: automate backups
- Crontab
- Cyberduck
-
Set up a cron job that will use a script to make regular backups to the selected space. For the purposes of this example, backups will be performed every minute.
-
Make the script executable:
chmod +x bkupscript.sh
-
Edit the crontab file so that the script runs every minute:
crontab -e
-
The first time you run the
crontab -e
command, you will be prompted to select an editor from the list:no crontab for root - using an empty one
Select an editor. To change later, run "select-editor".
1. /bin/ed
2. /bin/nano <---- easiest
3. /usr/bin/vim.basic
4. /usr/bin/vim.tiny
Choose 1-4 [2]: -
Choose the default nano or another text editor.
-
In crontab, add a line at the end of the script:
* * * * * * * ~/bkupscript.sh ~/backupthis nameofyourspace cronupload
-
Close the file (Ctrl+X) and save your changes (Y+Enter).
-
If you leave the cron job running unchanged, a new file will be copied to the selected space every minute. If cron is running successfully, reconfigure crontab to back up files at the desired interval.
On the local machine, there is a directory whose contents need to be periodically copied to object storage. This can be done with a script and a Cron job that sends a backup to the repository every day at a specified time.
Example job (additionally you can use other Cyberduck options):
#!/bin/bash
SELECTEL_ID=<selectel_account>
USERNAME=<username>
PASSWORD=<password>
BACKUP_PATH=<container_name>/<prefix>
LOCAL_PATH=<path>
duck --upload selectel:/<container_name>/<prefix> <path> --existing rename --username <selectel_account>:<username> --password <password> -q -y
Specify:
<selectel_account>
— Selectel Control Panel account number;<username>
— service user name;<password>
— service user password. If you forget your password, create a new one;<container_name>
is the name of the container;<prefix>
— object prefix;<path>
— the path to the local directory.
Here:
- The
--existing
key of theduck
command specifies what to do with files already in the repository; - The
rename
option renames an existing backup, adding the time and date to the name.
Cyberduck Options
When automate backups via Cyberduck, you can use additional options:
-
compare
— perform differential backups:duck --upload selectel:/<container_name>/<prefix> <path> --existing compare --username <selectel_account>:<username> --password <password>
The uploaded backup will be compared to the existing backup by size, modification date, and checksum. If the parameters are different, the old version will be deleted and the new version will be uploaded to the repository;
-
skip
— only new files (those that appeared in the folder on the local machine after the previous upload) will be uploaded to the storage. Already existing files will not be downloaded, even if they have been modified on the local machine; -
overwrite
— deletes an existing backup from the storage and loads a new one.