Example of setting up scheduled backups

Last update: November 12 2026Open in Markdown

Objective

Create a script that will regularly launch a command-line client to archive and transfer critical data to S3.

Requirements

• a console client (in the example, S3cmd with the crontab automation tool);
• a cloud or dedicated server running Ubuntu 18.04 or higher;
• a user with access to S3.

Result

The script will create a backup of a file or directory using tar and upload the backup to S3 using s3cmd.

Steps

1. Create a script.
2. Transfer files to S3.
3. Configure flow control.
4. Check the script.
5. Optional: automate backups using crontab or Cyberduck.

1. Create a script

1. Open the home directory on your server:

   cd ~

2. Using the nano editor, create an empty file (for example, named bkupscript):

   nano bkupscript.sh

3. Start writing the backup script in a text editor with a shebang:

   #!/bin/bash

   A shebang is an interpreter directive that allows scripts or data files to be run as commands and looks like a two-character sequence: # and !. Thanks to the shebang at the beginning of the script, the shell runs the file's commands in bash.

4. Below the shebang at the top of the text file, add variables to the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3

   Where:

   • DATETIME — the timestamp to append to the resulting file name. Each file backed up in the space will have a unique name. This timestamp is generated by calling the date command and formatting the output to display the last two digits of the year (% y), two digits of the month (% m), two digits of the day (% d), hour (% H), minutes (% M), and seconds (% S);
   • SRC — the source path for the file or folder being backed up. $1 indicates that the value is taken from the first parameter passed to the script;
   • DST — the file destination. In the example, this is the name of the space where the backup is uploaded. This name will be obtained from the second parameter passed to the script, as specified in $2;
   • GIVENNAME — the user-selected name for the destination file. The resulting file name will start with GIVENNAME, followed by DATETIME. This name comes from the third parameter passed to the script, $3.

5. Add the showhelp function to the backup script to output messages if the script fails:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }

6. Collect the necessary files and combine them into a single package that can be uploaded:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   ){
   
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC
   }

   When the if statement is called, the script executes the tar command and waits for the result. If the command succeeds, the lines after the then statement will be executed:

   • outputting a message that the tar process completed successfully;
   • returning exit code 0 so that the part of the code calling this function knows everything is working properly.

   The else part of the script will execute only if the tar command encounters an error during execution:

   • outputting a message that the tar command failed;
   • returning error code 1, which indicates that something went wrong.

7. End the if/then/else script with fi. The completed tarandzip function:

   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }

2. Transfer files to S3

1. Add the movetoSpace file transfer function to the backup script to upload to the selected space. To create the command that will place backup files in the selected space, use the previously declared variables:

   movetoSpace(){
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

   Where:

   • /bin/s3cmd — calls s3cmd, a command-line tool used to manage object storage buckets;
   • put — used by s3cmd to upload data to a bucket;
   • $GIVENNAME-$DATETIME.tar.gz — the name of the backup file to be uploaded to the space. It consists of the fourth and first declared variables, followed by .tar.gz, and is created by the tarandzip function;
   • s3://$DST; — the location where the file will be uploaded;
   • s3:// — a URI scheme used to describe object storage locations on the network, and $DST; is the third variable declared earlier.

2. Add a notification confirming that the file transfer process has started:

   movetoSpace(){
       echo “\n##### MOVING TO SPACE #####\n”
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

3. Configure the output of a message regarding the command execution result:

   if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
       echo "\n##### Done moving files to s3://"$DST" #####\n"
       return 0
   else
       echo "\n##### Failed to move files to the Space #####\n"
       return 1
   fi

3. Configure flow control

If the script is configured correctly, when run, it should read the input command, assign values from it to each variable, execute the tarandzip function, and then execute movetoSpace.

If the script fails at any step, it should print the output of the showhelp function to help with troubleshooting.

Add an if / then / else conditional statement to the end of the file:

if [ ! -z "$GIVENNAME" ]; then
    if tarandzip; then
        movetoSpace
    else
        showhelp
    fi
else
    showhelp
fi

4. Check the script

1. Check the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(){
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }
   movetoSpace(){
       echo "\n##### MOVING TO SPACE #####\n"
       if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
           echo "\n##### Done moving files to s3://"$DST" #####\n"
           return 0
       else
           echo "\n##### Failed to move files to the Spac #####\n"
           return 1
       fi
   }
   
   if [ ! -z "$GIVENNAME" ]; then
       if tarandzip; then
           movetoSpace
       else
           showhelp
       fi
   else
       showhelp
   fi

2. Before exiting nano, close the file ( **Ctrl+X **) and save the changes ( **Y+Enter **).

5. Optional: automate backups

Crontab

1. Set up a cron job that will use the script for regular backups to the selected space. In this example, the backup will be performed every minute.

2. Make the script executable:

   chmod +x bkupscript.sh

3. Edit the crontab file so that the script runs every minute:

   crontab -e

4. When running the crontab -e command for the first time, you will be prompted to select an editor from the list:

   no crontab for root - using an empty one
   Select an editor.  To change later, run "select-editor".
   1. /bin/ed
   2. /bin/nano        <---- easiest
   3. /usr/bin/vim.basic
   4. /usr/bin/vim.tiny
   Choose 1-4 [2]:

5. Choose nano by default or another text editor.

6. In crontab, add the line at the end of the script:

   * * * * * ~/bkupscript.sh ~/backupthis nameofyourspace cronupload

7. Close the file ( **Ctrl+X **) and save the changes ( **Y+Enter **).

8. If you leave the cron job running without changes, a new file will be copied to the selected space every minute. If cron is working successfully, reconfigure crontab to back up files at the required interval.

Cyberduck

There is a directory on your local machine whose contents need to be periodically copied to S3. You can do this using a script and a Cron job that sends a backup to the storage every day at a specified time.

Example job (you can also use other Cyberduck options):

#!/bin/bash
SELECTEL_ID=<selectel_account>
USERNAME=<username>
PASSWORD=<password>
BACKUP_PATH=<bucket_name>/<prefix>
LOCAL_PATH=<path>

duck --upload selectel:/<bucket_name>/<prefix> <path> --existing rename --username <selectel_account>:<username> --password <password> -q -y

Specify:

• <selectel_account> — Selectel Control Panel account number;
• <username> — service user name;
• <password> — service user password. If you forgot the password, change it;
• <bucket_name> — bucket name;
• <prefix> — object prefix;
• <path> — path to the local directory.

Where:

• the --existing option of the duck command specifies what to do with files already existing in storage;
• the rename option renames an existing backup by appending the time and date to the name.

Cyberduck options

When automating backups with Cyberduck, you can use additional options:

• compare — perform differential backups:

  duck --upload  selectel:/<bucket_name>/<prefix> <path> --existing compare --username <selectel_account>:<username> --password <password>

  The uploaded backup will be compared with the existing one by size, modification date, and checksum. If the parameters differ, the old version will be deleted and the new one uploaded to storage;

• skip — only new files will be uploaded to storage (those that appeared in the folder on the local machine after the previous upload). Existing files will not be uploaded, even if they were modified on the local machine;

• overwrite — deletes the existing backup from storage and uploads a new one.