An rclone backup script for cron

rclone-cron.sh

#!/bin/bash

##############################################################################
# An rclone backup script by Jared Males ([email protected])
# 
# Copyright (C) 2018 Jared Males <[email protected]>
#
# This script is licensed under the terms of the MIT license.
# https://opensource.org/licenses/MIT
#
# Runs the 'rclone sync' command.  Designed to be used as a cron job.
#
# 1) Backup Source
#    Edit the SRC variable below to point to the directory you want to backup.
#
# 2) Backup Destination
#    Edit the DEST variable to point to the remote and location (see rclone docs).
# 
# 3) Excluding files and directories
#    Edit the EXCLUDEFILE variable below to point to a file listing files and directories to exclude.
#    See the rclone docs for the format.
#
#    Also, any directory can be excluded by adding an '.rclone-ignore' file to it without editing the exclude file.
#    This file can be empty.  You can edit the name of this file with EXIFPRESENT below.
# 
# 4) You can change the bandwidth limits by editing BWLIMT, which includes a timetable facility.  
#    See rclone docs for more info.
#
# 5) Logs:
#    -- The output of rclone is written to the location specified by LOGFILE.  This is rotated with savelog. 
#       The details of synclog can be edited. 
#    -- The log rotation, and start and stop times of this script, are written to the location specified by CRONLOG.
#       This isn't yet rotated, probably should be based on size.
#   
##############################################################################

#### rclone sync options

SRC=/home/jrmales

#---- Edit this to the desired destination
DEST=b2:backup-mx9

#---- This is the path to a file with a list of exclude rules
EXCLUDEFILE=$SRC/.rclone/excludes 

#---- Name of exclude file
# NOTE: you need "v1.39-036-g2030dc13β" or later for this to work.
EXIFPRESENT=.rclone-ignore

#---- The bandwidth time table
BWLIMIT="08:00,512 00:00,off"

#---- Don't sync brand new stuff, possible partials, etc.
MINAGE=15m

#---- [B2 Specific] number of transfers to do in parallel.  rclone docs say 32 is recommended for B2.
TRANSFERS=32

#---- Location of sync log [will be rotated with savelog]
LOGFILE=$SRC/.rclone/rclone-sync.log
LOGS='-vv --log-file='$LOGFILE

#---- Location of cron log
CRONLOG=$SRC/.rclone/rclone-cron.log


###################################################
## Locking Boilerplate from https://gist.github.com/przemoc/571091
## Included under MIT License:
###################################################

## Copyright (C) 2009 Przemyslaw Pawelczyk <[email protected]>
##
## This script is licensed under the terms of the MIT license.
## https://opensource.org/licenses/MIT
#
# Lockable script boilerplate

### HEADER ###

LOCKFILE="/tmp/`basename $0`"
LOCKFD=99

# PRIVATE
_lock()             { flock -$1 $LOCKFD; }
_no_more_locking()  { _lock u; _lock xn && rm -f $LOCKFILE; }
_prepare_locking()  { eval "exec $LOCKFD>\"$LOCKFILE\""; trap _no_more_locking EXIT; }

# ON START
_prepare_locking

# PUBLIC
exlock_now()        { _lock xn; }  # obtain an exclusive lock immediately or fail
exlock()            { _lock x; }   # obtain an exclusive lock
shlock()            { _lock s; }   # obtain a shared lock
unlock()            { _lock u; }   # drop a lock

###################################################
# End of locking code from Pawelczyk
###################################################


#make a log entry if we exit because locked
exit_on_lock()      { echo $(date -u)' | rclone-cron.sh already running.' >> $CRONLOG; exit 1; }


#Now check for lock
exlock_now || exit_on_lock
#We now have the lock.

#Rotate logs.
savelog -n -c 7 $LOGFILE >> $CRONLOG

#Log startup
echo $(date -u)' | starting rclone-cron.sh . . .' >> $CRONLOG

#Now do the sync!
rclone sync $SRC $DEST --transfers $TRANSFERS --bwlimit "$BWLIMIT" --min-age $MINAGE --exclude-from $EXCLUDEFILE --exclude-if-present $EXIFPRESENT --delete-excluded $LOGS


#log success
echo $(date -u)' | completed rclone-cron.sh.' >> $CRONLOG

#release the lock
unlock


exit

thank you so much for this script. it was very useful.

for others who may have used it or are about to use it, i encountered a few problems in setting it up. this is what i did to resolve the problems.

i originally set it up on an old test server running centos 6.9, so some problems may have occurred as a result of that.

i encountered a line ending problem as a result of editing it on a windows machine
/bin/bash^M: bad interpreter: No such file or directory
i had to edit the file in vim instead

there was also an error in the log:
Config file "/.config/rclone/rclone.conf" not found - using defaults
this is solved by specifying the config file, appending this to the rclone call:
--config [/path/to/file]

finally, i got output which indicated the savelog could not be found.
i wasn't able to source savelog for my server, probably because it is no longer supported.
i dare say when i use it on other up to date servers i will not have a problem

thanks again

ps: i also had to add --copy-links to the command to ensure that symlinks are copied into my backup
i also ended up using logrotate instead of savelog to turn over the logs
finally, be aware that this script also enables verbose logging. unless you want excessive logging detail, it may be worth experimenting with different logging levels. i ended up removing the level indicator which meant it is using the default level. all problems are logged with this
LOGS='--log-file='$LOGFILE

hope all this helps someone

Thanks @raystrach

Thanks for sharing this useful script Jared.

Perhaps you can add an 'OPTIONS' variable for adding additional options (command line switches to rclone) and/or add --fast-list to it to speed up the sync and lower cost (due to much less Class C transactions on backblaze)

https://rclone.org/docs/#fast-list
https://forum.rclone.org/t/syncing-to-backlaze-b2-extremely-slow-unless-i-enable-fast-list/17222

Can you add multiple sources to same destination? Like "src1","src2" or "src1""src2" or something similar?

@CPC-STaylor a quick read of rclone docs says no, not without adding a lot to this code. rclone sync appears to only take a single path at a time. It might be simplest to make SRC an argument and call this from a separate script.

Just created an account to say thank you for this script. I was looking around for something that was a bit more than a single line, but didn't need the complexity of rclone jobber. Modified it to utilize copy instead of sync and it does exactly what I needed.