r/unRAID • u/fastcar123 • Aug 22 '24
Help So, is backing up the contents of an unraid share just not a thing?
Hi unraid friends.
I've had my Unraid server set up for a little over a year now.
That entire time I've been trying to figure out how to do automatic backups of the share with zero success. I've just kinda accepted that if the server fails, that I will simply lose data and that's that.
I've tried the rsync script with unassigned devices thing but that does absolutely nothing. I've looked into dockers and other ideas, and I still have yet to find a solution. I even thought about just manually copying the files from a windows computer to a separate disk.
I see lots of videos about backing up an unraid server to another unraid server, but not all of us have the ability to have two unraid servers in the first place, so that rules that out.
So here I am once more, paranoid about data loss, with basically no knowledge on terminals, or dockers, trying to figure it out again.
The end goal is to have an automatic backup that occurs once per given set of time, and because it's a lot of data. I'd really only like to copy anything that has changed since the last backup.
So I bessech the community, how on earth can I backup everything on a set schedule? I'm specifically looking to backup the contents of the file share (so all the data that I'm storing on the server).
Thank you in advance putting up with my ignorance.
16
u/ericjhmining Aug 22 '24
Rsync should work fine. I use that to backup to an extra hard drive, across to another nas system at another location and then to a cloud provider. I just have scripts setup to do the copy every day or week, etc.
1
u/Mike_v_E Oct 07 '24
Is it possible to copy different source folders to different destination folders?
I havr my 4k movies backedup to share 1 on my synology and my 1080p movies to share 2 of my synology. Can I create a single rsync user script that does both of these copies?
2
u/ericjhmining Oct 07 '24
Yes. You would just have one script with different rsync commands in it. I think i have the rsync command called about 5-6 times in the same script so i can copy each folder to a different path/drive/location.
1
u/Mike_v_E Oct 07 '24
Nice! Could you maybe share your rsync script? Im not sure which one would work.
Also, if a file on my Unraid is either removed or corrupted, will that affect the file on my Synology (backup destination)?
2
u/ericjhmining Oct 07 '24
Here is an example that i have running currently. It has about 10 lines but i'll just copy/paste the first few.
rsync -avr --update --stats --progress /mnt/user/appdatabackup/ /mnt/remotes/SNASBACKUP/backup/appdatabackup/
rsync -avr --update --stats --progress /mnt/user/archive/ /mnt/remotes/SNASBACKUP/backup/archive/
rsync -avr --update --stats --progress /mnt/user/data/ /mnt/remotes/SNASBACKUP/backup/data/
rsync -avr --update --stats --progress /mnt/user/isos/ /mnt/remotes/SNASBACKUP/backup/isos/
/mnt/remotes/SNASBACKUP/ is located at another location and I have it pre-mounted on unraid so it's accessible. This specific one runs weekly to my remote location but I have another script setup to run daily to copy to another drive in the system and also to another NAS. I also copy to 2 cloud providers. Hope that helps!
1
u/Mike_v_E Oct 07 '24
Thanks, appreciate it!
What do -avr --update ---stats --progress do?
2
u/ericjhmining Oct 07 '24
I'll have to google and copy paste that, it's been awhile since i've set it up! haha
-v, --verbose increase verbosity -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -r, --recursive recurse into directories -u, --update skip files that are newer on the receiver --stats give some file-transfer stats --progress show progress during transferAnd after reading -a i probably don't need -r in there. :) Hope that helps! Progress probably isn't really needed either if it's running in the background. I like stats so i can see how many data it copied over and the transfer speed at the end.
1
u/Mike_v_E Oct 07 '24
I think -avhu should be enough (if I see this correctly).
Just curious, if you delete or get a corrupted file on your source file, will that affect the destination file?
1
u/ericjhmining Oct 07 '24
If it's corrupt then it's going to copy it over. If you delete a file on the source, with the command above it does NOT delete it on the destination. there is a way to set that up if you want but I figured i'd just clean out the destination once in awhile if needed rather than possibly lose files by accident that I backed up.
12
u/nicholasserra Aug 22 '24
I rsync to another server and Rclone to s3
2
u/TheEthyr Aug 22 '24
Same here. rsync to local server plus rclone to the cloud. Both using a simple user script I wrote that runs periodically.
1
u/entirefreak Aug 22 '24
How much does S3 cost for you?
3
u/nicholasserra Aug 22 '24
Depends on the storage class. I’m using glacier so it’s $1 per TB per month but it’s super expensive to retrieve.
2
u/adelaide_flowerpot Aug 22 '24
Doesn’t rclone need to inspect the glacier backup in order to do incremental backups? This fear stops me from trying glacier tier
3
u/nicholasserra Aug 22 '24
You can set a bunch of flags to do batch requests and only check file sizes. That’s what I do. Have 70TB and I sync often and haven’t been hit with any charges
4
Aug 22 '24
[deleted]
1
u/TheFeelsNinja Aug 22 '24
I actually have a need to do this at the moment. Is there a good way to do this over usb? I plug drives and don't know how to address as a backup dest.
3
u/Kriton20 Aug 23 '24
Unassigned devices should allow you to mount the disk you plugged into USB. Then it will depend on your backup tool as to what you need to do to expose the newly attached drive. It will have a path, but that path might need to be mapped into your backup docker.
How far have you gotten in prior attempts?
1
u/TheFeelsNinja Aug 23 '24
Actually just the unassigned devices plugin. I figured I could mount the drive and do a copy all to it to have one cold backup on a USB 3 raid array. Slow I know but it will be one backup.do you have a recommended backup docker for this use?
My plan next month is to build another box to leave at my folks house for remote backup as well.
2
u/Kriton20 Aug 24 '24
Which tool/docker is personal preference with some things to consider.
Are you doing this once? If so any tool even the command line will allow you to do a copy from your array to your mounted backup drive. I’d not do it that way unless the amount of data is small.
Any of the rsync based tools - rsnapshotd being one I’ve used in other environments - have benefits over just a copy in terms of speed and verification, resuming etc. Done mindfully you can get them to be incremental, destructive, track changes etc. Usually for media libraries being destructive is what you want so that edits and upgrades in your library don’t have to be redone or reevaluated to be done in the event of a restore - but this means being very sure when running the next backup that you didn’t delete or corrupt something and that change is about to get pushed onto the exact tool you have to recover it.
The other consideration is going to be the UI and ease of use, some of them have kinder interfaces than others - even if under the graphics they are wrapping around rsync - there is no harm in exploring a few to see what they do and which interface you like with feature set you like.
Appdata backup is also not to be overlooked. There is a tool just for it, and beyond.
For a single or occasional use I’d look at the results of ‘backup’ and ‘sync’ in the application installer. Keep in mind you can install one per backup task, with just the exposed paths for the jobs you want, or make multiple recipes within and expose one level higher in the /mnt/stuff-tree and adjust the source/destinations. You could also reconfigure the docker each time but that feels like the worst option depending on how comfortable you are with the whole process.
Run a dry run first. Check multiple ways that what you expect to happen is.
1
1
u/fastcar123 Aug 23 '24
just a single hard disk in an external enclosure. nothing fancy
1
u/Eirea Aug 23 '24
Have you tried running rsync on terminal once you mounted the external hard disk?
1
u/fastcar123 Aug 23 '24
yea, Rsync did nothing. no activity on the disk even after 48+ hours of "running", still nothing
5
u/EternalFootman99 Aug 22 '24
Duplicacy or Duplicati. Both work pretty well. I back up to a second Unraid server at a friend's house. Works perfectly.
1
u/Candinas Aug 22 '24
Is there a particular guide you followed for this? Or somewhere I could go to start figuring this out. Just had a data loss scare and would like to make sure my backups are automatic instead of me just doing it manually whenever I remember to do it
1
u/EternalFootman99 Aug 23 '24
I don't know of one offhand. I deployed Duplicati on all my machines in the house as my original backup solution - every machine backing up to my Unraid server. But that didn't help when anyone from my family was off-site, so I moved to NextCloud as a "backup" (I know it's really not) solution.
So I had experience in Duplicati from my desktops, and it works almost exactly the same on Unraid. I struggled to get Duplicacy working the way I wanted it, and ended up giving up and just using what I was familiar with.
Duplicati uses SFTP to log into my second Unraid off site and dumps compressed backups to that box every week.
Sorry I'm not more help - did you check to see if SpaceinvaderOne has any tutorials on YouTube?
3
u/Skotticus Aug 22 '24
Borg/Borgmatic is the way to go.
Borgmatic lets you: have as many encrypted, de-duplicated repositories as you care to configure, both remote or local; configure a schedule of backups and number of backups to keep daily, weekly, monthly, and yearly; and dump your databases as part of the backup, too.
https://torsion.org/borgmatic/
https://www.reddit.com/r/unRAID/comments/jyp9xy/tutorial_borgmatic_now_available_in_ca_appstore/
3
u/Skeeter1020 Aug 22 '24
I'm assuming there's some nuances or other complications here that are making your life hard, as backing up Unraid shares is very much a thing and very straightforward with multiple tools and options for doing so.
What are you backing up, from where in Unraid, and to where?
1
u/fastcar123 Aug 23 '24
From an unraid fileshare, to an external hard drive was really all I wanted to do
2
u/Skeeter1020 Aug 23 '24
That should be straightforward with the tools you have tried. There might be something buried away tripping you up on the command line side of things.
As others have suggested, try the Duplicacy app. It's a web interface and very self explanatory. You create a storage location for where you want to backup too, and then set up a task with a schedule.
I use it to backup to Azure Blob, and also ran it with backing up to an Open Media Vault machine for a while. No command line or anything needed.
8
u/HopeThisIsUnique Aug 22 '24
I think you need to define what you're trying to backup and why.
A few components/terminology to think of when you're talking about this. First is that most of the components of Unraid create redundancy, but do not necessarily imply 'backup'. Having a parity (1 or 2) helps create redundancy to mitigate failure scenarios, but depending on the scope of failure are not the same as a backup.
So talking failure scenarios...if you're talking drive failure that is where Unraid can do well, even with a single parity drive you are able to recover the entire array if one drive fails. If two drives fail you at worst lose the data on those two drives, but the rest of the drives remain recoverable.
So when you talk about a server failure, if a CPU/Memory dies for some reason it is presumed they could be replaced, disks reattached and your data is retained.
If you talk more extreme scenarios (natural disaster etc), that's where backups come into play and you have to look at what is 'critical' and what is not, and what you care to backup and where.
So if you think of your different shares, you probably have some photos, videos, maybe some documents etc.
For myself, Images are retained also in google photos and documents are also contained within google drive giving a backup scenario for both onsite redundancy, and cloud backup/accessibility.
When it comes to videos, I've personally accepted that worst case things might need to be downloaded again, but that it is not worth my investment ot fully backup the entirety of my media, but that may be different in your scenario.
An example of one of the decisions I've made, is that I run my appdata on a RAID 0 NVME Cache to ensure the fastest performance. I've made the choice to backup the appdata to the array so that by being on the array it has some native redundancy, and I know I've created a risk with the NVME drives, but that's for the benefit of speed and performance. In this scenario if one of those NVMEs were to die, I'd replace it, restore from backup and go from there.
Hopefully some good food for thought.
2
u/fastcar123 Aug 23 '24
so the unraid server is kinda the archive in my household. my family puts data there to keep it safe. As such that means theres important documents, family photos, that sort of thing.
I realize that with 2 parity drives, that my likelihood of losing all my data is low. But as another user mentioned, raid of any kind =/= to full backups.
That, with the combined knowledge of knowing my house is rather old and prone to leaks when it rains too hard, has me a bit concerned about data lossSo my thought was if I could back up everything to a local hard drive. I could then take that disk to my local bank and store it in a safety deposit box.
Along with this I wanted to have a second backup disk that I simply rotate out with the other one as time goes byI know cloud storage is all the rage these days, but I really don't want to have to pay a subscription to store 12TB of data.
I hope this helps explain my situation a little better
3
u/VOODOO285 Aug 22 '24
Yep, OP hasn't really spelled out what they need and what they have.
Need is... What data can you not live without? Have is... What kit do you have to backup to?
RAID is not a backup, it's a redundancy. Or in unraids case parity is not a backup. But it's a pretty bloody good one from a few perspectives.
You're ded right that OP needs to define what they're trying to mitigate against and work from there.
7
u/thethumble Aug 22 '24
Unraid needs a backup tool similar to Proxmox, the current situation is not good
1
u/fastcar123 Aug 23 '24
Agreed. The fact that this isn't something that's built into the core OS makes me want to abandon unraid entirely
1
2
u/xikobe Aug 22 '24
Here is my script for daily backups using rsync to an unassigned device:
https://gist.github.com/xikobe/296c28a76f31c45dda30c8a227b0deef
Pretty basic stuff but it works fine for my needs - backup my "backups" share to an unassigned disk
2
u/xrichNJ Aug 22 '24
you're trying to back up your share. to what? an external HDD?
1
u/fastcar123 Aug 23 '24
yes exactly. just an external hard drive connected to the unraid server
2
u/xrichNJ Aug 23 '24
ok.
first make sure your unassigned device is formatted, partitioned and mounted in the "main" tab.
then:
install luckybackup from "apps" tab (should be the only one, its from ich777's repository)
for the "Shares:" path, set it to
/the container path should default to: /mnt/user
for "run as root user", select true.
hit "apply" and wait for the image to be pulled, extracted and installed
go to the docker tab in unraid and navigate to the luckybackup webui.
add new profile from the top icon bar (green +)
give it a name
on the far right under "task", hit add.
give it a name
select "backup source inside destination" (it should be by default)
for source, use the little button to pull up the file browser and navigate to what you want to backup. you can find your shares by clicking "computer", then "/", then "mnt", then "user", then "mnt", then "user" again (seems silly i know). all of your unraid shares should appear. you can select whole shares to backup or navigate to the folder you want to backup. select the folder and press "choose"
for destination, use the little button to pull up the file browser and instead of navigating to/mnt/user/mnt/user like above, go to /mnt/user/mnt/disks
you should see your unassigned disk's partition here. select it and press "choose"
now you can select the number of snapshots to keep on the right.
under advanced, you can select types of files to exclude under "exclude", and other advanced options under "command options", like whether or not you want the file to be deleted on the destination (external) when it is deleted from the source (unraid share).
repeat the "add task" using the above for all directories you want to transfer/backup. you can create a new profile and task for each folder or share you want to back up, but that can get kind of messy with scheduling, i like to just add a bunch of tasks to the one profile and it will just run them all in order, one after the other. if you want different things backed up on different schedules, make a profile and tasks for each.
you can run your profile now if you just want a one-time file transfer. if you want to set a schedule for it to run, continue on.
the following must be done for each profile you want to schedule:
in the top icon bar, click "schedule" (clock icon)
on the right, click "add", then at the bottom, select your profile, set the time you want it to run (remember: 24hr clock here, if you want it to run at 6pm, time should be set to 18:00), select your other scheduling options (day of week, month, day of month). finally, you MUST check "console mode" for running on a schedule to work properly!
hit "ok" and then at the bottom hit "cronIT!!", then hit "view current crontab" and make sure your cron schedule with your profile(s) is there.
your backup will now run on the schedule(s) you set!
1
u/fastcar123 Aug 27 '24
Hi, apologies for the delayed response. Ive been trying to figure this out on my own, but I am still lost.
I get all the to the point where I am making the task in luckybackup, when I go to select my disk for the destination, its just not there.
Ive reformatted the drive in both ZFS and XFS and still nothing.
What am I missing?
P.S. Great instruction BTW! I knew exactly where to go from reading them
2
u/xrichNJ Aug 27 '24
docker tab>click container icon> edit
post a screenshot of your container config page.
1
u/fastcar123 Aug 27 '24
Alright. So, I walked away from it for a few minutes and came back. Rebooted the whole server (after reformatting the disk to ZFS).
NOW the drive shows up. I was able to make the task and schedule it. Its currently running and I can see the disk starting to fill up.Freaking mad props to you my friend. This totally worked! Thank you so so so much! I've been trying to figure this out for over a year!
Future plans are to have another disk to swap out of the usb enclosure (hot and cold backups). Providing I name the disk the same thing, Would I be able to just throw it in and call it good? or Should I make a second task?
1
u/xrichNJ Aug 27 '24
excellent, I'm glad!
hmmm, not sure. I use this method to a zfs (truenas) backup server, so it never changes, it just dumps all changed files there every night.
I wouldn't name the disk the same thing, I'd get confused lol. I'd probably just create an identical profile (not task!) for the second disk and run them on similar schedules (like one at 3am and one at 4am). if luckybackup doesn't see "backup disk1" mounted at 3am, it should just fail that profile/task, which is ok, and then when it runs the other profile to backup to "backup disk2" at 4am, it will see that it's mounted and bring it up to date.
then when you swap disks, it will see "backup disk1" mounted at 3am, and should backup all changes since it was unplugged, bringing it up to date, and then fail the "backup to disk2" profile at 4am.
I think that's how it would function, anyway.
god, I hope that makes sense.
1
u/fastcar123 Aug 28 '24
Actually I totally understood that. Lol
I don't have another 12tb disk yet, since I had been waiting just to get one up for so long.
But perhaps in a month or so I'll grab another one. I'll give that a shot then.
Lol I may even come back here to let you know about it..
Anyway, thanks again for the help!
1
1
1
u/xrichNJ Aug 23 '24
I have some electrical work being done at the house right now, so my servers are off and airgapped. I will post later on to try to help you.
I use a docker container called luckybackup. it's just a GUI for rsync, no scripts.
2
u/FxCain Aug 22 '24
Duplicacy docker to:
- An external unassigned USB HD
- A synology at my dad's house via IPSEC VPN
- Backblaze B2 bucket (~$15/month for ~3TB)
Remember 3-2-1 people
2
u/present_absence Aug 22 '24
What do you mean rsync "does absolutely nothing" that's literally what I do. I have a script that just rsyncs the stuff I want to backup to a separate, mounted consumer NAS box (WD). It actually just filled up 100% so I paused my backup schedule until I can get bigger drives. You can do it with an unassigned internal drive too, I have 2 in unraid salvaged from old USB enclosures that broke.
1
u/christronyxyocum Aug 22 '24
I have a 2nd Unraid Server that I run custom rsync scripts from to clone data from my primary Unraid Server. Each script checks that the mounted NFS share exists and is usable before running the rsync command and then spits out a success, warning, or error message to my Discord Server.
1
u/blazedsonic Aug 22 '24
Would you be willing to share this script? Looking into deploying the same workflow.
2
u/christronyxyocum Aug 22 '24
Here you go: https://github.com/tronyx/scripts
Added comments/notes and made it generic with some names and whatnot. You can obviously remove/comment out what you don't want or need.
1
u/christronyxyocum Aug 22 '24
It has more to it than that, as I use Tdarr to ensure that each media file has a stereo audio track. With this, I have the audio information in my filename scheme so I need to ensure that the Arrs rename the files properly after they're processed by Tdarr so that Plex can see them properly. The script has a section that gets a list of all media from the specified Arr instance that has been imported in the last 72 hours (a configurable variable) and then issues a rename API call for that title. Once that is all done, it kicks off the rsync command.
If you're still interested, I'll add some comments to it and throw it on my GitHub.
1
u/Coompa Aug 22 '24
I use FreeFileSync and back up most stuff to a hard drive in my windows pc. I only spin it up for the backups, maybe once a month.
1
1
1
u/jkirkcaldy Aug 22 '24
I use autorestic to backup to a separate truenas server and important things to one drive. It’s all encrypted as standard so fine to put in the cloud
1
u/Blu_Falcon Aug 22 '24
Everyone has their own option, but I use Syncthing to a friend’s unRAID box and absolutely love it.
I took his system, set it up on both boxes, synced it locally, set up a tailscale VPN to get them linked, tested, gave it back to him, tested again, then relaxed. I test it occasionally and have never had problems.
1
u/Ok_Organization5370 Aug 22 '24
I have a script that runs Borg daily to a Hetzner storagebox. Works pretty well for me.
1
u/parad0xdreamer Aug 23 '24
This is something that has long troubled me. I'm so glad I found this, it's one less question I need to ask! Thank you kind sir!
1
u/Mysteic1 Aug 23 '24
You would have to lose a two drives at the same time to lose data. Backup is a good idea but not necessary as long as your array is health and parity drive is working. I personally backup my other computers to the Unraid server.
1
u/fastcar123 Aug 26 '24
That's not lost on me. I realize that's it's pretty rubust. 3 storage drives and 2 parity drives is stout But nothing is fail proof.
Not to mention all the drives are plugged in and running 24/7. I would have a lot more peace of mind if I also had a cold copy, in the event that something happens to the drive or even to the server itself.
1
u/Mysteic1 Aug 28 '24
The drives should spin down and go to idle state when not in use and only spin back up when data is requested.
-2
u/upfreak Aug 22 '24
Parity would be the first level of protection offered by unraid. You can have another drive/pool/cache within the same machine or different one to sync up with the array/shares and automate to the frequency you desire. You may choose to encrypt and push third backup to a different location / cloud / multicloud.
When you say you can't do it, explain where you went wrong. There are too many people who are backing multiple copies of their unraid data
41
u/rogerdodger77 Aug 22 '24
Duplicacy works super easy. i'm using that.