Posted on 15/05/2023 20:40:00
If you have been reading since last year July, you may know of my “Project ReuseNAS” - you may also know that my build date estimate for “end of 2022” or “Q1 2023” was never met as per obvious lack of posts on that front.
When I was originally planning the NAS build, I was very much all over the idea, that I could reuse all of my old PC components. Over time after discussing with members of the based.quest community, we came to the conclusion that I would be severly handicapping myself with the hardware I already had laying around.
I was unable to find a satisfactory case for the NAS build that could hold the HDDs on them - apparently cases with 4x 3.5" + 2x 2.5" bays are quite uncommon, as such I ended up buying a brand new case.
Further I did not trust getting an used power supply or used SSD/HDD drives. CPU cooler I most likely could have found used, but the project was starting to drag on for far too long - so I ended up getting a brand new one. I know, impatient.
My original plan was also to buy 4x4TB IronWolf Pro drives - as of right now, I only got 2x4TB IronWolf (non-Pro) drives.
The final rig ended up being:
CPU: i5-2400 (Second-hand)
CPU cooling: be quiet! Pure Rock 2 (Brand new)
Motherboard: MSI H61MA-E35(b3) (Second-hand)
RAM: 4x4GB DDR3 from my old “Celery Fujitsu” PC (Second-hand)
PSU: Corsair CV550 (Brand new)
Case: Krux Naos TG (Brand new)
HDD: 2x4TB Seagate IronWolf (Brand new)
Boot SSD: 2x120GB Patriot (Brand new)
As of writing this, a few of these are still being shipped, but that didn’t stop me from starting the experimentation, using my current PC as the test-bench.
One simple Friday evening, as I had finished getting final parts of the NAS - I asked the matrix chat for suggestions regarding backup solutions I could use. My criteria was simple - it must be able to do incremental backups and potentially deduplicate. I was thinking of some hacky rsync solution already until Kerum stepped in with the idea of using borgbackup - I was intrigued, it offers space efficient storage of backups, LZ4 compression, mountable backups with FUSE and claims to be easy to setup.
Without wasting further time - I decided to take my 2x4TB drives out, attach them to my main PC, build a software RAID 1 and install borgbackup. What I discovered was everything I could have asked for from a free solution and beyond.
Let’s start with the extras I wasn’t expecting for – the backups are all encrypted. If your HDDs get stolen for whatever reason, your backups are secure. Further you are secure from malicious actors - you do NOT need borg on the remote site even, you do not need to store any passwords on the repository server.
About performance - your initial pass is always going to be slower as borgbackup has to make note of blocks for deduplication and also encrypt the data. However, this is why I wanted the ability to do incremental backups - you only have to move what you need. Your next passes however, will be a lot quicker - depending on the amount of inodes in your server. For me, an incremental backup takes about ~ minutes presently. Here’s a summary from the last backup I just took (presently manually, going full-automatic once the NAS final components have shipped!).
Mon 15 May 2023 07:10:29 PM CEST Starting backup
/path/to/<redacted>.log: file changed while we backed it up.log
------------------------------------------------------------------------------
Repository: ssh://<user>@<nas>:<port>/path/to/borg-repo
Archive name: cernodile-2023-05-15T19:10:29
Archive fingerprint: 42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063
Time (start): Mon, 2023-05-15 19:10:30
Time (end): Mon, 2023-05-15 19:16:53
Duration: 6 minutes 22.94 seconds
Number of files: 741792
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 197.53 GB 149.67 GB 1.58 GB
All archives: 971.53 GB 754.30 GB 151.90 GB
Unique chunks Total chunks
Chunk index: 530571 4076474
------------------------------------------------------------------------------
terminating with warning status, rc 1
Mon 15 May 2023 07:16:59 PM CEST Pruning repository
Keeping archive (rule: daily #1): cernodile-2023-05-15T19:10:29 Mon, 2023-05-15 19:10:30 [42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063]
Pruning archive (1/1): cernodile-2023-05-15T06:15:29 Mon, 2023-05-15 06:15:31 [4266d525a871b036cb71f0ba0f8930e1dcd04499fa5293deba87305497fe957d]
Keeping archive (rule: daily #2): cernodile-2023-05-14T01:08:46 Sun, 2023-05-14 01:08:48 [0dc46a3106d1df91170fb60d0661a1b6ca92fc50694acfa8833745dd302d1fd6]
Keeping archive (rule: daily #3): cernodile-2023-05-13T09:09:41 Sat, 2023-05-13 09:09:41 [f0be5704c565202b405ac833e1256fd320ce0968946bfabfd3312d3a1aa3afc2]
Keeping archive (rule: daily[oldest] #4): cernodile-2023-05-13T00:51:50 Sat, 2023-05-13 00:51:51 [14be5350fa55603c65e897c5a6f37925943b7c5eed793db386d5090464d6a18a]
terminating with success status, rc 0
Mon 15 May 2023 07:17:18 PM CEST Compacting repository
Mon 15 May 2023 07:17:34 PM CEST Backup, Prune, and/or Compact finished
I seem to face a visual bug regarding the warning status (no actual warning is output!), but other than that, I’ve had no real issues so far - I have verified integrity of the backups (each one!), all match up.
Note the original and compressed sizes here. The original backed up data (aka actual FS usage) is 197 GB which gets compressed down to 149.67GB! With 3 other archives visible here, you can see that the incrementals have only added up about 3GB (which roughly aligns with the daily amount of data added too ~1GB/day).
As mentioned before, borgbackup supports mounting your backups with FUSE provided you have the space for it. Let’s say I accidentally make changes I didn’t really intend to today and I didn’t make a backup of it. So I want to naturally take my nginx.conf back from the previous day.
To do that, I go back to my PC terminal and check for my existing restore points with borg list /path/to/repo:
$ borg list /path/to/repo:
Enter passphrase for key /path/to/repo:
cernodile-2023-05-13T00:51:50 Sat, 2023-05-13 01:51:51 [14be5350fa55603c65e897c5a6f37925943b7c5eed793db386d5090464d6a18a]
cernodile-2023-05-13T09:09:41 Sat, 2023-05-13 10:09:41 [f0be5704c565202b405ac833e1256fd320ce0968946bfabfd3312d3a1aa3afc2]
cernodile-2023-05-14T01:08:46 Sun, 2023-05-14 02:08:48 [0dc46a3106d1df91170fb60d0661a1b6ca92fc50694acfa8833745dd302d1fd6]
cernodile-2023-05-15T19:10:29 Mon, 2023-05-15 20:10:30 [42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063]
As I said before, I want to see /etc/nginx from previous day, in this case, May 14th. To do that, I’ll have to use borg mount as such:
$ borg mount /path/to/repo/::cernodile-2023-05-14T01:08:46 /mnt/ /etc/nginx
Enter passphrase for key /path/to/repo:
$ cd /mnt/
$ ls -lh etc/nginx/nginx.conf
-rw-r--r-- 1 root root 1.5K 26. apr 23:33 etc/nginx/nginx.conf
Voilah! We have retrieved the configuration file that was present that day! Now we can retrieve the contents of it using whichever preferred utility you wish or transport it directly
to the server with scp/ftp/sftp.
Once you’re done, run borg umount /mnt.
I want to see if it’s possible to make my backups immutable in event of cernodile.com or based.quest being hacked.
Overall my first impressions so far are very positive, there is so much I haven’t discovered yet with this backup solution and it is everything I could have asked for. For my use case, this is ideal. The backup script I use is available in borgbackup’s own documentation. I recommend you read up on that as well if you are as interested in borgbackup as I am.
I am glad to finally have my NAS and backup solution set up, it’s been a long time coming.
Thank you so much for reading, hopefully without this long of a delay next time,
Join us at Matrix: #based-quest:cernodile.com
© 2021 - 2023 based.quest | Powered by Hugo | Donate | Atom RSS