We[1] built borg into our environment[2] as soon as it was stable, release software. In the years since, it has (ironically) supplanted use of rsync as the de facto standard that our users back up to us with.
As one of our users have said[2], borg is "the holy grail" of backups as it does everything rsync always did, and produces remotely encrypted backups that the provider has zero insight into.
It also does not have the inefficiencies that the older, duplicity software has.
If you are willing to go without (borg specific) technical support and do your retention with borg instead of our zfs snapshots, there is a special, discounted rate available.[4]
Borg is definitely not the holy grail of backups, unless one has very loose requirements.
Desktop backup solutions have typically a set of functionalities to compare against (block-based, safe on untrusted repositories, with realtime backup, compression-efficient, and optionally, with a functional GUI), that show how open source solutions are strangely lacking, each in a way or another.
> produces remotely encrypted backups that the provider has zero insight into.
> in a multiple-client scenario a repository can trick a client into reusing counter values by ignoring counter reservations and replaying the manifest
Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed.
> Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed.
Depending on your use model, this may or may not affect you. Whether you use multiple threads to compress a single stream or not probably matters a lot less if you have 15-30 backups going on at the same time.
Nice to see someone offering public borg hosting! I have two security questions:
(1) Are you doing any hardening of the borg remote side? The server is pretty complex and it has lots of code paths which can be exercised by a potentially untrusted client.
(2) Is there a way to protect compromised clients (like a cryptolocker)? borg offers "append only" mode, but it does not allow pruning (obviously). And trying to run "prune" using separate trusted connection will commit untrusted changes as well.
inefficiencies that the older, duplicity software has.
As much as I love duplicity, it does sort of suck when you get into larger numbers of files and gigabytes of data. It's so darn slow sometimes it's almost unusable. Good to know borg is better, I've been meaning to check that out!
Duplicity and Borg use two very distinct technologies under the hood - duplicity does differential partial backup archives (which all require a full backup AND a full list of all previous differential archives) and Borg uses deduplication. So yes it is much more efficient in many ways, not simply because it is newer and somehow is more efficiently coded, it is so because it uses different technology and backup model.
I've been using borg to backup small things, like schoolwork, config files, mail, and code, to rsync.net for ~1 year now, and can vouch that it's pretty simple to setup. If you've got a simple use case like I do, the example config in the borg tutorial is pretty much all you need. I've got it set to automatically run as a cron job, and I haven't had to touch it except to occasjionally do sanity checks that everything is being back up correctly.
If your customers back up multiple servers, are they still under the same rsync.net account and could they see each other's data?
This is one of the issues I aimed to solve with BorgBase.com[1]: every single repo is its own backup user and can't see other repos. This separation allowed to add some Borg-specific features, like append-only mode, monitoring for stale backups or using a specific Borg version.
rsync.net has "subaccounts" which allow you to control side-by-side backup directories using standard UNIX permissions and ownerships. That is, if the default configuration is not exactly what you'd like.
Subaccounts even get their own .ssh/authorized_keys file.
It's all very unix centered and command-line focused. There's no web interface for features like this.
I see that you offer "append only mode" as a core feature. How do you deal with pruning of old backups in this case?
Regular borg only offers two options: either pruning does not decrease space used at all; or pruning permanently commits changes, negating any security advantages. Both of those seem bad enough to prevent offering "append-only" commerically?
In my experience pretty reliable. I've not seen major downtime and bandwidth will depend entirely on routing to you in particular (FWIW my ISP peers with Hetzner directly and congestion hasn't been an issue, I can typically saturate my 1Gbps connection at any time of day).
> borg is "the holy grail" of backups as it does everything rsync always did
I tried borg for some time for full backups from / and there were 3 things I didn't like about it:
1. Mounting and navigating a snapshot was extremely slow. Like wait a few seconds for any `ls` to finish.
2. It needed the encryption password for many things I think it shouldn't.
3. The resulting backups are very opaque.
I've since switched to making my backups by rsync'ing to btrfs subvolumes. I snapshot the previous backup and rsync the differences to the new one. I get to navigate each snapshot with the same snappiness as any other directory in the filesystem. btrfs can also instantly tell me which files differ between 2 snapshots, no matter how huge they are. I don't remember all the options that borg offers on synching, but I doubt it has the same breadth of options as rsync. So for me, rsync + COW filesystem > borg.
I am a little concerned with my use of btrfs, but I'm planning to do the same thing using ext4's reflinks for some disk and filesystem redundancy.
How do I get rotation schemas to work with borg? Say I want to keep the last 3 daily snapshots, one per week for the last 4 weeks, and one per month for the last 6 months...
I understand this is kinda hard with "zero knowledge" encryption, but it is possible and not a wild feature to request from the self-named holy grail of backups soft.
Yes, but which server will run the purge command? It cannot be the same server as the one you are backuping, since one of the goal of the backup is to protect the server if it is compromised.
You can run command from inside your backup script, from the client. It needs the encryption key anyway to add backups, this way it is just one extra command that does all the pruning (quite quickly as well). And you don't have to worry about full/incremental backups, nor really about any "rotation" or managing them manually, it's all deduplicated and Borg just keeps X number of daily/weekly/whatever snapshots (like duplicacy, like restic, like bup, etc. etc.).
Yes, I got this. But if your server is compromised, then the attacker can erase your backups, which ruins the point of doing backups in the first place. This is why there is an append-only mode.
What I do is to have two different SSH keys. The one that is used by the backup script is --append-only. The one that I keep locally on my notebook has full access, so I `borg prune` from there when I see the disk usage reaching critical levels.
Don't rely on append-only mode in Borg, it does not work really as advertised. Every transaction in that mode is recorded in a log, which if you are very careful you can go back into (but even reading or monitoring it is a PITA, the "interface" requires you to carefully track id's of files and check with dates etc).
And being careful is difficult, because in Borg, once you do any other backup command from a full access account (for example to do pruning) - it will automatically, no warning, go through the log and apply it. You should really really read up on that functionality first before relying on it, the way Borg has implemented it is close to anti-feature.
Regarding compromisation of strictly the server itself, I believe there are commands to check the state of the repository? Isn't that enough?
If we are still talking about rsync.net, and an attacker gains access to your account and deletes your borg archive, you can still restore your backup from the rsync.net ZFS snapshots.
The snapshots are not deletable even with your full credentials.
There's so many backup systems out there these days, but none that are quite perfect. The closest I've found is `rdedup`[1], which may no longer be maintained. What I'd like in a backup solution is:
- compressed
- deduplicated
- encrypted using asymmetric encryption, so that the encrypting machine doesn't need to know a password
- durable (using par2 or directly supporting repository repair from an independent backup)/signed
- composed of standard open source tools so that if the backup software goes away, everything can still be retrieved
- supports differing upstream data hosting providers
- open source
There's lots of things that fill most of those requirements, but none that tick all the boxes.
- rdiff-backup does all of the above except deduplication; it's full + incremental which requires a management strategy
- restic doesn't compress, doesn't use asymmetric encryption and doesn't use par2 or similar tools for durability
- borg doesn't use asymmetric encryption and doesn't use par2/durability stuff; it's also pretty slow
I know people call borg the "holy grail", but I think we're still a wee way off that emerging, personally -- even though I do really like (and use!) borg.
Yeah, I've been searching for a great general archiver for a long time. I tried borgbackup a couple of years ago and immediately hit a handful of issues that put me off, though I can't really recall the specific technical details clearly (I'm vaguely recollecting something about incompatible block sizes).
Whatever the details were, I came away with the distinct impression that the borgbackup developers didn't really respect the gravity of archived records and long-term storage. While that's probably not a totally fair conclusion from a one-shot use, in my defense, I'll offer up the latest version of their changelog [0], where:
* the first thing listed is an apparently serious data corruption bug that lived through several stable releases
* the second thing listed is an apparently-very-serious security vulnerability ("a flaw in the cryptographic authentication scheme used in Borg allowed an attacker to spoof the manifest ...")
* the third thing listed is another data corruption bug titled "Pre-1.0.9 data loss"
* the fourth thing listed is another data corruption bug titled "Pre-1.0.4 potential repo corruption"
Note these are all post-1.0 versions. To be frank, I dare not scroll further.
Users are depending on this software to safeguard important files with the assumption that it will be able to reproduce them bit-for-bit-intact some years down the road. Any long-term storage software requires developers with a fanatical devotion to compatibility, longevity, and integrity if it's ever going to be more than a toy.
I hate to pile on open-source devs who're trying their hand at making software that a lot of people appreciate, but overall, it's hard for me to say I regret the choice to pass it up and stick to combinations of ZFS, rsync, and the trusty old tarball.
I still don't know how to really judge software from changelogs, but I think it's really hard to do and this post is not a great example either.
This is a changelog backup software which does data storage and encryption mainly. If it ever has bugs worth talking about they're going to be in... data storage and encryption. If we look at rsync for a change which is much older, simpler and mature: https://download.samba.org/pub/rsync/src/rsync-3.1.1-NEWS
- fixing .. traversal in V3 (how did that survive so long?)
- issues with copying attributes
(There won't be much data corruption since rsync does 1:1 copies)
Time-to-fix could give us a better idea maybe? Either way, software doing X has bugs in X is "normal".
I mean, the issue isn't so much that bugs exist -- like you say, that's par for the course, even among the best developers. It's about a large incidence of severe eat-your-data bugs in such a short span of time (roughly a year) in something that's quote-unquote stable and whose entire purpose is long-term data storage and retrieval.
It's even worse because unlike rsync, which synchronizes filetrees from point A to B and can be immediately confirmed to have either worked correctly or not, borg uses a bespoke storage format that's not easily verified or operated upon by standard utilities. You just have to trust it to pull the correct data out when you need it. That's a heavy burden to put on a tool, and the borg devs seem to be struggling under its weight.
Consider the standards expected of filesystem or database maintainers. borg repos are not really that different -- they're a big, opaque chunks of bytes that you expect to be able to produce specific data on demand with perfect reliability.
These types of systems don't get marked stable when they're still in primordial, eat-your-data development phases, and on the small handful of unfortunate occasions when data loss bugs sneak in, they're a) usually limited to some bizarre corner case; and b) taken extremely seriously, almost to the point of solemnity, and often result in major overhauls to a project's validation and QA routine.
If any stable filesystem or database had 4 widely-applicable database corruption bugs within a year, it'd be lights out for that project. All trust lost, reputation irreparably ruined, angry letters to a variety of mailing lists, permanently-increased scrutiny on any new projects or maintainers among the same class of projects, etc.
All I'm saying is that based on my admittedly-murky prior experience, the project's observed track record, and the high standard of care that must be met to qualify a project for storing authoritative copies of data, I'm personally not comfortable trusting borg with anything important any time soon. No one is obliged to share that evaluation, of course.
If compression is purely a function of the price, things are not so obvious.
I think Backblaze B2 is the cheapest, but it's not supported by Borg. So a backup system without compression support, but with B2 support, may ultimately be cheaper than a system with opposite features.
It's also important to consider the use case. If a user's bulk of data is photos/videos/audio files, there's virtually no use of compression - but of course, the proportion will vary per user.
I’m using restic too for part of my backups, having moved from duplicity. My experience has been (on a dataset which is relatively compressible ie not audio/video files) that deduplication has saved me quite a lot more space vs duplicity than I lost through not having compression.
Comparing duplicity to something a deduplicating software suite is not really correct, they don't even provide the same functionality (like mounting snapshots, like handling moving around of the files on the hard drive, like space-efficiently backing up files from different machines, like smart and efficient pruning, like speed of access(impossible for duplicity to achieve), etc.).
rclone is rsync-on-steroids, and you can use it for backups, but it's not the same shape of tool as the above tools. It's effectively "copy these files from $here to $there" where $here and $there are any cloud provider or network filesystem. It's got a built-in encryption layer and some other bits, but you'd still be responsible for ensuring $there is actually $there/backups/(date +%F) or whatever.
I'd treat it as a tool in a bigger backup system (which is roughly what rdedup does with it).
Rclone is not a backup system, it is a sync system. It does not do any archiving, deduplication, pruning, etc. It just syncs two directories (with possible encryption added on top). It can be efficiently utilized as an extra final step in a backup plan.
Well, with `gzip --rsyncable` and similar (most compression tools have such a mode), it's technically possible to compress first then dedup!
To be precise it's "chunk first, then compress", which still admits a 2nd "chunk, then dedup" pass to reuse data, but not as efficiently as a combined "chunk+dedup+compress each chunk"...
`gzip --rsyncable` and friends are mostly useful for when you already have compressed files on your filesystem, and want to sync/backup them byte-for-byte. Of course the benefit is lost unless most compressed files in the world are produced with such mode, and sadly most aren't :-(
The author of `zsync` did various experiments confirming gzip --rsyncable then sync is sub-optimal, and implemented a somewhat crazy "look inside" approach that can sync compressed files byte-for-byte AND very efficiently:
> gzip --rsync does fairly well, with both rsync and zsync transferring about 410kB at the optimum point. zsync with the look-inside method does much better than either of these, with as little as 140K transferred.
> -- http://zsync.moria.org.uk/paper/ch03s03.html
IIUC though it's more of a "2-files sync" scenario like rsync, not applicable to "chunk everything then dedup" approach of borg and similar tools.
Probably, yes. Sorry, I should’ve said “in no particular order” — this is part of the reason why I haven’t tried to tackle building something myself :).
I offer dedicated Borg hosting at BorgBase.com[1] from $5/TB. As opposed to normal SFTP-based backup services, every backup repo is fully isolated. This allowed to add features, like append-only mode and monitoring for stales backups.
We have also developed a Qt-based desktop client[2] that runs in the system tray and makes it easier to browse archives or do restores.
For headless deployments, I highly recommend Dan's Borgmatic[3]. You could deploy it all together with our Ansible role[4].
I switched from bup backup to borg backup, because it generally works better for my use cases; however, there are two things bup does do better:
1. remote backup through ssh, especially de-duplicating between different clients, especially concurrent client backups.
borg needs a compatible version installed on the other side (and compatibility has been broken between versions in the past); bup uses bare-bones ssh+sftp, so the other side can basically be anything.
borg will have a lot of download to client from server if multiple clients back up to the same repository (essentially every time in most common use cases); bup will have a minimal download.
borg maintains a repo lock, so multiple clients backing up to the same repo will be serialized; bup does not, so it can be concurrent.
2. Storage format
borg's format is it's own format; bup's underlying data format is basically a git repo (which you can treat as such; you may need to manually apply "cat" to rebuild files, bit "git" and "cat" are all you need).
I have heard good things about restic, but did not have a chance to evaluate it myself.
restic doesn't perform compression. Depending on your use case (I use it for photography backups, which don't compress well), that might be OK -- but it's something to be aware of.
When I was backing up several 100 tb, I tried every open source option at the time and found them all lacking.
What ultimately worked? Plain old rsync over ssh, to a zfs pool with snapshots and compression.
As far as I could figure, the only notable downside of this is that the storage device must be trusted, since it has access to all of the data, and that you effectively needed root permissions on the storage you're copying to for filesystem permissions which makes a multi tenant backup server cumbersome (you could chroot or use containers or something but these solutions can become fiddly, e.g. running multiple instances of ssh on nonstandard ports to enable multitenancy).
Borg backup is amazing. I've been using it for years now, backing up a few terabytes of data from a few dozen VMs.
The dedupe is downright magical. We've been able to remain super frugal on the storage allocation for backups solely because of how wonderful its dedupe is.
Restores are also really nice and easy since they are just a FUSE mount.
If you have a Linux host or hosts, I can wholeheartedly recommend borg. It's elegant, robust, and fast. And open source.
I benchmarked borg and restics deduplication on a few datasets and restic was the winner (1-2 years ago). Did you do any comparisons with other deduplicating backup solutions? (Benchmarked the efficiency not the speed. Test data set was a few 100GB)
We were previously using Bacula, which was way too heavyweight for what we wanted. I found Borg after looking into Attic, and wasn't aware of Restic at that point.
Compared to the Bacula and BackupExec, Borg was lightning fast and its disk usage was very frugal.
I'm still not totally happy with most of the backup solutions, or maybe I want too many features all at once (sth like Perkeep). Although, otherwise, if the tools are simple, I have to use multiple tools to cover all the features, and there will be lots of overlap, which is maybe not too much of a problem actually.
I use BorgBackup to make a local backup on linux and mac machines and rclone to upload that backup to the cloud.
It took me some time to figure it all out. I wrote a script to not manually repeat configuration steps next time. Although it's quite opinionated it still grew to 600 lines.
I'm using borg and I honestly like it. Not ready to only rely on it. I try to use the 3-2-1 principle of backup strategy and borg is really a space and bandwidth saver! My collection grew from few Gb to some Tb. It's already more than a year old and I was wondering whether it's better to start a fresh new repo every year or to let grow the original one until it reaches a certain size? Any recommendations from ppl that used it for +2 years?
One limitation that isn't mentioned often is that when doing anything with your borg repo, RAM use increases with the number of files you have backed up. In my case I had 15 million files, and mounting the repo took quite a bit of time (minutes) and used 11GB of RAM.
I (sadly) moved from you guys to Backblaze, as they're down at $0.005 (ie. 0.5 cents) per GB. They will charge me (1 cent/GB) if I have to restore the backup, but this is my extreme off-site everything-blew-up-including-the-backups-closer-to-me backup, so, hopefully that won't eventuate.
Fair enough. I still use and like your service for a second off-site backup of ~40GB of very very important stuff, but was unable to ignore the cost savings for bulk data.
I've been using Borg for a couple of years now (and before that, Attic[0]) and I have been very happy with it. It's seen me through three laptops and it has been reliable and easy-to-use.
I use borg extensively as well. One thing I haven't found a good solution for yet is monitoring.
How do you all keep track of whether your backups succeeded? I'd like to receive an email if a scheduled backup didn't run. The only thing I have for now is rsync's feature where they warn you if your data hasn't changed by X kb in the last Y hours/days but I find this lacking because multiple machines write to my rsync account. It'll only warn me if none made any change but I'd never know if only a few failed. Same thing if there is nothing new to backup, I get an email from rsync.net but I don't know if every backup job failed or there is just no changes.
How does this compare against S3QL? Does it do multithreaded compression when backing up?
So far S3QL is the least worst of all these deduplicating/compressing/encrypting backup solutions I have tried, but I havent tried borgbackup. Despite it's stated focus on cloud object storage, S3QL also works great on NFS and local filesystems as a target as well.. and sshfs..
I see that BorgBackup is available in Chocolatey[0] and it looks like the current version. It's not clear to me if this is an official port or not, but I am interested. :-)
I've never seen borg take 5+ seconds for anything except creating backups. The same `borg --help` command finishes in 0.4s on my machine. That's not blazing fast either but I'm okay with it.
Does it care about the underlying file systems? I have one Linux machine I need to backup a directory on, and dedupe would be a huge help on this one in particular, but I'm backing up to an SMB share that's mounted to a folder location.
Borg? Hardly, there is already a Google product with the same name.
But more importantly, it is the name for a castle in several languages. So unless Paramamount is planning to sue a lot of very old places in Norway and other countries...
I wonder if I'm the only one with this phobia: i see a tool written in a dynamically typed language and automatically trust it less. Especially something that is responsible for all my data. I wonder if this is a good candidate project to be rewritten in rust.
It actually would be for several reasons, which I could outline if people are interested (but I suspect they are not, so I will not go to the effort per-emptively). Also yes, there have been a number of bugs specifically due to both dynamic typing of the code and the dynamic data structures (msgpack, basically JSON) used.
I used borg for a while but the bugs you mentioned and some of the discussions in the Github issues gave me pause. For my taste, there is still too much development going on for me to rely on it as my main backup solution.
Software written in dynamically typed languages can be robust, just as software written in the most strongly typed language can still be buggy.
As an user of such critical piece of software, I would ask the following questions:
* Is the software popular?
* For how many years it has been battle tested?
* Are there many reports of data corruption?
* Is it well maintained?
* Do the maintainers provide support?
Borg/Attic has been greatly used for many years now and check all the boxes above. I don't think I've ever read about data corruption that was caused by a bug in it.
Which language it is written in is nothing more than a mere curiosity for me.
I feel, that, if someone would write a backup solution from scratch and use Rust, that may be a good selling point.
Emphasis on scratch not on RIR ("rewrite in Rust") ;-)
It would need to respect heterogenous networks (Microsoft Volume Shadow Copy, for example), networks at all, meaning global deduplication, full volume imaging and file-level backups.
As one of our users have said[2], borg is "the holy grail" of backups as it does everything rsync always did, and produces remotely encrypted backups that the provider has zero insight into.
It also does not have the inefficiencies that the older, duplicity software has.
If you are willing to go without (borg specific) technical support and do your retention with borg instead of our zfs snapshots, there is a special, discounted rate available.[4]
[1] rsync.net
[2] https://news.ycombinator.com/item?id=17408624
[3] https://www.stavros.io/posts/holy-grail-backups/
[4] https://www.rsync.net/products/borg.html