It's been almost 10 months since I joined SeeD as a Partner and Development Manager, which means I've had to change my worries: I can't think in number of sites anymore, it is about the number of servers now; at first things were fucked up but Aegir made things a breeze.
Since we moved to Aegir our processes have changed, and although most are dictated by how Aegir works (migrating sites being one of them), there was one that we still had to figure out how to do so we could sleep in peace: Backing Up everything!
Since I didn't have much time, I went for what to me seemed like a very simple yet powerful solution: Percona Server + JungleDisk.
Why JungleDisk? Because it was the only simple tool I found without spending more than 5 mins googling, and since my business is not backing up but producing websites and webapps, whoever does a good job at marketing their backing up tool works for me! (Ah, and it lets me choose RackSpace or Amazon for storage, cool).
JungleDisk is the easy part, I set it up to backup 2 locations, one is where Aegir is hosted (the whole thing) and one is where I'm backing up the DBs with the percona hot backup utility, and that (Percona/MySQL backups) was the fun part. I needed a script that would run backups of all the DBs every X minutes and needed to keep a history of at least 2 weeks, but didn't want to keep full backups for those 2 weeks or I would run out of HD space or go broke paying for the extra.
I couldn't find a script that would do that for me so I wrote one myself. The script is based on one from cdamian, but the new one does incremental backups! Nice! It will create a full backup every X minutes and will keep X number of full backups (plus its incremental ones).
Once I was happy with my script I set up a cron to run with the MySQL user and that was it! I am now sleeping fine, all my data is secure and I can quickly restore if some crazy asshole blows something up where my servers are something goes wrong.