Our company recently updated my colleague’s aging Powerbook to a brand spanking new dual-G5 desktop with a 20″ Apple Cinema Display. (I expect his productivity to quadruple 🙂 This is the same setup I have at home, and so Mike ask me to put together some notes regarding my backup strategy. I decided to write this up as a blog article, for later reference.
Overview and objectives.
My years of using Dantz’s Retrospect backup software were characterized by:
- Buggy software versions, that never quite seemed to keep up with the latest release of MacOS X.
- An inordinant amount of money and time spent purchasing, swapping and rotating media (tapes, DVDs, etc.)
- Recovering needed files was cumbersome, and sometimes impossible due to some subtle misconfiguration of my daily, weekly, monthly full- and incremental set of backup scripts.
I finally gave up on Retrospect, and decided to search for a new backup system/strategy. My objectives included:
- I should be able to immediately get back to work in case of primary disk failure.
- Operation of the backup system should involve my intervention as little as possible.
- Retrieval of files should be fast and reliable.
I settled on a system that meets these criteria, and have been happy with it for the past three or four years.
Tools.
Following are the tools used in my backup system:
- Qdea’s Synchronize Pro X. Synchronize Pro is a general purpose, industrial strength, backup and synchronization utility, capable of maintaining a bootable mirror, and archiving modified or deleted files.
- Michael Tsai’s DropDMG. DropDMG, as its name implies, is a utility for creating DMG (and now, other types of) archives. As to be expected from a MIT computer science grad, DropDMG is (like all Michael’s products) of top quality.
- Skytag’s File Buddy. FileBuddy is a general purpose searching and file management utility. Probably one of the oldest utilities on the MacOS platform.
- CDFinder. CDFinder is a disk cataloging utility.
Architecture and process.
Mirroring. My G5 desktop has a second internal drive, of equal capacity to the primary startup drive. I use SyncPro to, automatically each night, mirror my startup drive to my secondary drive. If my startup drive ever dies, I can immediately boot from the secondary drive, and at maximum have lost one day’s work. (If a Powerbook were my primary machine, I would use an external Firewire drive in the role that G5’s second internal drive serves in this architecture.)
Archiving. Connected to my G5, I have a 160 GB external Firewire drive, to which SyncPro archives modified and deleted files, when mirroring my primary drive to my secondary drive. Say I’m working on a file that I created several days ago called “proposal.doc,” which has changed during the day. At night, when my SyncPro mirror script runs, it will detect that this file needs copying to the mirror drive, overwriting the older version of this file. Before over-writing the older version, SyncPro will archive a copy of that older version to my external Firewire drive that I’ve specified as my target destination for archives.
As you can see in this image, the “Current Group” folder contains a timestamped folder of archived files for each time SyncPro is run. (This allows multiple archived copies of the same file). You will also note that SyncPro preserves the folder structure (path) of archived files. Finally, you’ll also note a folder that’s sitting outside the “Current Group”, whose name (in this case) starts with “3207805120”. This is a folder containing a set of archived files whose total size is suitable for offline archiving to DVD (see more about this below).
In my mirror script, I’ve configured SyncPro to maintain 5 GB of free space on this external drive, thereby allocating the other 155 GB to storage of archived files. Once there remains only 5 GB of free space, SyncPro will begin deleting older archived files, to make room for new ones. In my experience, 155 GB of space gives me about half a year of archives.
One final note about archiving, SyncPro also allows you to specify certain exclusion criteria. For example, although I back them up, I do not archive my VirtualPC disk image files, nor system log files, etc.
Offline Archiving. One of SyncPro’s archiving features is the ability to “group” archived files into folders suitably sized for DVD archiving. In the above image, one such group is the folder whose name begins with “3207805120”. If I want, I can drag these folders to DropDMG, which will create a compressed DMG archive, and with the latest version, even directly burn the DVD for me. I can then use CDFinder to catalog that DVD’s contents for later searching.
I’m very lazy when it comes these kinds of things, so after offline archiving for about a year, I decided to stop. Since my archive drive’s 160 GB holds about half a year of achived files, I just decided that was sufficient, and so I disabled SyncPro’s “grouping” feature in my mirror script.
File retrieval. Whenever I need to retrieve a file, I launch FileBuddy, configure my search, and have it search only my archival Firewire disk. (There are, of course, a multitude of search options, including Spotlight. However, I’ve been using FileBuddy for so long that I’m simply not inclined to switch.)
What’s missing?
The only thing missing from my backup strategy is an offsite backup, proctecting me in case of some disaster to the entire home. I do try, from time to time, to make a DVD-based snapshot of my important files, and store that DVD somewhere outside my home (e.g. my mother-in-law’s house). But, unfortunately, I don’t do this as often as I should.
Some might point out that should my archive drive fail, I’d also lose half a year of archived files. True. But that’d only cause me problems in case the drive failed and I needed to recover files. This, knock on wood, has never happened, and I expect the probability of it is quite low. (On that note, given how cheap external drives are, I could keep a mirror of my archive drive if I really wanted to.)
Advanced usage.
While the details are beyond the scope of this article, I use the same above procedure to keep a backup and archive of our company’s online directories, and our shared documents server. I use rsync and ssh to perform a nightly mirror to a location on my G5’s startup drive, and by way of the above procedure, maintain an archive of modified and deleted company files. (We also have a fancy corporate backup system in place, but this home mirroring and archiving gives me just that bit more peace of mind.)
Notes
- ChronoSync. ChronoSync is another excellent synchronization utility. I use it mainly for “folder” synchronization — for example, keeping certain folders in sync between my G5 and Powerbook. I like its user interface better than SyncPro’s. ChronoSync has just added archiving in its version 3.1, but still doesn’t support the creation of bootable backups (mirrors), and therefore still can’t serve as a SyncPro replacement for me. (And, honestly, as reliable as SyncPro has proven over the past several years, I’m not sure I’d replace it anyway.)
- SuperDuper!. For the task of only mirroring, SuperDuper! is an excellent utility. It seems very fast, and when I need to make a quick bootable copy of a drive (e.g. the Powerbook’s drive), I sometimes use SuperDuper!
- DiskTracker X. DiskTracker is an alternative cataloging utility that I was quite happy with for many years.
Matt,
I found a link to your backup post and thought – wow, this is exactly what I was looking for. I had been using two drives as a backup + archive for my primary drive. Only I’ve been doing it with clunky scripts. Thanks for sharing how to automate it!
But then I took to reading some of your other posts. And as a cyclist (road) and Spaniard found that I will need to come back and read more – you have some great posts + pictures. Thanks for sharing thos too!
Something that dawned on me was that you were a Mac user in Spain. Once of my biggest problems is the expectation that my dad has in my ability to keep his Windows ME PC up and running (it gets hit by a majot virus outbreak at least once a month). I’ve been trying to convince him to get a Mac for a while now – but one of the “reasons” he hides behind is that “in Spain they don’t use Macs”. Which I can understand as “I don’t know anything about them and if I need help I probably won’t find anyone around here that would be able to”. Any chance that you have some insight to share on how to help dispell this concern?
Thanks again
Javier
PS my email address is just fjl, remove the backwards spam to make it work.
Matt, have you tried Synk of Ben Riester (Grad student at CMU)? This program beats many which I have seen and its customer service is awesome — and free for students!
How fast is SyncPro? I used Synchronize Plus but I got annoyed by their upgrade & pricing policy, so I’m now using Chronosync. The only thing I don’t like about ChronoSync is that it’s very slow – it will take about an hour to back up my home directory to a server via NFS and it slows down my system very badly while it’s working. I back up 4 times a week to a Linux server with a 120G mirrored RAID.
A coincidence or not? This morning I found this post via delicious and a little later my G4 desktop started making noises, which I then tracked down to the aging 40GB drive inside. We had a few of the exact drives laying around at work, so I installed one, got a free 5 day license of SyncPro X, and I’m now creating a bootable mirror of my failing drive.
Hopefully this works and my drive doesn’t die on my in the process (when it makes noise, the whole machine seems to freeze and then unfreeze). I’m just glad I found this article today, because otherwise I’d be reinstalling OS X for the rest of the afternoon.
I’ll let you know how it turns out, but thanks in advance.
Simply put, THANKYOU!
I have been using my mac full time for about three years now, and have always been plagued by the lacking ease of backup knowledge. I have called apple tech support and told them about my external 200 gig firewire drive, and my 40 gig internal, and they acyually told me that it was a stupid idea to back up on another drive, and that they (the tech support guy) do it on dvds.
After reading this pretty comprehensive walkthrough, I am pondering getting another external drive, only 40 gigs, and then mirroring my internal drive onto my Lacie 200 gig daily/weekly, and monthly onto the 40 gig external. I would like to have a bootable copy of my hard drive a click away, and also have a recent-ish copy at another location.
Do you have any thoughts on my situation? Even though I have basically no back-up now, I would like to get one going. the problems include: 1. My extreme laziness 2. My extreme cheapness (well maybe not “extreme”)
Thanks Again.
Update:
I’m happy to say that the process, while not smooth, worked as expected. My G4 is now booting from my backup drive using the exact copy of my system from the failing drive.
During the process, the failing drive made a lot of unpleasant noises and had some I/O errors, causing the loss of a few files (from software I can reinstall if they’re important). The first run through died on the last 4mb of data as the machine halted. Luckily it restarted this morning and ran through the backup completely.
Thanks a lot Matt. I would’ve been seriously lost without this blog entry. Now I can finish erasing the old drive and throw it in the trash with a smile on my face.
Matt,
Thanks for this detail write-up. I was thrilled to see a strategy sans Retrospect, for exactly the reasons you specify. Synchronize Pro is a great piece of software.
Bob
ibackup is also worth a look. really simple to use and free for personal use. nice article
http://www.grapefruit.ch/iBackup
What kind of problems did you have with Retrospect? I’m still using Retrospect 5.0.238 for years, and even with alle versions of Mac OS X (incl. 10.4.3) I never met any problem.
On August 30, 2005, Patrick Haney said:
…Now I can finish erasing the old drive and throw it in the trash with a smile on my face.
END of quote
I no longer believe that software will fulfill the role of cleaning up. I no longer throw hard disks in the trash. I recommend destroying them with a good sledge hammer, the cost of the sledge hammer is well worth the loss of data that can be obtained using good forensic software.
See “Remembrance of Data Passed: A Study of Disk Sanitization Practices” IEEE Security & Privacy
http://www.simson.net/clips/academic/2003.IEEE.DiskDriveForensics.pdf
Which states on page 19 on the right side…
The most common techniques for properly sanitizing
hard drives include
ï Physically destroying the drive, rendering it unusable
ï Degaussing the drive to randomize the magnetic domainsómost likely rendering the drive unusable in the process
ï Overwriting the driveís data so that it cannot be recovered
Sanitizing is complicated by social norms.
Clearly, the best way to assure that a driveís information is protected is to physically destroy the drive.
END OF CITATION
Nice blog and article. All valid points you make but I would like to defend Retrospect to your readers. I have been a faithful Retrospect user for over 10 years. I have not tried many other back up systems because Retrospect has never let me down and the company (Dantz) is one of the best I have come across in 17 years of Mac computing. I especially like their technical support and that they keep their software up-to-date without the unnecessary upgrades so many companies use as cash cows. I would not consider the program easy to use or intuitive but now that the learning curve is over, there is nothing to quibble with. I just recently moved over to OS X (see how cautious I am) and updated Retrospect. I am very pleased to see it has not been needlessly changed just to look fresh and new.
Now a word about back ups in general. As far as I am concerned, a back up strategy that does not have regular, scheduled offsite backups is not much of a strategy at all. I back up our two computers (home business) to DAT tape and have 3 sets that I cycle. One is always at an offsite location and the other two in the house (one by the computer for easy access and one in a fire proof box). Can this plan be foiled? Of course! But having any offsite (at a relative’s house, the bank safe deposit, stashed in the back yard inside a tree hole, etc.) is, IMHO, exponentially safer than having no offsite at all. So, do yourself a favor and develop a backup plan with a regular offsite location. Just some friendly advice from an old man.
The comment was made that backups should be taken off-site on DVDs. Does anyone know if there is an easy way to do this? I have a bootable backup of my Powerbook on an external 120Gig FW drive. What’s the best way to take a snapshot of this to store off-site?
Another mirroring/cloning tool is Carbon Copy Cloner from Mike Bombich
http://www.bombich.com/
It works for straight manual cloning of bootable drives, but also has a scheduled cloning option.
It’s free, but you can donate to Mike if you use it.
Dave
@Jan, Mike & Dave: Yes, there are many sync programs out there, and many programs that can create bootable backups. Those utilities mentioned, while all fine applications, do not do archiving — which is central to this backup strategy I’ve outlined. (ChronoSync does now do archiving, but not bootable backups…)
I also use an external drive for my backups, using Lacie’s freeware Silverkeeper I would like to attract your attention to the fact that using an external drive puts you at risk of loosing everything not only in case of fire, but also in case of burglary, which is, depending where you live, not so unlikely. In that event, the computer and external drive are among the first items to leave…
Regular backups should be made and stored offsite. I use DVDs for this task.
Also, as we are talking about data security, we should always monitor the smart status of hard drives. On the mac, the free SMARTReporter will place an icon on the status bar and will send you an email in case of SMART event.
cheers
Raphael
A better way to mirror the disk is to use Tiger’s “Disk Utility” to set up a RAID system. Then the mirroring will be done in real time. Data will be written once to each drive. Reads can be done round robin and may pick up some performance from parallelism. The best part is that the system will not crash if a disk fails and you can continue runnig on one less drive until you can swap out the drive. Even with a mirror you stil need daily snapshots and offsite storage
Constant Coolsma: It’d take a complete article to describe all the problems I had with Retrospect. Not only technical problems, but also problems due to the backup model/process they enforce.
Chris: I’m not interested in RAID. The key component to my backup strategy is “archiving,” not up-to-the second mirror synchronization. And RAID doesn’t give you archiving.
The easiest way to take a snapshot of an external drive is to buy another external drive, mirror to it, and take it off-site.
Hard drives are cheap.
Here’s another concern: I’d like to make backups that are secure from prying eyes. I suppose the best way (or at least the most obvious solution) would be to make an encrypted disk image and then backup to it. Has anyone had any luck with this? How about just turning on FileSafe and then backing up your home directory’s disk image?
Retrospect has always been a big problem. Every client that I have ever had who has used Retrospect, has had problems with data loss of some type, whether it be subtle data loss like file modification dates being changed to the backup date, or more obvious data loss like files not being backed up at all, or files not able to be restored off the backup medium. It does not matter to me whether the problem is actual bugs in Retrospect, or user error resulting from the confusing user interface and options. Either way, I discourage the use of Retrospect.
The one thing that has always worked for me is Carbon Copy Cloner. It does one thing well. It’s free. It has never let me down.
Matt:
Hope Spain is treating you well, and you and family are cool. I am still living in New Jersey, but my client is Fannie Mae, in DC. No real contact with folks in Duluth other than my parents.
Thought of you when my brother-in-law was signed by a UK label and decide they want to move to Spain.
Darren
There were several tools here that I haven’t seen before. I’ll have to check them out.
I backup my system using the Disk Utility which works nicely because I can use my external hard drive to boot up my system on another Mac if mine goes down.
Thanks for writing this. I am updating my backup strategy, and detailed info on how others back up is a big help.
I was burned by .Mac backup when my PowerBook drive failed recently. Every few days I was copying my entire drive via rsync to a volume attached to my Mini. Nightly .Mac backups took care of the files which change often. Unfortunately, Backup’s restore option would run for hours, showing no movement on the progress bar. Eventually it would crap out. With a mixture of the command line and Finder I found that I could manually extract key files, but the process was tedious.
So, I’m looking for a new option for incremental backups and archiving. Synchronize! Pro X and ChronoSync are the top contenders. Have you used ChronoSync more since you wrote this piece? If you have any insight into how the two apps compare, I would love to read it.
Thanks,
–mkb
Hey Matt, thank you so much for this blog. It has a lot of good info that we can use. I work for a non-profit mission and this is our senario. We have a tower with two drives. The main one is use for the system and work, the second is the back up drive. This computer is networked through a hub to three other computers. At the moment I am using Silverkeeper, but i can’t get the other computers to back up over the network to the second drive. if the main computer is sleeping, then nothing works. i would like to back up all four computers ever night to the back up drive in the main computer tower. do you have any suggestions that would help make that work? Also when i do manually back up files and docements, i run into permission issues. I don’t know much about ‘permissions’ so if you have some advice there too, it would be greatly appreciated. thanks.
I must say that enjoy the concept of the internet, because there is so much info out there and now everyone can learn from each other. really cool. anyway, thanks in advance for your help.
Josh
I really need a backup solution that can wake a sleeping Mac… I’ve got Retropect 6 workgroup but if the worst bit of software I’ve ever used!
Hi Frank,
If I were you, I would probably buy a single external drive, partition it into two partitions:
As for brands, I recently bought LaCie and Maxtor, both of which have given me problems (disk problems). Our system administrator just bought for our main office Seagate drives. It seems that in general as disks get larger, their mean time to failure seems to go down, but probably in the future I’ll buy Seagate until that brand also demonstrates problems.
I am a recent Mac convert and have a Powerbook G4 running Tiger 10.4.6. I have yet to implement a backup process, so I was very interested in your fine article. I intend to follow it as closely as I can. But first I need to purchase at least one Firewire External hard drive. I would rather purchase complete unit(s) rather than building my own. I would appreciate any advice and recommendations you have about the available hardware. I am reading conflicting views about reliablity of Maxtor, LaCie, OWC, etc. My intention would be to a mirror of my 100GB internal drive and then do your suggested archiving. Would I need two separate external hard dirves to accomplish this?
Thnaks for your time and assisatnce.
Frank
I just bought an external Western Digital, Book Pro, 500 gig, Firewire 800 drive and I think I am going to partition it like you suggested. That way I can make one partition a backup drive (mirror) The drive came with Retrospect Express, but it seems like it has a learning curve on it.