My Backing Up Workflow aka ‘My Battle with Paranoia’

Written by: Glyn Dewis

Published: October 29, 2010

Category: General

“Hard Drives are only ever in one of two states…failing or about to fail” ~ Unknown

Now I’ve no idea who it was said this but it’s something that’s stuck with me and I’ve no doubt has contributed to my paranoia about backing up, but then as a photographer I don’t think I’m alone. Certainly the experience of having a hard drive fail on me earlier in the year didn’t help…despite Apple’s Time Machine saving the day.

Workflow is something that’s very personal born out of previous experiences & recommendations but also something that’s forever evolving in the search for the perfect solution. Speaking for myself I’ve been looking for a better way of doing things for sometime now as I’ve come to realise that the attitude of “That’ll do” maybe isn’t the way to be.

My Backing Up Workflow
Over the past few months I’ve made a few changes with my Workflow….both in Post Production and with Backing Up, so I thought I’d share my own Backing Up Workflow with you here. Now I understand that we all do things differently but this is what I’m doing (at the moment) and I’ll bet you any money it’ll change again at some point in the future; such is the way in the digital world.

Point of Capture
More and more I’m shooting tethered to my MacBook Pro with images going directly into Lightroom but of course there are also the shoots when I’m shooting directly onto the cameras memory cards so here’s what I do in both of those situations:

Direct to the camera
One of the many things I love about shooting with a Nikon D3 is that there are slots for two (2) memory cards and consequently a few choices as to how to the camera uses them. One way is so that you have the second card act as an overflow so you have more space to store files as you shoot and another is to have RAW images appear on one card and the same images in JPEG format written to the other.

Now I always shoot in RAW so what I do is choose the other option which is to have the camera write identical images on both cards so immediately at the point of capture I have a backup. Setting up the D3 to perform this backup is really simple and accessed via the SHOOTING MENU:

Before I was shooting with a Nikon D3 I used to use an Epson P5000 Multimedia Player to backup the cards immediately at the end of the shoot but to be honest it seemed to take forever so I stopped using it. I know now that Epson have since updated this unit to the Epson P7000 which is a massive improvement and very popular so who knows in the near future I may well go back to using one.

Shooting Tethered
Whenever I’m shooting in the studio and more increasingly ‘on location’ I’m shooting with my Nikon D3 tethered to my MacBook Pro with images going directly into Lightroom, but the one thing I’m not keen on is the fact that the images are not written to the memory cards. So to make sure that I’m as safe as possible I have an external Hard Drive attached to the MacBook Pro also. This 500Gb Hard Drive is partitioned, giving over 400Gb to Time Machine and the remaining 100Gb to image files.

Basically as I shoot, images appear in Lightoom and those original files are written onto the MacBook Pro’s Hard Drive. As this happens I use a piece of software called SuperDuper to then Backup those original files on to the external Hard Drive, so now I have two(2) versions of the same image. Finally, every hour the whole system is backed up using Time Machine incase the unthinkable happens:

This whole process happens automatically leaving me to just keep shooting and the only thing I manually do is to make another Time Machine backup when we’re all finished and packing up.

Importing & Backing Up
Ok so once the shoot is over the next stage is to get the images into the main computer for sorting through and Post Production.

I use a superb piece of software called Photo Mechanic for importing the image files directly into a folder on my desktop which I have named ‘PM_IMPORT’ and the reason I use Photo Mechanic for importing the files is that it’s blisteringly fast. I used to import files directly into Lightroom but always found it a slow process plus another advantage of Photo Mechanic is that I can import from more than one card reader at a time.

*Note: If I’ve been shooting tethered then the files are imported directly off the MacBook Pro’s external Hard Drive which I connect to the iMac.

Ok so to explain the above workflow diagram…

  • Image files are imported off the memory cards and/or the external Hard Drive via Photo Mechanic into a folder on the desktop called “PM_IMPORT”
  • I’ll then sort through these RAW files selecting the ‘keepers’ and it’s these ‘keepers’ that are then imported into Lightroom.
  • All of the RAW ‘keeper’ files are then backed up onto the Drobo, andย Time Machine is activated manually to create an entire system backup.
  • The folder in which all the RAW files from every shoot are stored is backed up ‘off site’ automatically using Carbonite; a superb online utility that uploads your files to a secure server when your computer is inactive…that way it doesn’t interrupt your internet speed if you’re browsing the web, using email etc…
  • Finally I burn a copy of the RAW files onto a DVD and store this in a Fire Proof Safe, and also make a copy of that DVD and store it off-site.

* I also have an external Hard Drive that is used to automatically back up Time Machine using SuperDuper.
* Backing up onto DVD used to be all I did but having heard that DVD’s might degrade over time I thought it about time for a new way of doing things)

Once this whole process has gone through, then and only then will I format the memory cards and/or delete the files off the MacBook Pro’s external Hard Drive.

Now there are things that I do at a later stage once I’ve gone through all the Post Production such as exporting all the edited images to their own folder and backing this up on the Drobo, Time Machine and Carbonite. That’s something I do since I had a catalogue corrupt in Lightroom and all my edits were lost; like I said workflow is born out of experiences ๐Ÿ™‚

โ€ข ย  ย  โ€ข ย  ย  โ€ข

So there you have it, my own personal backing up workflow which like I said at the beginning I’m sure will change over time but for now I’m quite happy with this.
BIG thanks to everyone on Twitter that recommended some of the utilities that I’m now using such as Carbonite and SuperDuper; awesome bits of kit!

Sure there’ll be ways that you do things very differently to the way I do, and that’s great because there’s no real right or wrong way so long as you backup in some form or another, and that being said it’s always great to get some feedback, so if you have any questions, comments or maybe some recommendations then as always feel free to make use of the comments section below.

Enjoy ๐Ÿ™‚

ps> I was going to ask if you thought that doing this amount of backing up made me paranoid but then decided against it as you were probably talking about me last night anyway to your friends saying exactly that ๐Ÿ™‚

Keep up with Glyn โ€˜Day to Dayโ€™ on Twitter
Get more โ€˜Behind the Scenesโ€™ by becoming a โ€˜Fan on Facebookโ€™

You may also like…

Interviewed by Ron Clifford
Interviewed by Ron Clifford

A few weeks back I was invited onto Ron Clifford YouTube LIVE Channel to speak with him about all things photography....

20 Comments

  1. Keith Hammond

    Paranoid yes the whole post smacks of paranoia ๐Ÿ™‚
    Only ribbing you mate, you just cant take chances with your digi negs, as i said in my last comment we used to keep our film negs in a draw or box and not give them another thought untill we wanted a re print but because we know electronics no matter how sophisticated can fail at any time we have to do all we can to cover that situation.
    Gotta say that is some set up mate, the more you can automate the system the easier it will be.
    I wonder what the future has in store for us re storage, i suppose it should get cheaper, smaller and easier we will have to wait and see in the mean time back up, back up again and may be once again just in case ๐Ÿ™‚
    BTW Have you given Neal a list of all this gear he has to get next, i bet he thought it stopped with the purchase of his D700 eh ๐Ÿ™‚ ๐Ÿ™‚ ๐Ÿ™‚
    PS i think i will be going down the Mac road soon so a beer and a Mac chat needs to be arranged

    Reply
    • Glyn

      @Keith…Yeah I guess the future is going to be really interesting when it comes to backing up; with storage solutions getting smaller and alot more use of ‘Cloud’ utilities I expect…but we’ll see.

      Re Neal…he’s been away fishing this past week so he knows nothing about this…yet ๐Ÿ™‚

      Reply
  2. Neil Glover

    Great post mate and very timely since I’m reviewing my backup strategy right now. I’ve been using external hard drives but even those are now running out of space. I know I need to get better at deleting the non-keepers because there is little point in me backing those up. Out of interest are you importing into lightroom to sort out the keepers?

    I’m considering the Drobo too but it’s a pricey option. What’s your opinion on it? I wonder whether to just buy a couple of huge hard drives and do the striping myself.

    I’ve been using AllSynch which is a great bit of free software for synching my laptop with external drives for back up. You can set up several jobs and have the synch go in various directions.

    cheers

    Reply
    • Glyn

      @Neil…Hi mate, thanks for stopping by and commeting.
      Regarding importing and sorting out the ‘keepers’ I use Photo Mechanic to get the files off the memory card or external hard drive and they then go into a folder on my desktop called ‘PM_Import’. Once they’re in the computer I then use the browser window in Photo Mechanic to sort through the images and put my initial keepers into another folder designated to the shoot. It’s this folder that I then import into Lightroom and then start working on. Occasionally I’ll reject the odd photo or two once they’re in Lightroom and I’ve had another look through them.

      Once they’re into Lightroom and all the backing up had been done I’ll then reformat the memory cards (in camera) or delete them off the external HDD and the MacBook Pro…does that make sense?

      Thanks for the ‘heads up’ re AllSync…I’ll head over and check that out cos I’m liking the sound of what you say it can do.

      Cheers,
      Glyn

      Reply
  3. Dave Clayton

    Great article Glyn, even though I am not a photographer I do create a lot of Photoshop and Illustrator files as well as photos. Seriously looking at Carbonite for offsite back up AND for releasing some space on my Mac drive and backup drives for both essential and non-essential files.
    I too have lost work in the past an now use time machine AND 3 external back up drives.

    Reply
    • Glyn

      @Dave…Hi Mate. Yeah horrible feeling when you lose the files huh ๐Ÿ™ … not good.
      Carbonite seems to work really well; the first backup takes quite a while but once that’s done it flies along nicely.

      Cheers,
      Glyn

      Reply
  4. Shivakumar

    Hey Glyn,

    Thats a very aptly written blog with full info on what goes on in your shooting and backup workflow. But i have few specific queries which others might be interested in knowing also ๐Ÿ™‚

    a. Am i right in saying you have about 9 backups or copies of your files at the end of a day ?

    1 Macbook pro
    1 Macbook pro ext HDD
    1 Macbook pro Timemachine
    1 iMac HDD
    1 DROBO
    1 Carbonite
    2 iMac Timemachine
    1 DVD

    Does this mean at the end of a month or a time period you flush out everything in Macbook pro and/or iMac and/or Macbookpro HDD – and retain only others ?

    b. I presume the DROBO is a RAID type backup so there again you have multiple instance or copies of your data ?

    c. How do you organize your files when you shoot both when you shoot on camera and when on tethered – ie folder structures and file naming – and does everything and anything from your shoot goes into all the backup or only keepers ?

    d. Do you backup your high res processed tiff images and/or webimages + video tutorials etc in the same way ? ie do they fit in the same workflow or they get backed up in a separate channel ?

    Hope i didnt bombard you with too many questions ๐Ÿ™‚

    Once again thanks for the insight into your workflow model

    Cheers,
    Shiv

    Reply
    • Glyn

      @Shiv…Thanks for dropping by and commenting mate.
      Re your queries…

      If I’ve been shooting tethered then once the images are into the iMac and then backed up fully i then remove them off the external HDD and the MacBook Pro but leave the Time Machine to do it’s thing.
      Image files remain in the iMac for editing and once they’re edited, the final versions are exported into a folder of their own and this folder is also backed up onto the Drobo, Time Machine, MCarbonite etc… I’ll then remove the RAW files etc off the iMac leaving only the folder of edited images in Lightroom; does that make sense?

      Re the Drobo…I love it!!! It basically shares your data over a series of discs and should one fail it then empties it and shares it across the others. Certainly helps with the sleep at night ๐Ÿ™‚

      I have no specific way of organising files when shooting both tethered and in camera for the same job. I generally just import them all and back up in the normal way.

      All of the video files, web images etc are all backed up automatically onto the Drobo also using SuperDuper; saves me having to think about it. A ‘watched’ folder is backed up each evening.

      Hope that helps mate, but feel free to give me a shout if there’s anything else.
      Cheers,
      Glyn

      Reply
  5. Dan Davies

    Well you’ve gone and done it boots and braces. Typically thorough of you Glyn.

    I’ve always struggled with the competing needs of keeping something as simple and automated as possible so that it runs without thinking and having the security of knowing that you’ve manually done something and you actually know it’s worked. You seem to be pretty close to acheiving both of these.

    I’m currently trialling BackBlaze (backblaze.com) as a Carbonite type solution. What I’m not sure though is how able it is to keep up with the new images during busy periods and of course theres a risk to any type of cloud storage should the company go bust. Let’s hope we never find out eh?

    Cheers

    Dan

    Reply
    • Glyn

      @Dan…Thanks for commenting mate ๐Ÿ™‚
      Yeah backing up as you know is v.important but it’s keeping it simple and remembering to do it that’s the hard part. Thankfully utilities like SuperDuper, Time Machine etc take the worry of forgettinbg to do something out of the equation.

      Thanks for the ‘heads up’ re BackBlaze…sounds interesting so I’ll go check that out too.

      Cheers,
      Glyn

      Reply
  6. Keith Hammond

    “Iโ€™ll then remove the RAW files etc off the iMac leaving only the folder of edited images in Lightroom”

    Don’t you keep your RAW’s, or have i missed that back up part ?

    Reply
    • Glyn

      @Keith…The original RAW files have already been backed up on the Drobo and the two (2) external HDD’s acting as Time Machine and Time Machine Backup plus Carbonite and the DVD’s so I just get rid of them off the iMac’s HDD to free up space. I definitely keep the original RAW files but just not on the iMac once all the editing has bee done…hope that makes sense ๐Ÿ™‚

      Reply
  7. Keith Hammond

    yes…….i got a little lost in what was saved where ๐Ÿ™‚
    that Drobo looks good, been watching some demo vids on it…….another item to add to the list

    Reply
    • Glyn

      @Keith…Worth it’s weight in gold mate; love it ๐Ÿ™‚

      Reply
  8. A.J. Wood

    That’s quite a lot going on, but I can understand the need for 100% uptime & multiple redundancies. I would ask the following questions:

    1. Where/How do you find an image if you need to recall it later? I assume your edits are all divided out into Collections or do you goto the Exported Edits folder you created?

    2. Are you backing up both the RAW “keepers” and the edited versions?

    3. Are you writing out your changes from Lightroom’s database to the RAW files? (Hence there would not be a need to export files, except for the client)

    4. Have you tested the backups? (Yeah, I know I hit you with this one already, but we weren’t on the blog)

    I have not incorporated Photo Mechanic into my workflow, although I continually hear good things about it. My process is similar to Scott Kelby’s making an initial backup of the RAW files, then overwriting that backup with the edited versions later.

    Reply
    • Glyn

      @A.J…It sure is mate. Re your Q’s…

      1. Yeah all my edits are in their own collections but to be honest this is only something I’ve made use of over the past couple of months so it was a nightmare to find files not knowing where they were stored. All of the image files are imported with individual keywords relating to that specific shoot as opposed to generic terms like…’portrait’ ‘landscape’ and so on. Sure I use those but I also keyword the subjects name in there too; easy to search for them then ๐Ÿ™‚

      2. Yes mate…keeping both which I guess does mean more space but hey, that’s my paranoia kicking in ๐Ÿ™‚

      3. Once all the editing has been done I generally export to the ‘Export’ folder. This is ‘generally’ a temporary place they are held before sending off or burning to discs etc…

      4. I’ve tested the backups so far as loading images in off the HDD’s but no more; oh no now you’ve got me thinking…lol ๐Ÿ™‚

      PhotoMechanic is superb mate but to be honest not really necessary unless like me you want to import really quickly and from a couple of readers at a time.

      Hope that makes sense but any suggestions you may have…I’m all ears ๐Ÿ™‚

      Cheers,
      Glyn

      Reply
  9. David Kelly

    Geez that’s a set-up and a half there Glyn but absolutely understand the need for being safe than sorry!

    Haven’t had a look a Carbonite but I currently use Mozy to back up my PC’s files off site. The problem I’ve had until recently though is just the amount of time it takes to back-up upstream. Thankfully with my new fibre DSL service recently installed it now whizzes away upstream at mega speeds.

    Need to get some extra HDs soon though as I’m running out of space for drive images & back-up. Nice that HD prices are always falling down in price, just pity that our space requirements keep going up ๐Ÿ™‚

    Regards,
    David

    Reply
    • Glyn

      @David…Yeah the ‘off site’ backing up can take quite a while…especially for the first backup. Your mega speed uploads sounds superb mate; hopefully in the not too distant future we can all benefit from warp speed broadband ๐Ÿ™‚

      Might be worth looking at a Drobo mate to save on space especially as the 4 Bay version seems to be dropping in price on an almost daily basis.

      Must try and meet up soon,
      All the best to you,
      Glyn

      Reply
  10. Callum Winton

    A footnote to your Lightroom corruption that you had ….
    If you save the changes to .XMP then you only lose virtual copies if the catalogue corrupts.

    In LR go to:
    Edit | Catalogue Settings | Automatically write changes into XMP

    I used to convert to .DNG, but by using .XMP this saves a lot of time when I make lots of changes as the .xmp file are tiny.

    I still backup everything though, but using .xmp is a big time saver and safer in the longrun.

    CW

    Reply
    • Glyn

      @Callum…That’s great; thanks for that ๐Ÿ™‚

      Reply

Trackbacks/Pingbacks

  1. New Mac & Backup strategy | bikes and things - [...] My Backing Up Workflow aka โ€˜My Battle with Paranoiaโ€™ – Gyn Dewis [...]

Submit a Comment

Your email address will not be published. Required fields are marked *