Comments on: Using Amazon S3 to backup Media Temple’s Grid (gs)

By: Links of Interest | Proba

Links of Interest | Proba — Sun, 31 Aug 2008 22:15:28 +0000

[...] Warren has a comprehensive and excellent tutorial on creating a Ruby script to back up your entire web server, including databases, and upload them [...]

By: Automatically back up your entire web server files and databases to Amazon S3 | QuickPipe

Sun, 31 Aug 2008 04:25:04 +0000

[...] Warren has a comprehensive and excellent tutorial on creating a Ruby script to back up your entire web server, including databases, and upload them [...]

By: Christina

Christina — Wed, 27 Aug 2008 20:54:30 +0000

Matt,
I remember you had that error. I'll do what I can to investigate why this seems to not be working. It might be necessary to create two separate Cron jobs - one for the biggest site, one of the others - to see if that is an acceptable workaround.

Peter -- OK, try it now -- I changed the URL structure. It was working fine, I'm not sure what could have changed. If you still have issues, let me know.

Marcus -- yay! I'm so happy this worked for you!

By: Marcus McCurdy

Marcus McCurdy — Wed, 27 Aug 2008 20:13:50 +0000

Thanks for this great write up. I went through it last night and it worked like a champ and I learned a thing or two in the process. This really makes backing up a media temple site a breeze.

By: Peter

Peter — Wed, 27 Aug 2008 14:31:26 +0000

I was hoping I could download your full backup script and customize it to my needs, but it looks like access it denied on the file from S3. Is there any chance you could make it public?

By: Matthew Barker

Matthew Barker — Wed, 27 Aug 2008 04:15:34 +0000

Nope, not getting any errors that matter it seems. Earlier I reported this error:

Connection reset: Connection reset by peer
99 retries left, sleeping for 30 seconds
98 retries left, sleeping for 30 seconds
...

But that doesn't seem to really matter. I am backing up 10 sites, with only 1 being larger than 500 MB; the gzipped/tarred file is currently 1.3 GB in size. The odd thing about all of this is that sometimes everything works when I do it manually, but that is only sometimes. It generally quits when transferring the 1.3 GB file to Amazon, with no error messages encountered. But with the cron job running, it sometimes quits when tarring the 1.3 GB site, but generally tars everything just fine, but doesn't transfer a thing. That's the hard part about trying to troubleshoot this problem; sometimes it works, sometimes it doesn't; and when it doesn't work, it doesn't die at the same place every time.

By: Christina

Christina — Tue, 26 Aug 2008 22:29:21 +0000

Matt,
Are you getting an error of any kind? About how much data is being copied over before it dies? I've successfully copied over more than 500 megabytes before using the script (a test case and also a backup of some photographs I uploaded to my gs temporarily when at my parent's house). Let's see if we can figure out why it isn't working.

By: Matt

Matt — Tue, 26 Aug 2008 20:35:05 +0000

The backup part works great for me, but not the s3sync. Cron job doesn't even bother with copying anything over. When I do the site copy to S3 manually, it usually dies after copying just a few sites. Wish I could get that part working as that is the important part!

By: Christina

Christina — Tue, 26 Aug 2008 13:40:58 +0000

Philip,
I was having some problems having effective recursive backups, so it's just doing everything. Your comment reminds me to re-investigate the best/most effective way to do it recursively though (it would be like a few character changes to the script), so I'll do that later this week and post an update with my findings. Realistically, your databases are going to be changing more frequently than your actual directories, so you can always set a separate CRON job to run the databases and certain folders every day, and other folders less frequently. That's what I do anyway -- my blogs and databases are backed up daily and a few domains that are just basically image storing for right now get updated once a week or once a month.

By: Links of Interest - CSS-Tricks

Links of Interest - CSS-Tricks — Tue, 26 Aug 2008 12:46:12 +0000

[...] Warren has a comprehensive and excellent tutorial on creating a Ruby script to back up your entire web server, including databases, and upload them [...]