christinawarren.com

For news tips, pitches, speaking engagements or media inquiries: Contact

S3 to backup Media Temple’s Grid (gs)">Using Amazon S3 to backup Media Temple’s Grid (gs)

Proper backups are like eating your vegetables -- we all say we'll do it and that it is a good idea, but it is so much easier NOT to do it and eat Oreo cookies instead. Don't risk losing your website because you didn't bother backing up.

Prop­er back­ups are like eat­ing your veg­eta­bles — we all say we’ll do it and that it is a good idea, but it is so much eas­i­er NOT to do it and eat Oreo cook­ies instead. Then you wake up one day, are 25 years old and are a real­ly picky eater and annoy your boyfriend because you won’t go eat at the Indi­an place he loves that doesn’t have a menu but only serves veg­e­tar­i­an stuff that scares you. And the peo­ple at Sub­way give you dirty looks when you tell them you don’t want any­thing on your sand­wich. Don’t risk los­ing your web­site because you didn’t both­er back­ing up.

Update: I post­ed a video tuto­r­i­al that walks through all of these steps here. I still rec­om­mend read­ing through this page because the video tuto­r­i­al assumes that you will be fol­low­ing these steps.

This a tuto­r­i­al for cre­at­ing an auto­mat­ed back-up sys­tem for (mt) Media Temple’s (gs) Grid Ser­vice. Although it will almost cer­tain­ly work on oth­er servers and con­fig­u­ra­tions, this is writ­ten for users who are on the Grid who want an easy way to do auto­mat­ed back­ups. I per­son­al­ly feel most com­fort­able hav­ing my most impor­tant files backed-up off­site, so I use Amazon’s S3 ser­vice. S3 is fast, super cheap (you only pay for what you use) and reli­able. I use S3 to store my web­site back­ups and my most impor­tant com­put­er files. I spend about $1.50 a month, and that is for near­ly 10 GBs of stor­age.

You can alter the script to sim­ply store the data in a sep­a­rate loca­tion on your serv­er (where you can then just FTP or SSH in and down­load the com­pressed archive), but this process is assum­ing that you are using both the (gs) and S3.

This tuto­r­i­al assumes that you know how to login to your (gs) via SSH using either the Ter­mi­nal in OS X or Lin­ux or PuT­TY for Win­dows. If SSH is still con­fus­ing, check out (mt)‘s Knowl­edge Base arti­cle and take a deep breath. It looks more scary than it real­ly is.

Acknowledgements

I would be remiss if I didn’t give a GIGANTIC shout-out to David at Stress Free Zone and Paul Sta­ma­tiou (I met Paul at the Tweet-up in March) who both wrote great guides to back­ing stuff up serv­er side to S3. I bla­tant­ly stole from both of them and rolled my own script that is a com­bi­na­tion of the two. Seri­ous­ly, thank you both for your awe­some arti­cles.

Fur­ther­more, none of this would even be pos­si­ble with­out the bril­liant S3Sync Ruby util­i­ty.

Installing S3Sync

Although PHP and Perl script exist to con­nect with the S3 servers, the Ruby solu­tion that the S3Sync dudes cre­at­ed is much, much bet­ter.

The (gs) already has Ruby on it (ver­sion 1.8.5 as of this writ­ing), which is up-to-date enough for S3Sync.

OK, so log-in to your (gs) via SSH. My set­tings (and the defaults for (gs), I assume) are to place you in the .home direc­to­ry as soon as you login to SSH.

Once you are at the com­mand line, type in the fol­low­ing com­mand:

[bash]wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz[/bash]

This will down­load the lat­est S3Sync tar­ball to your .home fold­er

[bash]tar xvzf s3sync.tar.gz[/bash]

This uncom­press­es the archive to its own direc­to­ry.

[bash]rm s3sync.tar.gz

cd s3sync

mkdir certs

cd certs

wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar

sh ssl.certs.shar

cd ..

mkdir s3backup[/bash]

That will delete the com­pressed archive, make a direc­to­ry for cer­tifi­cates (certs), down­load an SSL cer­tifi­cate gen­er­a­tor script, exe­cute that script and cre­ate a back­up direc­to­ry with­in the s3sync direc­to­ry called “s3backup.”

Now, all you need to do is edit two files in your new­ly cre­at­ed s3sync fold­er. You can use TextE­d­it, Text­Mate, NotePad or any oth­er text edi­tor to edit these files. You are only going to be chang­ing a few of the val­ues.

I edit­ed the files via Trans­mit, but you can use vi straight from the com­mand line if you are com­fort­able.

The first file you want to edit is called s3config.yml.sample

You want to edit that file so that the aws_access_key and aws_secret_access_key fields cor­re­spond to those from your S3 account. You can find those in the Access Infor­ma­tion area after log­ging into Amazon.com’s Web Ser­vices page.

Make sure that the ssl_cert_dir: has the fol­low­ing val­ue (if you cre­at­ed your s3sync fold­er in the .home direc­to­ry): /home/xxxxx/users/.home/s3sync/certs were xxxxx is the name of your serv­er.

You can get your entire access path by typ­ing in [bash]pwd[/bash] at the com­mand line.

Save that file as s3config.yml

The next step is some­thing I had to do in order to get the s3 part of the script to con­nect, but it may not be required for all serv­er set-ups, but it was for the (gs).

Edit the s3config.rb file so that the area that says [bash]confpath = [xxxxx][/bash]

looks like this [bash]confpath = [“./”, “#{ENV[‘S3CONF’]}”, “#{ENV[‘HOME’]}/.s3conf”, “/etc/s3conf”][/bash]

Writing the backup script (or editing mine)

OK, that was the hard part. The rest is pret­ty sim­ple.

I cre­at­ed the fol­low­ing back­up script called, “backup_server.sh” This script will back­up the con­tent of the domain direc­to­ries you spec­i­fy (because if you are like me, some of your domain fold­ers are real­ly just sym­links) and all of your MySQL data­bas­es. It will then upload each direc­to­ry and data­base in its own com­pressed archive to the S3 Buck­et of your choice. Buck­ets are unique, so cre­ate a Buck­et using either the S3Fox tool or Trans­mit or anoth­er S3 man­ag­er that is spe­cif­ic for your web­site.

This is the con­tent of the script:

[bash]#!/bin/sh

A list of website directories to back up

websites=“site1.com site2.com site3.com”

The destination directory to backup the files to

destdir=/home/xxxxx/users/.home/s3sync/s3backup

The directory where all website domain directories reside

domaindir=/home/xxxxx/users/.home/domains

The MySQL database hostname

dbhost=internal-db.sxxxxx.gridserver.com

The MySQL database username — requires read access to databases

dbuser=dbxxxxx

The MySQL database password

dbpassword=xxxxxxx

echo date “: Begin­ning back­up process…” > $destdir/backup.log

remove old backups

rm $destdir/*.tar.gz

backup databases

for dbname in echo 'show databases;' | /usr/bin/mysql -h $dbhost -u$dbuser -p$dbpassword do if [ $dbname != “Data­base” ]; then echo date “: Back­ing up data­base $dbname…” » $destdir/backup.log /usr/bin/mysqldump –opt -h $dbhost -u$dbuser -p$dbpassword $dbname > $destdir/$dbname.sql tar -czf $destdir/$dbname.sql.tar.gz $destdir/$dbname.sql rm $destdir/$dbname.sql fi done

backup web content

echo date “: Back­ing up web con­tent…” » $destdir/backup.log for web­site in $web­sites do echo date “: Back­ing up web­site $web­site…” » $destdir/backup.log tar -czf $destdir/$website.tar.gz $domaindir/$website done

echo date “: Back­up process com­plete.” » $destdir/backup.log

The directory where s3sync is installed

s3syncdir=/home/xxxxx/users/.home/s3sync

The directory where the backup archives are stored

backupdir=/home/xxxxx/users/.home/s3sync/s3backup

The S3 bucket a.k.a. directory to upload the backups into

s3bucket=BUCKET-NAME

cd $s3syncdir ./s3sync.rb $backupdir/ $s3bucket: [/bash]

For (mt) Media Tem­ple (gs) Grid Serv­er users, you just need to change the “site1.com” val­ues to your own domains (you can do as many as you want) and sub­sti­tute all the places where marked “xxxxx” with your serv­er num­ber (again, you can find this by enter­ing “pwd” at the com­mand line) and with your data­base pass­word (which is vis­i­ble in the (mt) con­trol pan­el under the “Data­base” mod­ule.

Make sure you change the val­ue at the end of the script that says “BUCKET-NAME” to the name of the S3 Buck­et you want to store you back­ups in.

Now that you have edit­ed the script, upload it to your /data direc­to­ry.

Change the per­mis­sions (you can do this either via SSH [bash]chmod a+x backup_server.sh[/bash] or using your FTP client to 755.

Now, test the script.

In the com­mand line type this in: [bash] cd data

./backup_server.sh

[/bash]

 

And watch the mag­ic. Assum­ing every­thing was cor­rect­ly input, an archived ver­sion of all your domain direc­to­ries and all of your MySQL data­bas­es will be put in a fold­er called “s3backup” and then uploaded direct­ly to your S3 serv­er. Next time you run the script, the back­up files will be replaced.

Check to make sure that the script is work­ing the way you want it to work.

Auto­mate the script

You can either run the script man­u­al­ly from the com­mand line or set it to run auto­mat­i­cal­ly. I’ve set mine to run each night at mid­night. To set up the cron job, just click on the Cron Jobs but­ton in the (mt) Admin area:

 

(mt) AccountCenter - filmgirl.tv : (gs) GridControls Uploaded with plasq’s Skitch!

 

 

and set you para­me­ters. The path for your script is: /home/xxxxx/data/backup_server.sh.

Enjoy your back­ups!

One note: The com­pressed domain archives retain their entire direc­to­ry struc­ture, as such, there is a .home direc­to­ry that may not appear in Find­er or Win­dows Explor­er unless you have invis­i­ble or hid­den files turned on. Don’t wor­ry, all your data is still retained in those archives.

Update (7/27/2008): If you are get­ting an error that says some­thing like Permanent redirect received. Try setting AWS_CALLING_FORMAT to SUBDOMAIN

Add the fol­low­ing array to your s3config.yml file AWS_CALLING_FORMAT: SUBDOMAIN

The error is either because your buck­et is in the EU or there is some­thing else funky with its URL struc­ture. Chang­ing that val­ue should allow the script to per­form as intend­ed.

Feeds-

Alltop, all the cool kids (and me)

Instagram

Follow me on App.net

Calendar-

June 2008
S M T W T F S
« May   Jul »
1234567
891011121314
15161718192021
22232425262728
2930  

112 people have left comments

Paul Sta­ma­tiou - Gravatar

Paul Sta­ma­tiou said:

S3 is great huh? That s3config.rb path thing must have been a change with the recent s3sync as I had to do the same thing when I just upgrad­ed.. didn’t have to do that when I had pub­lished it. Any­ways, thanks for bring­ing that to my atten­tion, I’ve updat­ed my arti­cle.

Posted on: June 24, 2008 at 6:39 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Thanks for your orig­i­nal tuto­r­i­al and arti­cle Paul — it helped me out tremen­dous­ly when I was set­ting the whole thing up. And yes, S3 is the bees knees as they say.

Posted on: June 24, 2008 at 6:43 pmQuote this Comment
George Orn­bo - Gravatar

George Orn­bo said:

Great write up. The Ama­zon S3 ser­vice is per­fect for back­ing up a web serv­er remote­ly, safe­ly and cheap­ly. I’m also using it to do off site back­ups for machines inside the net­work. For a small busi­ness it is a great solu­tion.

Posted on: June 25, 2008 at 3:00 amQuote this Comment
Mike Mar­ley - Gravatar

Mike Mar­ley said:

Neat arti­cle Christi­na.

/me adds it to list of KB arti­cles that need to be writ­ten.

Mike Mar­ley Senior Tech­ni­cal Sup­port (mt) Media Tem­ple, Inc.

Posted on: June 25, 2008 at 10:47 amQuote this Comment
Ross - Gravatar

Ross said:

Nice post.…very very detailed. Hope­ful­ly you put it in the how to part of the MT forum?

Posted on: July 1, 2008 at 10:41 amQuote this Comment
Michael - Gravatar

Michael said:

Excel­lent resource, thank you. One note on my expe­ri­ence when con­fig­ur­ing on the gs:

I was receiv­ing errors like “-o is not a valid vari­able” or some­thing like this, when the script was try­ing to exe­cute the mysql dump. I changed it to –opt (vs. -opt in your script).

Thanks again!

Posted on: July 5, 2008 at 8:32 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Thanks for the info Michael! The script actu­al­ly DOES say –opt, but the way that “code” is dis­played on this page didn’t show the dash­es clear­ly (I’ll have to try to change that) — the down­load­able script has the cor­rect dash­es too. I’m glad that it worked for you and I appre­ci­ate the feed­back on the –opt thing. I’ll do my best to change the text dis­play now.

Posted on: July 6, 2008 at 2:37 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

OK — I fixed the code for­mat­ting! It should all be cor­rect­ly dis­played now.

Posted on: July 6, 2008 at 3:56 amQuote this Comment
duivesteyn - Gravatar

duivesteyn said:

I’ve mod­i­fied this to bet­ter suit CPan­el based sites with sql sup­port at http://duivesteyn.net/2008/amazon-s3-backup-for-webserver-public_html-sql-bash/

hope it helps some­one

Posted on: July 6, 2008 at 4:48 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

duivesteyn said: I’ve mod­i­fied this to bet­ter suit CPan­el based sites with sql sup­port at http://duivesteyn.net/2008/amazon-s3-backup-for-webserver-public_html-sql-bash/ hope it helps some­one

Oh that’s awe­some! Thanks for post­ing your script!

Posted on: July 6, 2008 at 4:51 amQuote this Comment

The­jesh GN said:

s3 is amaz­ing. I also use it deliv­er images and oth­er media files to blog since its fast.

Karl Hardis­ty - Gravatar

Karl Hardis­ty said:

Christi­na,

Thanks for putting in the time and effort to not only come up with the script, but to make it robust/structured/pretty enough for shar­ing. It’s good to know that some­one has test­ed it and put it to pub­lic scruti­ny, and that it’s worth­while. You’ve saved me (and plen­ty oth­ers I’m sure) a lot of time.

Much appre­ci­at­ed.

Posted on: July 7, 2008 at 7:56 amQuote this Comment
Josh Price - Gravatar

Josh Price said:

Sim­ple yet very effec­tive

Thanks!

Posted on: July 8, 2008 at 12:43 pmQuote this Comment
Host Dis­ci­ple » Blog Archive » Inside the host­ing mind of a blog­ger - Gravatar

Host Dis­ci­ple » Blog Archive » Inside the host­ing mind of a blog­ger said:

[…] host­ed there (calm down fel­las she is dat­ing some­one already). Christi­na wrote a very infor­ma­tive how-to on back­ing up your Medi­aTem­ple GS […]

Posted on: July 9, 2008 at 12:06 pmQuote this Comment
Patrick Siko­rs­ki - Gravatar

Patrick Siko­rs­ki said:

Can I bor­row your wis­dom? I’m get­ting some weird errors: [email protected]:~/data$ ./backup_server.sh tar: Removing leading /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing ‘/’ from mem­ber names You didn’t set up your envi­ron­ment vari­ables; see README.txt s3sync.rb [options] ver­sion 1.2.6 –help -h –ver­bose -v –dryrun -n –ssl -s –recur­sive -r –delete –pub­lic-read -p –expires=”” –cache-con­trol=”” –exclude=”” –progress –debug -d –make-dirs –no-md5 One of or must be of S3 for­mat, the oth­er a local path. Reminders: * An S3 for­mat­ted item with buck­et ‘mybuck­et’ and pre­fix ‘mypre’ looks like: mybucket:mypre/some/key/name * Local paths should always use for­ward slash­es ‘/’ even on Win­dows * Whether you use a trail­ing slash on the source path makes a dif­fer­ence. * For exam­ples see README.

Not sure where my prob­lem is, do you have any idea?

Awe­some arti­cle by the way!

Posted on: July 9, 2008 at 9:38 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Patrick Siko­rs­ki said: Can I bor­row your wis­dom? I’m get­ting some weird errors: [email protected]:~/data$ ./backup_server.sh tar: Remov­ing lead­ing ‘/’ from mem­ber names tar: Remov­ing lead­ing …

Patrick, OK, this was a prob­lem I had in the begin­ning, and I had to change the s3config.rb file so that confpath = ["./", "#{ENV['S3CONF']}", "#{ENV['HOME']}/.s3conf", "/etc/s3conf"] — make sure that has been changed and try again.

As for the tar: remov­ing lead­ing “/” from mem­ber names, that’s fine.

Hope this helps!

Posted on: July 9, 2008 at 11:41 pmQuote this Comment
Patrick Siko­rs­ki - Gravatar

Patrick Siko­rs­ki said:

Did this go as smooth­ly for you as it did for me lol. Now for some rea­son I’m get­ting this.

./s3sync.rb:28:in `require': ./s3config.rb:19: syntax error, unexpected tIDENTIFIER, expecting ']' (SyntaxError) config = YAML.load_file("#{path}/s3config.yml") ^ ./s3config.rb:25: syntax error, unexpected kEND, expecting $end from ./s3sync.rb:28

After you told me about that code, I real­ized that I didn’t copy it right. This is prob­a­bly some­thing just as stu­pid.

Posted on: July 10, 2008 at 8:56 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

Patrick, Did you rename the s3confi.yml.sample file to s3config.yml?

If you did, I’ll have to check the code­base (it is pos­si­ble a new ver­sion of S3sync was released since I’ve writ­ten the arti­cle) and inves­ti­gate.

We’ll get this work­ing!

This might be the sort of thing I should do a screen­cast of, from start to fin­ish, to sup­ple­ment the writ­ten guide. Hmm…

Posted on: July 10, 2008 at 10:20 amQuote this Comment
Patrick Siko­rs­ki - Gravatar

Patrick Siko­rs­ki said:

Yes I renamed the file. I guess a new ver­sion could have been released… but you didn’t write the arti­cle that long ago. Update your ver­sion of it and see if it breaks (back­up first lol). A screen­cast would be cool.…!

Posted on: July 10, 2008 at 10:28 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

OK — a new ver­sion has NOT been released, so I’m think­ing this is prob­a­bly as sim­ple as a mis-typed com­ma or peri­od some­where.

I’ll make a screen­cast today, going from start to fin­ish.

Posted on: July 10, 2008 at 11:04 amQuote this Comment
Patrick Siko­rs­ki - Gravatar

Patrick Siko­rs­ki said:

Awe­some, I’ll delete every­thing and start over when you make the screen cast. Thanks!

Posted on: July 10, 2008 at 11:53 amQuote this Comment
Karl Hardis­ty - Gravatar

Karl Hardis­ty said:

Christi­na,

Excel­lent news. I can guar­an­tee at least one view­er.

Posted on: July 10, 2008 at 12:08 pmQuote this Comment

Video Tuto­r­i­al: Auto­mate Media Tem­ple (gs) back­ups with Ama­zon S3 | www.ChristinaWarren.com said:

[…] This entry was post­ed on Sun, July 13th, 2008. You can fol­low any respons­es to this entry through the RSS 2.0 feed. You can leave a response, or track­back from your own site. Pre­vi­ous Entry […]

Matt - Gravatar

Matt said:

The back­ups are going fine, but the s3sync por­tion keeps giv­ing me:

Con­nec­tion reset: Con­nec­tion reset by peer 99 retries left, sleep­ing for 30 sec­onds Con­nec­tion reset: Con­nec­tion reset by peer 98 retries left, sleep­ing for 30 sec­onds Con­nec­tion reset: Con­nec­tion reset by peer 97 retries left, sleep­ing for 30 sec­onds Con­nec­tion reset: Con­nec­tion reset by peer … and so on

Any ideas?

Posted on: July 16, 2008 at 3:31 pmQuote this Comment
Matt - Gravatar

Matt said:

Hmmm… I guess that’s nor­mal? I checked my S3 buck­et and it had all the files there and in the right size. So are those mes­sages just there to say that it is still work­ing?

Posted on: July 16, 2008 at 3:47 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Hmm, I don’t get any of those Matt — but if the copies are trans­fer­ring over cor­rect­ly, I guess its fine.

Posted on: July 17, 2008 at 10:09 pmQuote this Comment
Cedric - Gravatar

Cedric said:

I tried 2 times, but it doesn’t work at all for me :

tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names Per­ma­nent redi­rect received. Try set­ting AWS_CALLING_FORMAT to SUBDOMAIN S3 ERROR: # ./s3sync.rb:290:in +': can't convert nil into Array (TypeError) from ./s3sync.rb:290:ins3TreeRecurse’ from ./s3sync.rb:346:in main' from ./thread_generator.rb:79:incall’ from ./thread_generator.rb:79:in initialize' from ./thread_generator.rb:76:innew’ from ./thread_generator.rb:76:in initialize' from ./s3sync.rb:267:innew’ from ./s3sync.rb:267:in ‘main’ from ./s3sync.rb:735

Posted on: July 25, 2008 at 5:38 pmQuote this Comment
Maxwell Scott-Slade - Gravatar

Maxwell Scott-Slade said:

I have set it up exact­ly the same as your blog (btw, thanks for all this) but I also get this error:

tar: Removing leading /' from member names tar: Removing leading/’ from mem­ber names Per­ma­nent redi­rect received. Try set­ting AWS_CALLING_FORMAT to SUBDOMAIN S3 ERROR: # ./s3sync.rb:290:in +': can't convert nil into Array (TypeError) from ./s3sync.rb:290:ins3TreeRecurse’ from ./s3sync.rb:346:in main' from ./thread_generator.rb:79:incall’ from ./thread_generator.rb:79:in initialize' from ./thread_generator.rb:76:innew’ from ./thread_generator.rb:76:in initialize' from ./s3sync.rb:267:innew’ from ./s3sync.rb:267:in ‘main’ from ./s3sync.rb:735

Posted on: July 27, 2008 at 6:59 amQuote this Comment
Ced - Gravatar

Ced said:

Maybe s3sync has been updat­ed since this post ?

Posted on: July 27, 2008 at 9:32 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

OK, so both Cedric and Maxwell are get­ting the same error. I looked up that error and it appears to be asso­ci­at­ed with EU buck­ets. Are either of you using buck­ets in the EU?

To change this, you need to add this line to your s3config.yml file:

AWS_CALLING_FORMAT: SUBDOMAIN

Posted on: July 27, 2008 at 9:38 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

Ced — no, I just dou­ble-checked. I think the issue is either with using an EU buck­et name or some­thing else in the buck­et-name not being cor­rect. EU Buck­et names can­not con­tain cap­i­tal let­ters (so they are not case-sen­si­tive), where­as US buck­et names can.

Make sure your buck­et name is cor­rect in the script. I think adding the AWS_CALLING_FORMAT para­me­ter to the yml file will solve the prob­lem.

Posted on: July 27, 2008 at 9:43 amQuote this Comment
Ced - Gravatar

Ced said:

@Christina : Yes it works now ! Thanks for your help.

Posted on: July 27, 2008 at 11:13 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

Ced, Glad to hear it! I’ve updat­ed the post with that infor­ma­tion in case any­one else runs into the same issue.

Posted on: July 27, 2008 at 11:59 amQuote this Comment
Maxwell Scott-Slade - Gravatar

Maxwell Scott-Slade said:

Thanks Christi­na, that’s all work­ing fine now. It’s so awe­some to know that the site is get­ting backed up every­day to a safe place 100%. It’s nice to turn on the email fea­ture in the Cron job sec­tion so you know it’s all done.

A guide to remem­ber. I nev­er used SSH before, now that I have I feel pret­ty hap­py it all works!

Posted on: July 28, 2008 at 4:07 amQuote this Comment

Faire un back­up de son blog sur Ama­zon S3 | 64k said:

[…] avec ce ser­vice, mais je n’avais pas vrai­ment pris le temps de tester. Je suis tombé sur un bil­let de Christi­na War­ren qui m’a décidé, puisqu’il décrit toute la procé­dure pour faire un back­up d’un […]

Eas­i­ly Back­up Your Entire Web­site to S3 | High­Ed­WebTech said:

[…] sent me a great link to a Ruby script that will back­up your web­site and push that back­up file to Ama­zon S3 for safe, […]

Philip - Gravatar

Philip said:

So does this back­up every­thing or just the con­tent that’s changed since the last back­up?

Posted on: August 24, 2008 at 6:45 pmQuote this Comment
Links of Inter­est — CSS-Tricks - Gravatar

Links of Inter­est — CSS-Tricks said:

[…] War­ren has a com­pre­hen­sive and excel­lent tuto­r­i­al on cre­at­ing a Ruby script to back up your entire web serv­er, includ­ing data­bas­es, and upload them […]

Posted on: August 26, 2008 at 8:46 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

Philip, I was hav­ing some prob­lems hav­ing effec­tive recur­sive back­ups, so it’s just doing every­thing. Your com­ment reminds me to re-inves­ti­gate the best/most effec­tive way to do it recur­sive­ly though (it would be like a few char­ac­ter changes to the script), so I’ll do that lat­er this week and post an update with my find­ings. Real­is­ti­cal­ly, your data­bas­es are going to be chang­ing more fre­quent­ly than your actu­al direc­to­ries, so you can always set a sep­a­rate CRON job to run the data­bas­es and cer­tain fold­ers every day, and oth­er fold­ers less fre­quent­ly. That’s what I do any­way — my blogs and data­bas­es are backed up dai­ly and a few domains that are just basi­cal­ly image stor­ing for right now get updat­ed once a week or once a month.

Posted on: August 26, 2008 at 9:40 amQuote this Comment
Matt - Gravatar

Matt said:

The back­up part works great for me, but not the s3sync. Cron job doesn’t even both­er with copy­ing any­thing over. When I do the site copy to S3 man­u­al­ly, it usu­al­ly dies after copy­ing just a few sites. Wish I could get that part work­ing as that is the impor­tant part!

Posted on: August 26, 2008 at 4:35 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Matt, Are you get­ting an error of any kind? About how much data is being copied over before it dies? I’ve suc­cess­ful­ly copied over more than 500 megabytes before using the script (a test case and also a back­up of some pho­tographs I uploaded to my gs tem­porar­i­ly when at my parent’s house). Let’s see if we can fig­ure out why it isn’t work­ing.

Posted on: August 26, 2008 at 6:29 pmQuote this Comment
Matthew Bark­er - Gravatar

Matthew Bark­er said:

Nope, not get­ting any errors that mat­ter it seems. Ear­li­er I report­ed this error:

Con­nec­tion reset: Con­nec­tion reset by peer 99 retries left, sleep­ing for 30 sec­onds 98 retries left, sleep­ing for 30 sec­onds …

But that doesn’t seem to real­ly mat­ter. I am back­ing up 10 sites, with only 1 being larg­er than 500 MB; the gzipped/tarred file is cur­rent­ly 1.3 GB in size. The odd thing about all of this is that some­times every­thing works when I do it man­u­al­ly, but that is only some­times. It gen­er­al­ly quits when trans­fer­ring the 1.3 GB file to Ama­zon, with no error mes­sages encoun­tered. But with the cron job run­ning, it some­times quits when tar­ring the 1.3 GB site, but gen­er­al­ly tars every­thing just fine, but doesn’t trans­fer a thing. That’s the hard part about try­ing to trou­bleshoot this prob­lem; some­times it works, some­times it doesn’t; and when it doesn’t work, it doesn’t die at the same place every time.

Posted on: August 27, 2008 at 12:15 amQuote this Comment
Peter - Gravatar

Peter said:

I was hop­ing I could down­load your full back­up script and cus­tomize it to my needs, but it looks like access it denied on the file from S3. Is there any chance you could make it pub­lic?

Posted on: August 27, 2008 at 10:31 amQuote this Comment
Mar­cus McCur­dy - Gravatar

Mar­cus McCur­dy said:

Thanks for this great write up. I went through it last night and it worked like a champ and I learned a thing or two in the process. This real­ly makes back­ing up a media tem­ple site a breeze.

Posted on: August 27, 2008 at 4:13 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Matt, I remem­ber you had that error. I’ll do what I can to inves­ti­gate why this seems to not be work­ing. It might be nec­es­sary to cre­ate two sep­a­rate Cron jobs — one for the biggest site, one of the oth­ers — to see if that is an accept­able workaround.

Peter — OK, try it now — I changed the URL struc­ture. It was work­ing fine, I’m not sure what could have changed. If you still have issues, let me know.

Mar­cus — yay! I’m so hap­py this worked for you!

Posted on: August 27, 2008 at 4:54 pmQuote this Comment

Auto­mat­i­cal­ly back up your entire web serv­er files and data­bas­es to Ama­zon S3 | Quick­Pipe said:

[…] War­ren has a com­pre­hen­sive and excel­lent tuto­r­i­al on cre­at­ing a Ruby script to back up your entire web serv­er, includ­ing data­bas­es, and upload them […]

Links of Inter­est | Pro­ba - Gravatar

Links of Inter­est | Pro­ba said:

[…] War­ren has a com­pre­hen­sive and excel­lent tuto­r­i­al on cre­at­ing a Ruby script to back up your entire web serv­er, includ­ing data­bas­es, and upload them […]

Posted on: August 31, 2008 at 6:15 pmQuote this Comment
Matt - Gravatar

Matt said:

Thanks Christi­na … this was real­ly help­ful. How about a Part 2 with rolling back­ups? 😉

Posted on: September 8, 2008 at 6:08 pmQuote this Comment

Auto­mat­i­cal­ly back up Meti­aTem­ple Grid (gs) via Ama­zon S3 | Ran­domess of Josh Price said:

[…] months now, and thought I’d share it with you guys.  Christi­na War­ren has writ­ten up some excel­lent instruc­tions on how to back up MediaTemple’s Grid (gs) to your Ama­zon S3 account.  There is also a video. […]

Michael York - Gravatar

Michael York said:

Hey Christi­na,

Not sure if you got my e-mail or not…

Any­ways, I want­ed to know why I keep get­ting “Con­nec­tion Reset: con­nec­tion reset by peer” error mes­sages. When look­ing at the S3 buck­et, not all of the files have trans­ferred over…

Thanks! Michael

Posted on: September 25, 2008 at 5:39 pmQuote this Comment
Pet­ter - Gravatar

Pet­ter said:

Matthew Bark­er said: Nope, not get­ting any errors that mat­ter it seems. Ear­li­er I report­ed this error: Con­nec­tion reset: Con­nec­tion reset by peer 99 retries …

I am expe­ri­enc­ing the same prob­lem. It is a fair­ly large site. About 2.5GB worth of pho­tos. I have split the back­up script up in two parts, the DB and the Site. The DB works fine and goes up to S3 eas­i­ly. Leads me to believe it is relat­ed to the size :). Have to divide it up in small­er chunks or sim­i­lar.

Mat, Christi­na, how did/would you approach it? any sug­ges­tions.

Posted on: September 29, 2008 at 10:34 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

Peter, Mat, and any­one else receiv­ing the con­nec­tion reset mes­sage: It IS size relat­ed. I’m work­ing on a new post that details chang­ing the num­ber of resets and does recur­sive back­ups for large files — so that you don’t trans­fer every­thing over every sin­gle time. I need to test it out and make sure it is sol­id before post­ing, but it should be up some­time this week!

Posted on: September 29, 2008 at 10:37 amQuote this Comment
Jor­dan - Gravatar

Jor­dan said:

Christi­na,

Thanks for the well writ­ten guide. I was able to get my back­ups run­ning per­fect­ly but did have one ques­tion.

I am using a slight­ly mod­i­fied ver­sion of your back­up script, and while it works well as far as cre­at­ing the back­up files, I don’t see any of the echo mes­sages in either the shell or in the cron emails. I only see the fol­low­ing:

tar: Removing leading /' from member names tar: Removing leading/’ from mem­ber names

Oth­er­wise, this seems to be the per­fect back­up sys­tem for Media Tem­ple gs users.

Posted on: September 30, 2008 at 4:44 pmQuote this Comment
William B. - Gravatar

William B. said:

Hey Christi­na,

This has real­ly helped my mis­sion, thanks for the guide. Works flaw­less­ly for me.

Posted on: October 5, 2008 at 10:59 amQuote this Comment
Poll Results: How do you back up your web­sites? — CSS-Tricks - Gravatar

Poll Results: How do you back up your web­sites? — CSS-Tricks said:

[…] this poll was to tap into your col­lec­tive knowl­edge and find some sweet auto­mat­ed back­up solu­tions. Christina’s solu­tion is absolute­ly awe­some and fits into my sit­u­a­tion per­fect­ly (being on Media Tem­ple, and hav­ing an S3 […]

Posted on: October 8, 2008 at 8:45 amQuote this Comment
Tim Pea­cock - Gravatar

Tim Pea­cock said:

Christi­na, real­ly appre­ci­ate the help with this. A cou­ple of errors, but got these ironed out, and now work­ing per­fect­ly. Saved me so much time !!

Posted on: October 13, 2008 at 6:25 amQuote this Comment
Ced - Gravatar

Ced said:

How to choose the data­base to save ? (and not all the data­bas­es) Because with more than 5–6 db, it uses too much ressources : “(mt) Media Temple’s auto­mat­ed MySQL mon­i­tor­ing sys­tems have detect­ed an increase in data­base activ­i­ty for your Grid-Ser­vice”

Posted on: October 15, 2008 at 3:22 amQuote this Comment
Matt - Gravatar

Matt said:

Any word on when this will be ready? I could def­i­nite­ly use it; my back­ups fail con­sis­tent­ly because of one site that is around 2GB in size.

Christi­na said: Peter, Mat, and any­one else receiv­ing the con­nec­tion reset mes­sage: It IS size relat­ed. I’m work­ing on a new post that …
Posted on: October 15, 2008 at 10:49 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Matt, It will be up by the end of the week. I had work com­mit­ments and then went on vaca­tion and this week has been crazy busy!

Posted on: October 15, 2008 at 10:57 pmQuote this Comment
Matt - Gravatar

Matt said:

Any luck on the size lim­i­ta­tion issue? Thanks!

Posted on: October 27, 2008 at 7:04 amQuote this Comment
Week in Review (Octo­ber 20 through Octo­ber 26) | Grand­mas­terB dot com - Gravatar

Week in Review (Octo­ber 20 through Octo­ber 26) | Grand­mas­terB dot com said:

[…] I also set­up an account on the Ama­zon AWS to remote­ly back­up my sites. After read­ing a tuto­r­i­al by Christi­na War­ren, I set­up the nec­es­sary soft­ware and a shell script to do auto­mat­ic back­ups of my site dai­ly. The […]

Posted on: October 27, 2008 at 4:45 pmQuote this Comment
creede - Gravatar

creede said:

I get a con­nec­tion reset too, but it seems to go through even­tu­al­ly. Let me know if you get rolling back­ups going. Thanks again for the great tool!

Posted on: November 12, 2008 at 6:30 pmQuote this Comment

Ama­zon S3 for Back­ups — Effec­tive Pro­gram­ming said:

[…] Christi­na War­ren […]

Mikkel - Gravatar

Mikkel said:

I get the same error as Matt when I test the script:

Con­nec­tion reset: Con­nec­tion reset by peer

99 retries left, sleeping for 30 seconds

How­ev­er when I look in my buck­et (using trans­mit), I see the files are turn­ing up just fine, so I guess I should just ignore the error. But it would be nice to have it run­ning smooth­ly.

An yes, I’m using a EU buck­et (sit­ting in Den­mark).

Anoth­er thing I’ve noticed; It backs up all of my datab­ses, but only the domain direc­to­ries I choose in the “backup_server.sh” file. Is there a way to exlude data­bas­es? I have a lot of test sites (with db’s), that arent realy that impor­tant to back­up.

Posted on: December 8, 2008 at 7:32 amQuote this Comment
Samm - Gravatar

Samm said:

First, Christi­na, thank you for this arti­cle — I am using mt (gs) and S3 and am hav­ing prob­lem. Prob­a­bly a mis­take on my part, but look­ing for some help. I have com­plet­ed the set­up, and when I run the com­mand ./backup_server.sh, I get the mes­sage: “bad inter­preter: No such file or direc­to­ry”. The file is clear­ly there, I can see it via FTP and it shows with the dir com­mand in Put­ty. Any ideas?

Posted on: December 10, 2008 at 12:26 pmQuote this Comment
Michael - Gravatar

Michael said:

Hey Christi­na,

Any news on the recur­sive script?

Thanks! –Michael

Posted on: December 10, 2008 at 2:08 pmQuote this Comment
trs21219 - Gravatar

trs21219 said:

i got this run­ning fine but i want the back­ups to go into sub­di­rec­to­ries in my s3 buck­et for each day that it backs up. i got it doing it for my serv­er but how do i make it make the fold­ers on S3?

im adding this to the end of the direc­to­ry path at the top and bot­tom of the backup.sh

directorypath........./date +%m-%d-%y/

Posted on: December 16, 2008 at 2:29 amQuote this Comment
trs21219 - Gravatar

trs21219 said:

trs21219 said: i got this run­ning fine but i want the back­ups to go into sub­di­rec­to­ries in my s3 buck­et for each …

nev­er­mind i got it ! here is my backup_server.sh file

!/bin/sh

A list of website directories to back up

websites=“domains.com domain2.com”

The destination directory to backup the files to

destdir=/home/xxxxx/users/.home/s3sync/s3backup/date +%m-%d-%y

The directory where all website domain directories reside

domaindir=/home/xxxx/users/.home/domains

The MySQL database hostname

dbhost=internal-db.sxxxxx.gridserver.com

The MySQL database username — requires read access to databases

dbuser=xxxxxx

The MySQL database password

dbpassword=xxxxxxx

mkdir $dest­dir rm $destdir/*.tar.gz

echo date “: Begin­ning back­up process…” > $destdir/backup.log

backup databases

for dbname in echo 'show databases;' | /usr/bin/mysql -h $dbhost -u$dbuser -p$dbpassword do if [ $dbname != “Data­base” ]; then echo date “: Back­ing up data­base $dbname…” » $destdir/backup.log /usr/bin/mysqldump –opt -h $dbhost -u$dbuser -p$dbpassword $dbname > $destdir/$dbname.sql tar -czf $destdir/$dbname.sql.tar.gz $destdir/$dbname.sql rm $destdir/$dbname.sql fi done

backup web content

echo date “: Back­ing up web con­tent…” » $destdir/backup.log for web­site in $web­sites do echo date “: Back­ing up web­site $web­site…” » $destdir/backup.log tar -czf $destdir/$website.tar.gz $domaindir/$website done

echo date “: Back­up process com­plete.” » $destdir/backup.log

echo date “: Com­pact­ing Pack­age Total…” » $destdir/backup.log tar -czf $destdir.tar.gz $dest­dir

rm -r -f $dest­dir

The directory where s3sync is installed

s3syncdir=/home/xxxxx/users/.home/s3sync

The directory where the backup archives are stored

backupdir=/home/xxxxxx/users/.home/s3sync/s3backup/

The S3 bucket a.k.a. directory to upload the backups into

s3bucket=BUCKET NAME HERE

cd $s3syncdir ./s3sync.rb $backupdir/ $s3bucket:

hope this helps some peo­ple… i got it from trail and error… thanks again christi­na for the awe­some script!

Posted on: December 16, 2008 at 3:06 amQuote this Comment
Adam - Gravatar

Adam said:

I thought I did every­thing right, but got this after run­ning the script man­u­al­ly:

com­mand not found line 4:

com­mand not found line 8:
com­mand not found line 10:
com­mand not found line 14:
com­mand not found line 16:
com­mand not found line 20:
com­mand not found line 22:
com­mand not found line 26:
com­mand not found line 28:
com­mand not found line 32:
com­mand not found line 34:
com­mand not found line 40: ./backup_server.sh: line 42: /backup.log: Read-only file sys­tem
com­mand not found line 44: rm: can­not remove /*.tar.gz': No such file or directory rm: cannot remove\r’: No such file or direc­to­ry
com­mand not found line 50: ‘/backup_server.sh: line 70: syn­tax error near unex­pect­ed token '/backup_server.sh: line 70:fi

What did I do wrong?

Posted on: January 7, 2009 at 4:45 pmQuote this Comment
Adam - Gravatar

Adam said:

Adam said: I thought I did every­thing right, but got this after run­ning the script man­u­al­ly: : com­mand not found line 4: : com­mand …

Nev­er­mind. My buck­et name had a typo. Sor­ry.

Does any­one know of a way to fil­ter out cer­tain direc­to­ries (for exam­ple: “cache” fold­ers)? I also have drop­box fold­ers that I don’t need to back­up, etc. I’d love to exclude them.

Posted on: January 8, 2009 at 5:18 pmQuote this Comment
A - Gravatar

A said:

Prop­er back­ups are like eat­ing your veg­eta­bles — we all say we’ll do it and that it is a good idea, but it is so much eas­i­er NOT to do it and eat Oreo cook­ies instead”

Its refresh­ing to see analo­gies like these..once in a while atleast..web would be so much green­er, warmer & health­i­er if even guy geeks start giv­ing such analogies..i wish..

Posted on: January 17, 2009 at 5:31 amQuote this Comment
Adam - Gravatar

Adam said:

Adam said: Does any­one know of a way to fil­ter out cer­tain direc­to­ries (for exam­ple: “cache” fold­ers)? I also have drop­box fold­ers that I don’t need to back­up, etc. I’d love to exclude them.

In case any­one else is inter­est­ed, I made two changes to the script that I’m real­ly hap­py with. I changed the com­pres­sion type to bzip2 (bz2) instead of gzip (gz), and my file sizes shrunk dra­mat­i­cal­ly! Addi­tion­al­ly, I exclud­ed some files and direc­to­ries that I didn’t need backed up and knew were full of tons on files. This saves a lot of time and end file size. Here’s the main change, but a cou­ple oth­ers need to be made for this to ful­ly work:

tar -jcf $destdir/$website.tar.bz2 –exclude ‘/cache’ –exclude ‘/dropbox’ –exclude ‘*.zip’ $domaindir/$website

Posted on: January 17, 2009 at 1:09 pmQuote this Comment
John­ny - Gravatar

John­ny said:

hi!

Is there any way to back­up the e-mails as well from the grid-serv­er?

(Thanks for the tuto­r­i­al, it’s work­ing fine on my side.)

Posted on: January 18, 2009 at 7:59 amQuote this Comment
jota - Gravatar

jota said:

Hel­lo,

It’s been a few weeks since the script stopped work­ing for me.

Now I’m get­ting this error

Read from remote host domain.com: Con­nec­tion reset by peer Con­nec­tion to domain.com closed.

Any clue what might be hap­pen­ing?

Posted on: June 6, 2009 at 8:25 amQuote this Comment
Matt - Gravatar

Matt said:

Hel­lo Christi­na,

Excel­lent script! It seems to work fine for me all the files and dbs get backed up but I get this error for some rea­son:

/usr/bin/mysqldump: Got error: 1044: Access denied for user ‘db62917’@’%’ to data­base ‘information_schema’ when using LOCK TABLES

Not quite sure why this is, any ideas?

Matt

Posted on: June 23, 2009 at 4:38 amQuote this Comment
Pietro - Gravatar

Pietro said:

Hi Christi­na, thanks a lot! Your tuto­r­i­al is fan­tas­tic!

I’m get­ting the same error as Matt, though: /usr/bin/mysqldump: Got error: 1044: Access denied for user ‘dbxxxxx’@’%’ to data­base ‘information_schema’ when using LOCK TABLES

Any­way, the script is work­ing…

Pietro

Posted on: July 20, 2009 at 4:21 amQuote this Comment
Karl Gech­lik - Gravatar

Karl Gech­lik said:

I am get­ting errors, can you help me out? Any help would be great­ly appre­ci­at­ed.

tar: Remov­ing lead­ing /' from member names S3 command failed: list_bucket prefix max-keys 200 delimiter / With result 404 Not Found S3 ERROR: # ./s3sync.rb:290:in+’: can’t con­vert nil into Array (Type­Er­ror) from ./s3sync.rb:290:in s3TreeRecurse' from ./s3sync.rb:346:inmain’ from ./thread_generator.rb:79:in call' from ./thread_generator.rb:79:inini­tial­ize’ from ./thread_generator.rb:76:in new' from ./thread_generator.rb:76:inini­tial­ize’ from ./s3sync.rb:267:in new' from ./s3sync.rb:267:inmain’ from ./s3sync.rb:735

Posted on: August 4, 2009 at 10:20 amQuote this Comment
Christi­na - Gravatar

Christi­na said:

Hey guys, OK — I’m upload­ing new copies of the screen­cast now and will have those links switched in about 15 min­utes. Karl, give me a lit­tle bit to look over the code, I need to do a fol­low-up post (13 months lat­er, wow!) any­way 🙂

Posted on: August 4, 2009 at 10:25 amQuote this Comment
Karl Gech­lik - Gravatar

Karl Gech­lik said:

Thank you so much Christi­na I am sit­ting on the edge of my seat pulling my hair out for a few days now (more like a week!) I can gen­er­ate the back­up file but I have to man­u­al­ly xfer it to s3. I am won­der­ing if it is my buck­et name it has a — in it. Any assis­tance is REALLY REALLY appre­ci­at­ed! You rock.

Posted on: August 4, 2009 at 10:34 amQuote this Comment
Mat­tO - Gravatar

Mat­tO said:

SOME HELP FOR NEWBIES ON HOW TO DO THIS AND SOME PROBLEMS I RAN INTO:

Hi Christi­na,

You Rock! This is a huge help — you have no idea how hard this stuff can be to get done when your not a pro.

I just want to doc­u­ment a few prob­lems that I encoun­tered when try­ing this so oth­ers can hope­ful­ly skip my frus­tra­tions on Sat­ur­day Morn­ing : )

1) If you run into the prob­lem of not being able to see the s3sync direc­to­ry that you cre­ate.

  • In Christina’s video it shows up as a direc­to­ry next to your domain fold­er, how­ev­er, in my expe­ri­ence it wasn’t any­where to be found. I knew it exist­ed because I could see it in SSH, but in FTP and my Account Cen­ter it was invis­i­ble.

  • To rec­ti­fy this you can go into your FTP set­ting and show hid­den files. In filezil­la this is under the Serv­er Menu and is called “Force show­ing hid­den files”

-Then, your not done yet — to actu­al­ly see the s3sync direc­to­ry — go into your users direc­to­ry and you should now see a direc­to­ry called “.home” — you’ll find it in there.

2) Put your server_backup.sh script in the data direc­to­ry — this is the main data direc­to­ry in the same hier­ar­chi­cal lev­el as your domains fold­er and etc fold­er

3) When you fig­ure all this out and you actu­al­ly run the script you’ll get some errors — the first ones I hit were relat­ed to buck­ets in AWS.

-If your new at all this you need to cre­ate a buck­et, but this can be kin­da con­fus­ing.

-I final­ly found this sim­ple way to do it: http://it.toolbox.com/blogs/oracle-guide/cloud-studio-for-aws-ec2-and-s3-28117

-Essen­tial­ly you down­load the cloud stu­dio free­ware (FYI — I’m on MAC), open it, go into the set­tings and enter your account info, don’t wor­ry about all the RED text (that has to do with anoth­er amaza­on image tool), go down toward the bot­tom of the screen and add a new buck­et. That should do the trick — now go to the server_backup.sh script and put the name of your buck­et there like she tells you to do.

4) You might still be get­ting an error that has this in it: “s3sync/s3backup/*.tar.gz’: No such file or direc­to­ry”

-This is because the way that the ama­zon direc­to­ries are down­loaded is a bit dif­fer­ent than the script is writ­ten (I think).

-To solve this sim­ply go into your FTP and sim­ply move the S3backup direc­to­ry inside of the s3sync direc­to­ry where it is look­ing for it.

OK! I know it’s the clear­est explaina­tion, but I hope it saves some­one a few hours!

Thanks again for the detailed tuto­r­i­al!

-Mat­tO

http://mattoden.com

Posted on: August 22, 2009 at 3:25 pmQuote this Comment
Mat­tO - Gravatar

Mat­tO said:

Hi Christi­na,

Quick Ques­tion:

Is there a way to mod­i­fy the script so that you don’t over­write the back­ups on S3?

For Exam­ple — I would like to run a dai­ly back­up and on S3 I would like the html direc­to­ries and the sql data­bas­es to append the date to each ver­sion. Day one: Example.com_8_25_09.tar.gz, Example.com_8_26_09.tar.gz, etc.

That way, if there is a prob­lem, I can roll back to the last ful­ly func­tion­ing date.

Thanks,

Mat­tO

Posted on: August 22, 2009 at 3:35 pmQuote this Comment
Christi­na - Gravatar

Christi­na said:

Matt, Amaz­ing, amaz­ing — I’ll be updat­ing the post this week (I’ve been say­ing that for a year but I actu­al­ly mean it this time) and will be using your cor­rec­tions — the for­mat­ting has changed a bit in the last year and I need to update the script.

I’ll also include a way for you to write a new file each time (with a date append­ed). I’m sure we can fig­ure out a way to have a rule to only keep the last X or what­ev­er too.

Posted on: August 23, 2009 at 2:03 pmQuote this Comment
Coa­lesce - Gravatar

Coa­lesce said:

Thanks for the help­ful tuto­r­i­al. We’re going to be await­ing your revised script!

Posted on: August 27, 2009 at 12:34 pmQuote this Comment
Andrew - Gravatar

Andrew said:

Just imple­ment­ed this on my gs serv­er, thanks for the great script. Ear­gly antic­i­pat­ing the revised script with the dat­ed back­ups.

Posted on: August 28, 2009 at 11:24 pmQuote this Comment
zxr - Gravatar

zxr said:

Hi guys!

I still have a prob­lem when work­ing with a big back­up file. One of my sites is around 1gb of data and this method does not work for It. I run the script but It says comthing like the time has expired…

Any solu­tiono for this?

Thanks!

Posted on: August 29, 2009 at 12:33 pmQuote this Comment
Mikkel Hansen - Gravatar

Mikkel Hansen said:

Any news on the update? Would realy love to be able to keep the last x days of back­ups on the serv­er!

Posted on: September 8, 2009 at 5:50 pmQuote this Comment
Andrew - Gravatar

Andrew said:

I was won­der­ing if you had an ETA for a way to keep mul­ti­ple dat­ed back­ups, I need­ed that today. Thanks for the script though, it was a life­saver.

Posted on: September 15, 2009 at 12:06 amQuote this Comment
Brett Wilcox - Gravatar

Brett Wilcox said:

Hel­lo Christi­na!

I just want­ed to share with you and every­one here that I have cre­at­ed a VERY robust back­up script that keeps mul­ti­ple rotat­ing back­ups and such.

I will be releas­ing more infor­ma­tion soon but I have post­ed the code at http://www.brettwilcox.com/2009/09/20/amazon-s3-backup-script/ for any­one inter­est­ed.

Shoot me an email if any­one has any ques­tions!

[email protected]

Posted on: September 21, 2009 at 1:21 amQuote this Comment
Antho­ny Abraira - Gravatar

Antho­ny Abraira said:

This is prob­a­bly a stu­pid ques­tion, but is there a means of doing some­thing where this can instead zip up to one file that goes to a des­ig­nat­ed fold­er on the site. That way one can just go in and down­load the zip file and have their back up with­out need­ing an addi­tion­al serv­er stor­age space.

If this has been answered I am sor­ry, can some­body point me in the right direc­tion?

Posted on: September 21, 2009 at 2:49 pmQuote this Comment
Brett Wilcox - Gravatar

Brett Wilcox said:

Hel­lo Antho­ny,

I have cre­at­ed that very script that you are talk­ing about.

http://www.brettwilcox.com/2009/09/20/amazon-s3-backup-script/

Just dis­able the ama­zon S3 Sync option and it will cre­ate mul­ti­ple rotat­ing dat­ed back­ups with­out the need for ama­zon S3.

Posted on: September 21, 2009 at 2:55 pmQuote this Comment
Brett Wilcox - Gravatar

Brett Wilcox said:

I have released a new­er ver­sion of the script and pub­lished it to google code — http://code.google.com/p/ziplinebackup/

Posted on: September 30, 2009 at 5:09 pmQuote this Comment
Hugh Esco - Gravatar

Hugh Esco said:

A cou­ple of thoughts I have not seen addressed above:

I would be care­ful about delet­ing the pre­vi­ous back­up pri­or to cre­at­ing the new one.

If mem­o­ry serves, an S3 buck­et is lim­it­ed to a giga­byte or so. This may be the issue folks are encoun­ter­ing back­ing up larg­er sites.

Essen­tial touch to time stamp back­ups and keep the last sev­er­al around. I once had to roll back sev­er­al days to find an uncor­rupt­ed data­base back­up. That was when I gave up email (for scp and rsync) as a trans­port for files which mat­tered.

if you can add some sort of log rotate func­tion to this script you’ll have a full fea­tured back­up sys­tem in place.

Back­ing up to S3 is a fine way to store pub­lic data acces­si­ble on the web any­way, but for folks deal­ing with pro­pri­etary data, inhouse file stores, or at least encrypt­ed S3 buck­ets are prob­a­bly a more appro­pri­ate way to go.

And one last impor­tant step, as you add new sites and new resources which ought to be backed up, it impor­tant to update your back­up script to account for those.

Thanks for the arti­cle. Was unaware of this ruby tool.

– Hugh

Posted on: October 20, 2009 at 1:27 amQuote this Comment
irms - Gravatar

irms said:

This was awe­some. Thanks!

Posted on: December 14, 2009 at 8:16 amQuote this Comment
Clay­ton - Gravatar

Clay­ton said:

Thanks for the tuto­r­i­al, Christi­na. This post, along with your video, were a huge help for me.

Posted on: January 5, 2010 at 8:06 pmQuote this Comment
Clay­ton - Gravatar

Clay­ton said:

This appears to be work­ing, but I am get­ting an error mes­sage at the top of my Cron job noti­fi­ca­tion email: “Got error: 1044: Access denied for user ‘dbXXXXX’@’%’ to data­base ‘information_schema’ when using LOCK TABLES

Any­one else get this?

Posted on: January 8, 2010 at 10:07 amQuote this Comment
Christi­na War­ren - Gravatar

Christi­na War­ren said:

Clay­ton, Yes — a user was nice enough to put up a new script on github — the link is in the com­ments (a few posts before yours) — I des­per­ate­ly need to update this blog post with that info — thanks for the reminder!

Posted on: January 8, 2010 at 10:09 amQuote this Comment
Karl L. Gech­lik | AskTheAdmin.com - Gravatar

Karl L. Gech­lik | AskTheAdmin.com said:

Thanks for your help. I rewrote the script to cre­ate cus­tom file­names with the date in them. I then croned anoth­er script to remove files after 7 days keep­ing 7 back­ups. I could not get your MySql part to work so I cronned out MySql dumps and put them in the tar. Final­ly I reduced the file size with com­pres­sion and fil­ter­ing out spe­cif­ic file types I did­nt need backed up. I will get it doc­u­ment­ed and up on http://www.askTheAdmin.com as soon as I can.

Posted on: January 8, 2010 at 10:14 amQuote this Comment
Jesse - Gravatar

Jesse said:

Thank you so much for spelling all this out for us! This was extreme­ly help­ful! I am using Ama­zon S3 now, it’s kind of tricky to use but this helped me save much time 🙂

Posted on: February 24, 2010 at 5:12 pmQuote this Comment

Media Tem­ple back­ups naar Ama­zon S3 met S3Sync said:

[…] GB per maand + upload en down­load fees. Er is geen ver­plichte afname van het aan­tal GBs. Kudos voor Christi­na War­ren en haar blog­post die de basis vormt voor deze Ned­er­landse ver­tal­ing en kudos voor Brett Wilcox, […]

Andre Kreft - Gravatar

Andre Kreft said:

Hi Christi­na, love the script and has been work­ing for a while now. Late­ly I got some strange error’s I cant seem to fix.

Like: No such file or direc­to­ry in my Cron Dea­mon emails and when I try to kick off my cron job through SSH I can’t kick off ./server_backup.sh in my data fold­er. It keep say­ing: -bash: /backup_server.sh: No such file or direc­to­ry

Any­thing changed on MT side? or am I miss­ing some­thing. Hope you can help.

Posted on: March 17, 2010 at 4:41 amQuote this Comment
the Blog But­ler - Gravatar

the Blog But­ler said:

Thanks for writ­ing this up Cristi­na, was a great help and got me off to a good start. I have every­thing work­ing fine when I run it from an SSH ses­sion but when the cron kicks the job off it will back up the site but will NOT trans­fer it to my S3 buck­et. It does it just fine when I run it man­u­al­ly but not from cron.

Any­one have any clue on this?

Posted on: June 29, 2010 at 6:40 pmQuote this Comment
Tan­ner Hobin - Gravatar

Tan­ner Hobin said:

Thank you, thank you, thank you!

So I tried to make things a lit­tle eas­i­er on myself by mak­ing the “list of web­site direc­to­ries to back­up” /domains and “the direc­to­ry where all web­site domain direc­to­ries reside” /home/XXXXX/users/.home and the result­ing .tar.gz file only had fold­ers in it for home>XXXXX>users. No domains backed-up.

Sim­ply put, I was hop­ing to skip hav­ing to list all domains and the need to update the script every time I add/remove a domain. Any idea how I might be able to do that?

Again, thank you.

Posted on: July 16, 2010 at 5:47 pmQuote this Comment
Brett - Gravatar

Brett said:

Any idea on how to cor­rect these errors in the zipline scripte?

======================================================================

Database Backup Start Time Fri Oct 8 01:33:30 PDT 2010

Back­ing up data­bas­es from /home/86415/data/database_backups tar: Remov­ing lead­ing ‘/’ from mem­ber names tar: /home/86415/data/database_backups: Can­not stat: No such file or direc­to­ry tar: Error exit delayed from pre­vi­ous errors Com­plete

======================================================================

Database Backup End Time Fri Oct 8 01:33:30 PDT 2010

======================================================================

Starting Transfer to Online Storage Backup Fri Oct 8 01:33:30 PDT 2010

Now trans­fer­ing Back­ups to Ama­zon S3 S3 com­mand failed: list_bucket max-keys 200 pre­fix delim­iter / With result 404 Not Found S3 ERROR: # ./s3sync.rb:290:in +': can't convert nil into Array (TypeError) from ./s3sync.rb:290:ins3TreeRecurse’ from ./s3sync.rb:346:in main' from ./thread_generator.rb:79:incall’ from ./thread_generator.rb:79:in initialize' from ./thread_generator.rb:76:innew’ from ./thread_generator.rb:76:in initialize' from ./s3sync.rb:267:innew’ from ./s3sync.rb:267:in ‘main’ from ./s3sync.rb:735

Com­plete

Posted on: October 8, 2010 at 4:36 amQuote this Comment
Brett - Gravatar

Brett said:

Oh crap this is the wrong blog to ask that ques­tion, I should real­ly not mess with this sort of thing this late at night. ><

Posted on: October 8, 2010 at 4:38 amQuote this Comment
Bar­ton - Gravatar

Bar­ton said:

I also get the error:

Access denied for user ‘dbxxxxx_s3sync’@’%’ to data­base ‘yyyyy’ when using LOCK TABLES

Posted on: October 18, 2010 at 2:25 pmQuote this Comment
Bar­ton - Gravatar

Bar­ton said:

Okay, to all peo­ple with the “Access denied” … “using LOCK TABLES” error:

The back­up is work­ing, its just fail­ing when it tries to back­up the information_schema, which isn’t required.

Posted on: October 18, 2010 at 2:30 pmQuote this Comment
nosaukums - Gravatar

nosaukums said:

Some ideas, I defined in the begin­ning of back­up script: DATESTAMP=date +%Y-%m-%d

and mod­i­fied to include it in all file­names i.e.:

back­up data­bas­es

for dbname in echo 'show databases;' | /usr/bin/mysql -h $dbhost -u$dbuser -p$dbpassword do if [ $dbname != “Data­base” ]; then echo date “: Back­ing up data­base $dbname…” » $destdir/backup.log /usr/bin/mysqldump -h $dbhost –opt –skip-lock-tables -u$dbuser -p$dbpassword $dbname > $destdir/$DATESTAMP.$dbname.sql tar -czf $destdir/$DATESTAMP.$dbname.sql.tar.gz $destdir/$DATESTAMP.$dbname.sql rm $destdir/$DATESTAMP.$dbname.sql fi done

back­up web con­tent

echo date “: Back­ing up web con­tent…” » $destdir/$DATESTAMP.backup.log for web­site in $web­sites do echo date “: Back­ing up web­site $web­site…” » $destdir/backup.log tar -czf $destdir/$DATESTAMP.$website.tar.gz $domaindir/$website done

also use­ful thing — exclude a spe­cif­ic direc­to­ry (in my case cash, save some pre­cious band­width 😉 ) tar -czf $destdir/$DATESTAMP.$website.tar.gz $domaindir/$website --exclude 'notthis/cache/*'

also could­nt fig­ure out data­base lock­ing issue so i added “–skip-lock-tables” to com­mand. so far so good.

thanks so much fur such a great tuto­r­i­al.

Posted on: May 9, 2011 at 4:15 amQuote this Comment
nosaukums - Gravatar

nosaukums said:

ouch, for­mat­ting screwed up.

any­way.. just add “$DATESTAMP.” to all “$dbname.sql.tar.gz” and to first “$website.tar.gz”

tadaa..

and not test­ed yet, but I replaced the orig­i­nal “rm $destdir/*.tar.gz” (remove all tar.gz files) with “find /$destdir -type f -mtime +10 -exec rm {} \;” (delete all files old­er then 10 days.)

Posted on: May 9, 2011 at 4:20 amQuote this Comment
hum­bert - Gravatar

hum­bert said:

mmm­mm cook­ies.…..!

Posted on: October 26, 2011 at 10:56 pmQuote this Comment
Rosamun­da - Gravatar

Rosamun­da said:

Thanks Christi­na! I was so help­less about what to do to back­up remote­ly my sites in my brand new MT account!! THANK YOU VERY MUCH!!! Gra­cias! Rosamun­da from Buenos Aires, Argenti­na

Posted on: August 19, 2012 at 1:04 pmQuote this Comment
Dave F - Gravatar

Dave F said:

Thanks so much for post­ing this. It was EXACTLY what I was look­ing for. Per­fect­ly writ­ten and com­ment­ed. THANK YOU!. I did run into one snag. I keep get­ting this error when try­ing to back­up my domain archive. Broken pipe: Broken pipe 99 retries left, sleeping for 30 seconds Broken pipe: Broken pipe 98 retries left, sleeping for 30 seconds

The sql archive uploaded to s3 per­fect­ly but it fails with this file for some rea­son. Any idea why? My site is VERY image depen­dant (for a mod­el­ing agency). The site archive is about 10GB. It cre­at­ed the archive but its fail­ing to upload it from my MT Grid Serv­er.

Any help is great­ly appre­ci­at­ed.

Thanks,

Dave

Posted on: May 6, 2013 at 1:10 amQuote this Comment
Dave F - Gravatar

Dave F said:

Just a fol­low up. I split the very large tar file into 950MB parts and it seems to solve the Bro­ken Pipe prob­lem. Hope this helps some­one.

Posted on: May 6, 2013 at 10:04 amQuote this Comment