christinawarren.com

For news tips, pitches, speaking engagements or media inquiries: Contact

S3 to backup Media Temple’s Grid (gs)">Using Amazon S3 to backup Media Temple’s Grid (gs)

Proper backups are like eating your vegetables -- we all say we'll do it and that it is a good idea, but it is so much easier NOT to do it and eat Oreo cookies instead. Don't risk losing your website because you didn't bother backing up.

Proper back­ups are like eat­ing your veg­eta­bles — we all say we’ll do it and that it is a good idea, but it is so much eas­ier NOT to do it and eat Oreo cook­ies instead. Then you wake up one day, are 25 years old and are a really picky eater and annoy your boyfriend because you won’t go eat at the Indian place he loves that doesn’t have a menu but only serves veg­e­tar­ian stuff that scares you. And the peo­ple at Sub­way give you dirty looks when you tell them you don’t want any­thing on your sand­wich. Don’t risk los­ing your web­site because you didn’t bother back­ing up.

Update: I posted a video tuto­r­ial that walks through all of these steps here. I still rec­om­mend read­ing through this page because the video tuto­r­ial assumes that you will be fol­low­ing these steps.

This a tuto­r­ial for cre­at­ing an auto­mated back-up sys­tem for (mt) Media Temple’s (gs) Grid Ser­vice. Although it will almost cer­tainly work on other servers and con­fig­u­ra­tions, this is writ­ten for users who are on the Grid who want an easy way to do auto­mated back­ups. I per­son­ally feel most com­fort­able hav­ing my most impor­tant files backed-up off­site, so I use Amazon’s S3 ser­vice. S3 is fast, super cheap (you only pay for what you use) and reli­able. I use S3 to store my web­site back­ups and my most impor­tant com­puter files. I spend about $1.50 a month, and that is for nearly 10 GBs of storage.

You can alter the script to sim­ply store the data in a sep­a­rate loca­tion on your server (where you can then just FTP or SSH in and down­load the com­pressed archive), but this process is assum­ing that you are using both the (gs) and S3.

This tuto­r­ial assumes that you know how to login to your (gs) via SSH using either the Ter­mi­nal in OS X or Linux or PuTTY for Win­dows. If SSH is still con­fus­ing, check out (mt)‘s Knowl­edge Base arti­cle and take a deep breath. It looks more scary than it really is.

Acknowl­edge­ments

I would be remiss if I didn’t give a GIGANTIC shout-out to David at Stress Free Zone and Paul Sta­ma­tiou (I met Paul at the Tweet-up in March) who both wrote great guides to back­ing stuff up server side to S3. I bla­tantly stole from both of them and rolled my own script that is a com­bi­na­tion of the two. Seri­ously, thank you both for your awe­some articles.

Fur­ther­more, none of this would even be pos­si­ble with­out the bril­liant S3Sync Ruby util­ity.

Installing S3Sync

Although PHP and Perl script exist to con­nect with the S3 servers, the Ruby solu­tion that the S3Sync dudes cre­ated is much, much better.

The (gs) already has Ruby on it (ver­sion 1.8.5 as of this writ­ing), which is up-to-date enough for S3Sync.

OK, so log-in to your (gs) via SSH. My set­tings (and the defaults for (gs), I assume) are to place you in the .home direc­tory as soon as you login to SSH.

Once you are at the com­mand line, type in the fol­low­ing command:

wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz

This will down­load the lat­est S3Sync tar­ball to your .home folder

tar xvzf s3sync.tar.gz

This uncom­presses the archive to its own directory.

rm s3sync.tar.gz</p>

<p>cd s3sync</p>

<p>mkdir certs</p>

<p>cd certs</p>

<p>wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar</p>

<p>sh ssl.certs.shar</p>

<p>cd ..</p>

<p>mkdir s3backup

That will delete the com­pressed archive, make a direc­tory for cer­tifi­cates (certs), down­load an SSL cer­tifi­cate gen­er­a­tor script, exe­cute that script and cre­ate a backup direc­tory within the s3sync direc­tory called “s3backup.”

Now, all you need to do is edit two files in your newly cre­ated s3sync folder. You can use TextE­dit, Text­Mate, NotePad or any other text edi­tor to edit these files. You are only going to be chang­ing a few of the values.

I edited the files via Trans­mit, but you can use vi straight from the com­mand line if you are comfortable.

The first file you want to edit is called s3config.yml.sample

You want to edit that file so that the aws_access_key and aws_secret_access_key fields cor­re­spond to those from your S3 account. You can find those in the Access Infor­ma­tion area after log­ging into Amazon.com’s Web Ser­vices page.

Make sure that the ssl_cert_dir: has the fol­low­ing value (if you cre­ated your s3sync folder in the .home direc­tory): /home/xxxxx/users/.home/s3sync/certs were xxxxx is the name of your server.

You can get your entire access path by typ­ing in

pwd
at the com­mand line.

Save that file as s3config.yml

The next step is some­thing I had to do in order to get the s3 part of the script to con­nect, but it may not be required for all server set-ups, but it was for the (gs).

Edit the s3config.rb file so that the area that says

confpath = [xxxxx]

looks like this

confpath = ["./", "#{ENV['S3CONF']}", "#{ENV['HOME']}/.s3conf", "/etc/s3conf"]

Writ­ing the backup script (or edit­ing mine)

OK, that was the hard part. The rest is pretty simple.

I cre­ated the fol­low­ing backup script called, “backup_server.sh” This script will backup the con­tent of the domain direc­to­ries you spec­ify (because if you are like me, some of your domain fold­ers are really just sym­links) and all of your MySQL data­bases. It will then upload each direc­tory and data­base in its own com­pressed archive to the S3 Bucket of your choice. Buck­ets are unique, so cre­ate a Bucket using either the S3Fox tool or Trans­mit or another S3 man­ager that is spe­cific for your website.

This is the con­tent of the script:

#!/bin/sh</p>

<h1>A list of website directories to back up</h1>

<p>websites="site1.com site2.com site3.com"</p>

<h1>The destination directory to backup the files to</h1>

<p>destdir=/home/xxxxx/users/.home/s3sync/s3backup</p>

<h1>The directory where all website domain directories reside</h1>

<p>domaindir=/home/xxxxx/users/.home/domains</p>

<h1>The MySQL database hostname</h1>

<p>dbhost=internal-db.sxxxxx.gridserver.com</p>

<h1>The MySQL database username - requires read access to databases</h1>

<p>dbuser=dbxxxxx</p>

<h1>The MySQL database password</h1>

<p>dbpassword=xxxxxxx</p>

<p>echo <code>date</code> ": Beginning backup process..." > $destdir/backup.log</p>

<h1>remove old backups</h1>

<p>rm $destdir/*.tar.gz</p>

<h1>backup databases</h1>

<p>for dbname in <code>echo 'show databases;' | /usr/bin/mysql -h $dbhost -u$dbuser -p$dbpassword</code>
do
if [ $dbname != "Database" ];
then
echo <code>date</code> ": Backing up database $dbname..." >> $destdir/backup.log
/usr/bin/mysqldump --opt -h $dbhost -u$dbuser -p$dbpassword $dbname > $destdir/$dbname.sql
tar -czf $destdir/$dbname.sql.tar.gz $destdir/$dbname.sql
rm $destdir/$dbname.sql
fi
done</p>

<h1>backup web content</h1>

<p>echo <code>date</code> ": Backing up web content..." >> $destdir/backup.log
for website in $websites
do
echo <code>date</code> ": Backing up website $website..." >> $destdir/backup.log
tar -czf $destdir/$website.tar.gz $domaindir/$website
done</p>

<p>echo <code>date</code> ": Backup process complete." >> $destdir/backup.log</p>

<h1>The directory where s3sync is installed</h1>

<p>s3syncdir=/home/xxxxx/users/.home/s3sync</p>

<h1>The directory where the backup archives are stored</h1>

<p>backupdir=/home/xxxxx/users/.home/s3sync/s3backup</p>

<h1>The S3 bucket a.k.a. directory to upload the backups into</h1>

<p>s3bucket=BUCKET-NAME</p>

<p>cd $s3syncdir
./s3sync.rb $backupdir/ $s3bucket:

For (mt) Media Tem­ple (gs) Grid Server users, you just need to change the “site1.com” val­ues to your own domains (you can do as many as you want) and sub­sti­tute all the places where marked “xxxxx” with your server num­ber (again, you can find this by enter­ing “pwd” at the com­mand line) and with your data­base pass­word (which is vis­i­ble in the (mt) con­trol panel under the “Data­base” module.

Make sure you change the value at the end of the script that says “BUCKET-NAME” to the name of the S3 Bucket you want to store you back­ups in.

Now that you have edited the script, upload it to your /data directory.

Change the per­mis­sions (you can do this either via SSH

chmod a+x backup_server.sh
or using your FTP client to 755.

Now, test the script.

In the com­mand line type this in:

cd data</p>

<p>./backup_server.sh</p>

<p>

 

And watch the magic. Assum­ing every­thing was cor­rectly input, an archived ver­sion of all your domain direc­to­ries and all of your MySQL data­bases will be put in a folder called “s3backup” and then uploaded directly to your S3 server. Next time you run the script, the backup files will be replaced.

Check to make sure that the script is work­ing the way you want it to work.

Auto­mate the script

You can either run the script man­u­ally from the com­mand line or set it to run auto­mat­i­cally. I’ve set mine to run each night at mid­night. To set up the cron job, just click on the Cron Jobs but­ton in the (mt) Admin area:

 

(mt) AccountCenter - filmgirl.tv : (gs) GridControls Uploaded with plasq’s Skitch!

 

 

and set you para­me­ters. The path for your script is: /home/xxxxx/data/backup_server.sh.

Enjoy your backups!

One note: The com­pressed domain archives retain their entire direc­tory struc­ture, as such, there is a .home direc­tory that may not appear in Finder or Win­dows Explorer unless you have invis­i­ble or hid­den files turned on. Don’t worry, all your data is still retained in those archives.

Update (7/27/2008): If you are get­ting an error that says some­thing like Permanent redirect received. Try setting AWS_CALLING_FORMAT to SUBDOMAIN

Add the fol­low­ing array to your s3config.yml file AWS_CALLING_FORMAT: SUBDOMAIN

The error is either because your bucket is in the EU or there is some­thing else funky with its URL struc­ture. Chang­ing that value should allow the script to per­form as intended.

Feeds-

Alltop, all the cool kids (and me)

TwitterCounter for @film_girl

Instagram

Calendar-

June 2008
S M T W T F S
« May   Jul »
1234567
891011121314
15161718192021
22232425262728
2930  

112 people have left comments

Paul Sta­ma­tiou - Gravatar

Paul Sta­ma­tiou said:

S3 is great huh? That s3config.rb path thing must have been a change with the recent s3sync as I had to do the same thing when I just upgraded.. didn’t have to do that when I had pub­lished it. Any­ways, thanks for bring­ing that to my atten­tion, I’ve updated my article.

Posted on: June 24, 2008 at 6:39 pmQuote this Comment
Christina - Gravatar

Christina said:

Thanks for your orig­i­nal tuto­r­ial and arti­cle Paul — it helped me out tremen­dously when I was set­ting the whole thing up. And yes, S3 is the bees knees as they say.

Posted on: June 24, 2008 at 6:43 pmQuote this Comment
George Ornbo - Gravatar

George Ornbo said:

Great write up. The Ama­zon S3 ser­vice is per­fect for back­ing up a web server remotely, safely and cheaply. I’m also using it to do off site back­ups for machines inside the net­work. For a small busi­ness it is a great solution.

Posted on: June 25, 2008 at 3:00 amQuote this Comment
Mike Mar­ley - Gravatar

Mike Mar­ley said:

Neat arti­cle Christina.

/me adds it to list of KB arti­cles that need to be written.

Mike Mar­ley Senior Tech­ni­cal Sup­port (mt) Media Tem­ple, Inc.

Posted on: June 25, 2008 at 10:47 amQuote this Comment
Ross - Gravatar

Ross said:

Nice post.…very very detailed. Hope­fully you put it in the how to part of the MT forum?

Posted on: July 1, 2008 at 10:41 amQuote this Comment
Michael - Gravatar

Michael said:

Excel­lent resource, thank you. One note on my expe­ri­ence when con­fig­ur­ing on the gs:

I was receiv­ing errors like “-o is not a valid vari­able” or some­thing like this, when the script was try­ing to exe­cute the mysql dump. I changed it to –opt (vs. –opt in your script).

Thanks again!

Posted on: July 5, 2008 at 8:32 pmQuote this Comment
Christina - Gravatar

Christina said:

Thanks for the info Michael! The script actu­ally DOES say –opt, but the way that “code” is dis­played on this page didn’t show the dashes clearly (I’ll have to try to change that) — the down­load­able script has the cor­rect dashes too. I’m glad that it worked for you and I appre­ci­ate the feed­back on the –opt thing. I’ll do my best to change the text dis­play now.

Posted on: July 6, 2008 at 2:37 amQuote this Comment
Christina - Gravatar

Christina said:

OK — I fixed the code for­mat­ting! It should all be cor­rectly dis­played now.

Posted on: July 6, 2008 at 3:56 amQuote this Comment
duivesteyn - Gravatar

duivesteyn said:

I’ve mod­i­fied this to bet­ter suit CPanel based sites with sql sup­port at http://duivesteyn.net/2008/amazon-s3-backup-for-webserver-public_html-sql-bash/

hope it helps someone

Posted on: July 6, 2008 at 4:48 amQuote this Comment
Christina - Gravatar

Christina said:

duivesteyn said: I’ve mod­i­fied this to bet­ter suit CPanel based sites with sql sup­port at http://duivesteyn.net/2008/amazon-s3-backup-for-webserver-public_html-sql-bash/ hope it helps someone

Oh that’s awe­some! Thanks for post­ing your script!

Posted on: July 6, 2008 at 4:51 amQuote this Comment

The­jesh GN said:

s3 is amaz­ing. I also use it deliver images and other media files to blog since its fast.

Karl Hardisty - Gravatar

Karl Hardisty said:

Christina,

Thanks for putting in the time and effort to not only come up with the script, but to make it robust/structured/pretty enough for shar­ing. It’s good to know that some­one has tested it and put it to pub­lic scrutiny, and that it’s worth­while. You’ve saved me (and plenty oth­ers I’m sure) a lot of time.

Much appre­ci­ated.

Posted on: July 7, 2008 at 7:56 amQuote this Comment
Josh Price - Gravatar

Josh Price said:

Sim­ple yet very effective

Thanks!

Posted on: July 8, 2008 at 12:43 pmQuote this Comment
Host Dis­ci­ple » Blog Archive » Inside the host­ing mind of a blogger - Gravatar

Host Dis­ci­ple » Blog Archive » Inside the host­ing mind of a blogger said:

[…] hosted there (calm down fel­las she is dat­ing some­one already). Christina wrote a very infor­ma­tive how-to on back­ing up your Medi­aTem­ple GS […]

Posted on: July 9, 2008 at 12:06 pmQuote this Comment
Patrick Siko­rski - Gravatar

Patrick Siko­rski said:

Can I bor­row your wis­dom? I’m get­ting some weird errors: warning.com@cl01:~/data$ ./backup_server.sh tar: Removing leading /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing ‘/’ from mem­ber names You didn’t set up your envi­ron­ment vari­ables; see README.txt s3sync.rb [options] ver­sion 1.2.6 –help –h –ver­bose –v –dryrun –n –ssl –s –recur­sive –r –delete –public-read –p –expires=”” –cache-control=”“ –exclude=”” –progress –debug –d –make-dirs –no-md5 One of or must be of S3 for­mat, the other a local path. Reminders: * An S3 for­mat­ted item with bucket ‘mybucket’ and pre­fix ‘mypre’ looks like: mybucket:mypre/some/key/name * Local paths should always use for­ward slashes ‘/’ even on Win­dows * Whether you use a trail­ing slash on the source path makes a dif­fer­ence. * For exam­ples see README.

Not sure where my prob­lem is, do you have any idea?

Awe­some arti­cle by the way!

Posted on: July 9, 2008 at 9:38 pmQuote this Comment
Christina - Gravatar

Christina said:

Patrick Siko­rski said: Can I bor­row your wis­dom? I’m get­ting some weird errors: warning.com@cl01:~/data$ ./backup_server.sh tar: Remov­ing lead­ing ‘/’ from mem­ber names tar: Remov­ing leading …

Patrick, OK, this was a prob­lem I had in the begin­ning, and I had to change the s3config.rb file so that confpath = ["./", "#{ENV['S3CONF']}", "#{ENV['HOME']}/.s3conf", "/etc/s3conf"] — make sure that has been changed and try again.

As for the tar: remov­ing lead­ing “/” from mem­ber names, that’s fine.

Hope this helps!

Posted on: July 9, 2008 at 11:41 pmQuote this Comment
Patrick Siko­rski - Gravatar

Patrick Siko­rski said:

Did this go as smoothly for you as it did for me lol. Now for some rea­son I’m get­ting this.

./s3sync.rb:28:in `require': ./s3config.rb:19: syntax error, unexpected tIDENTIFIER, expecting ']' (SyntaxError) config = YAML.load_file("#{path}/s3config.yml") ^ ./s3config.rb:25: syntax error, unexpected kEND, expecting $end from ./s3sync.rb:28

After you told me about that code, I real­ized that I didn’t copy it right. This is prob­a­bly some­thing just as stupid.

Posted on: July 10, 2008 at 8:56 amQuote this Comment
Christina - Gravatar

Christina said:

Patrick, Did you rename the s3confi.yml.sample file to s3config.yml?

If you did, I’ll have to check the code­base (it is pos­si­ble a new ver­sion of S3sync was released since I’ve writ­ten the arti­cle) and investigate.

We’ll get this working!

This might be the sort of thing I should do a screen­cast of, from start to fin­ish, to sup­ple­ment the writ­ten guide. Hmm…

Posted on: July 10, 2008 at 10:20 amQuote this Comment
Patrick Siko­rski - Gravatar

Patrick Siko­rski said:

Yes I renamed the file. I guess a new ver­sion could have been released… but you didn’t write the arti­cle that long ago. Update your ver­sion of it and see if it breaks (backup first lol). A screen­cast would be cool.…!

Posted on: July 10, 2008 at 10:28 amQuote this Comment
Christina - Gravatar

Christina said:

OK — a new ver­sion has NOT been released, so I’m think­ing this is prob­a­bly as sim­ple as a mis-typed comma or period somewhere.

I’ll make a screen­cast today, going from start to finish.

Posted on: July 10, 2008 at 11:04 amQuote this Comment
Patrick Siko­rski - Gravatar

Patrick Siko­rski said:

Awe­some, I’ll delete every­thing and start over when you make the screen cast. Thanks!

Posted on: July 10, 2008 at 11:53 amQuote this Comment
Karl Hardisty - Gravatar

Karl Hardisty said:

Christina,

Excel­lent news. I can guar­an­tee at least one viewer.

Posted on: July 10, 2008 at 12:08 pmQuote this Comment

Video Tuto­r­ial: Auto­mate Media Tem­ple (gs) back­ups with Ama­zon S3 | www.ChristinaWarren.com said:

[…] This entry was posted on Sun, July 13th, 2008. You can fol­low any responses to this entry through the RSS 2.0 feed. You can leave a response, or track­back from your own site. Pre­vi­ous Entry […]

Matt - Gravatar

Matt said:

The back­ups are going fine, but the s3sync por­tion keeps giv­ing me:

Con­nec­tion reset: Con­nec­tion reset by peer 99 retries left, sleep­ing for 30 sec­onds Con­nec­tion reset: Con­nec­tion reset by peer 98 retries left, sleep­ing for 30 sec­onds Con­nec­tion reset: Con­nec­tion reset by peer 97 retries left, sleep­ing for 30 sec­onds Con­nec­tion reset: Con­nec­tion reset by peer … and so on

Any ideas?

Posted on: July 16, 2008 at 3:31 pmQuote this Comment
Matt - Gravatar

Matt said:

Hmmm… I guess that’s nor­mal? I checked my S3 bucket and it had all the files there and in the right size. So are those mes­sages just there to say that it is still working?

Posted on: July 16, 2008 at 3:47 pmQuote this Comment
Christina - Gravatar

Christina said:

Hmm, I don’t get any of those Matt — but if the copies are trans­fer­ring over cor­rectly, I guess its fine.

Posted on: July 17, 2008 at 10:09 pmQuote this Comment
Cedric - Gravatar

Cedric said:

I tried 2 times, but it doesn’t work at all for me :

tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names tar: Remov­ing lead­ing /' from member names tar: Removing leading/’ from mem­ber names Per­ma­nent redi­rect received. Try set­ting AWS_CALLING_FORMAT to SUBDOMAIN S3 ERROR: # ./s3sync.rb:290:in +': can't convert nil into Array (TypeError) from ./s3sync.rb:290:ins3TreeRecurse’ from ./s3sync.rb:346:in main' from ./thread_generator.rb:79:incall’ from ./thread_generator.rb:79:in initialize' from ./thread_generator.rb:76:innew’ from ./thread_generator.rb:76:in initialize' from ./s3sync.rb:267:innew’ from ./s3sync.rb:267:in ‘main’ from ./s3sync.rb:735

Posted on: July 25, 2008 at 5:38 pmQuote this Comment
Maxwell Scott-Slade - Gravatar

Maxwell Scott-Slade said:

I have set it up exactly the same as your blog (btw, thanks for all this) but I also get this error:

tar: Removing leading /' from member names tar: Removing leading/’ from mem­ber names Per­ma­nent redi­rect received. Try set­ting AWS_CALLING_FORMAT to SUBDOMAIN S3 ERROR: # ./s3sync.rb:290:in +': can't convert nil into Array (TypeError) from ./s3sync.rb:290:ins3TreeRecurse’ from ./s3sync.rb:346:in main' from ./thread_generator.rb:79:incall’ from ./thread_generator.rb:79:in initialize' from ./thread_generator.rb:76:innew’ from ./thread_generator.rb:76:in initialize' from ./s3sync.rb:267:innew’ from ./s3sync.rb:267:in ‘main’ from ./s3sync.rb:735

Posted on: July 27, 2008 at 6:59 amQuote this Comment
Ced - Gravatar

Ced said:

Maybe s3sync has been updated since this post ?

Posted on: July 27, 2008 at 9:32 amQuote this Comment
Christina - Gravatar

Christina said:

OK, so both Cedric and Maxwell are get­ting the same error. I looked up that error and it appears to be asso­ci­ated with EU buck­ets. Are either of you using buck­ets in the EU?

To change this, you need to add this line to your s3config.yml file:

AWS_CALLING_FORMAT: SUBDOMAIN

Posted on: July 27, 2008 at 9:38 amQuote this Comment
Christina - Gravatar

Christina said:

Ced — no, I just double-checked. I think the issue is either with using an EU bucket name or some­thing else in the bucket-name not being cor­rect. EU Bucket names can­not con­tain cap­i­tal let­ters (so they are not case-sensitive), whereas US bucket names can.

Make sure your bucket name is cor­rect in the script. I think adding the AWS_CALLING_FORMAT para­me­ter to the yml file will solve the problem.

Posted on: July 27, 2008 at 9:43 amQuote this Comment
Ced - Gravatar

Ced said:

@Christina : Yes it works now ! Thanks for your help.

Posted on: July 27, 2008 at 11:13 amQuote this Comment
Christina - Gravatar

Christina said:

Ced, Glad to hear it! I’ve updated the post with that infor­ma­tion in case any­one else runs into the same issue.

Posted on: July 27, 2008 at 11:59 amQuote this Comment
Maxwell Scott-Slade - Gravatar

Maxwell Scott-Slade said:

Thanks Christina, that’s all work­ing fine now. It’s so awe­some to know that the site is get­ting backed up every­day to a safe place 100%. It’s nice to turn on the email fea­ture in the Cron job sec­tion so you know it’s all done.

A guide to remem­ber. I never used SSH before, now that I have I feel pretty happy it all works!

Posted on: July 28, 2008 at 4:07 amQuote this Comment

Faire un backup de son blog sur Ama­zon S3 | 64k said:

[…] avec ce ser­vice, mais je n’avais pas vrai­ment pris le temps de tester. Je suis tombé sur un bil­let de Christina War­ren qui m’a décidé, puisqu’il décrit toute la procé­dure pour faire un backup d’un […]

Eas­ily Backup Your Entire Web­site to S3 | HighEdWebTech said:

[…] sent me a great link to a Ruby script that will backup your web­site and push that backup file to Ama­zon S3 for safe, […]

Philip - Gravatar

Philip said:

So does this backup every­thing or just the con­tent that’s changed since the last backup?

Posted on: August 24, 2008 at 6:45 pmQuote this Comment
Links of Inter­est — CSS-Tricks - Gravatar

Links of Inter­est — CSS-Tricks said:

[…] War­ren has a com­pre­hen­sive and excel­lent tuto­r­ial on cre­at­ing a Ruby script to back up your entire web server, includ­ing data­bases, and upload them […]

Posted on: August 26, 2008 at 8:46 amQuote this Comment
Christina - Gravatar

Christina said:

Philip, I was hav­ing some prob­lems hav­ing effec­tive recur­sive back­ups, so it’s just doing every­thing. Your com­ment reminds me to re-investigate the best/most effec­tive way to do it recur­sively though (it would be like a few char­ac­ter changes to the script), so I’ll do that later this week and post an update with my find­ings. Real­is­ti­cally, your data­bases are going to be chang­ing more fre­quently than your actual direc­to­ries, so you can always set a sep­a­rate CRON job to run the data­bases and cer­tain fold­ers every day, and other fold­ers less fre­quently. That’s what I do any­way — my blogs and data­bases are backed up daily and a few domains that are just basi­cally image stor­ing for right now get updated once a week or once a month.

Posted on: August 26, 2008 at 9:40 amQuote this Comment
Matt - Gravatar

Matt said:

The backup part works great for me, but not the s3sync. Cron job doesn’t even bother with copy­ing any­thing over. When I do the site copy to S3 man­u­ally, it usu­ally dies after copy­ing just a few sites. Wish I could get that part work­ing as that is the impor­tant part!

Posted on: August 26, 2008 at 4:35 pmQuote this Comment
Christina - Gravatar

Christina said:

Matt, Are you get­ting an error of any kind? About how much data is being copied over before it dies? I’ve suc­cess­fully copied over more than 500 megabytes before using the script (a test case and also a backup of some pho­tographs I uploaded to my gs tem­porar­ily when at my parent’s house). Let’s see if we can fig­ure out why it isn’t working.

Posted on: August 26, 2008 at 6:29 pmQuote this Comment
Matthew Barker - Gravatar

Matthew Barker said:

Nope, not get­ting any errors that mat­ter it seems. Ear­lier I reported this error:

Con­nec­tion reset: Con­nec­tion reset by peer 99 retries left, sleep­ing for 30 sec­onds 98 retries left, sleep­ing for 30 seconds …

But that doesn’t seem to really mat­ter. I am back­ing up 10 sites, with only 1 being larger than 500 MB; the gzipped/tarred file is cur­rently 1.3 GB in size. The odd thing about all of this is that some­times every­thing works when I do it man­u­ally, but that is only some­times. It gen­er­ally quits when trans­fer­ring the 1.3 GB file to Ama­zon, with no error mes­sages encoun­tered. But with the cron job run­ning, it some­times quits when tar­ring the 1.3 GB site, but gen­er­ally tars every­thing just fine, but doesn’t trans­fer a thing. That’s the hard part about try­ing to trou­bleshoot this prob­lem; some­times it works, some­times it doesn’t; and when it doesn’t work, it doesn’t die at the same place every time.

Posted on: August 27, 2008 at 12:15 amQuote this Comment
Peter - Gravatar

Peter said:

I was hop­ing I could down­load your full backup script and cus­tomize it to my needs, but it looks like access it denied on the file from S3. Is there any chance you could make it public?

Posted on: August 27, 2008 at 10:31 amQuote this Comment
Mar­cus McCurdy - Gravatar

Mar­cus McCurdy said:

Thanks for this great write up. I went through it last night and it worked like a champ and I learned a thing or two in the process. This really makes back­ing up a media tem­ple site a breeze.

Posted on: August 27, 2008 at 4:13 pmQuote this Comment
Christina - Gravatar

Christina said:

Matt, I remem­ber you had that error. I’ll do what I can to inves­ti­gate why this seems to not be work­ing. It might be nec­es­sary to cre­ate two sep­a­rate Cron jobs — one for the biggest site, one of the oth­ers — to see if that is an accept­able workaround.

Peter — OK, try it now — I changed the URL struc­ture. It was work­ing fine, I’m not sure what could have changed. If you still have issues, let me know.

Mar­cus — yay! I’m so happy this worked for you!

Posted on: August 27, 2008 at 4:54 pmQuote this Comment

Auto­mat­i­cally back up your entire web server files and data­bases to Ama­zon S3 | QuickPipe said:

[…] War­ren has a com­pre­hen­sive and excel­lent tuto­r­ial on cre­at­ing a Ruby script to back up your entire web server, includ­ing data­bases, and upload them […]

Links of Interest | Proba - Gravatar

Links of Interest | Proba said:

[…] War­ren has a com­pre­hen­sive and excel­lent tuto­r­ial on cre­at­ing a Ruby script to back up your entire web server, includ­ing data­bases, and upload them […]

Posted on: August 31, 2008 at 6:15 pmQuote this Comment
Matt - Gravatar

Matt said:

Thanks Christina … this was really help­ful. How about a Part 2 with rolling back­ups? ;-)

Posted on: September 8, 2008 at 6:08 pmQuote this Comment

Auto­mat­i­cally back up Meti­aTem­ple Grid (gs) via Ama­zon S3 | Ran­domess of Josh Price said:

[…] months now, and thought I’d share it with you guys.  Christina War­ren has writ­ten up some excel­lent instruc­tions on how to back up MediaTemple’s Grid (gs) to your Ama­zon S3 account.  There is also a video. […]

Michael York - Gravatar

Michael York said:

Hey Christina,

Not sure if you got my e-mail or not…

Any­ways, I wanted to know why I keep get­ting “Con­nec­tion Reset: con­nec­tion reset by peer” error mes­sages. When look­ing at the S3 bucket, not all of the files have trans­ferred over…

Thanks! Michael

Posted on: September 25, 2008 at 5:39 pmQuote this Comment
Pet­ter - Gravatar

Pet­ter said:

Matthew Barker said: Nope, not get­ting any errors that mat­ter it seems. Ear­lier I reported this error: Con­nec­tion reset: Con­nec­tion reset by peer 99 retries …

I am expe­ri­enc­ing the same prob­lem. It is a fairly large site. About 2.5GB worth of pho­tos. I have split the backup script up in two parts, the DB and the Site. The DB works fine and goes up to S3 eas­ily. Leads me to believe it is related to the size :). Have to divide it up in smaller chunks or similar.

Mat, Christina, how did/would you approach it? any suggestions.

Posted on: September 29, 2008 at 10:34 amQuote this Comment
Christina - Gravatar

Christina said:

Peter, Mat, and any­one else receiv­ing the con­nec­tion reset mes­sage: It IS size related. I’m work­ing on a new post that details chang­ing the num­ber of resets and does recur­sive back­ups for large files — so that you don’t trans­fer every­thing over every sin­gle time. I need to test it out and make sure it is solid before post­ing, but it should be up some­time this week!

Posted on: September 29, 2008 at 10:37 amQuote this Comment
Jor­dan - Gravatar

Jor­dan said:

Christina,

Thanks for the well writ­ten guide. I was able to get my back­ups run­ning per­fectly but did have one question.

I am using a slightly mod­i­fied ver­sion of your backup script, and while it works well as far as cre­at­ing the backup files, I don’t see any of the echo mes­sages in either the shell or in the cron emails. I only see the following:

tar: Removing leading /' from member names tar: Removing leading/’ from mem­ber names

Oth­er­wise, this seems to be the per­fect backup sys­tem for Media Tem­ple gs users.

Posted on: September 30, 2008 at 4:44 pmQuote this Comment
William B. - Gravatar

William B. said:

Hey Christina,

This has really helped my mis­sion, thanks for the guide. Works flaw­lessly for me.

Posted on: October 5, 2008 at 10:59 amQuote this Comment
Poll Results: How do you back up your web­sites? — CSS-Tricks - Gravatar

Poll Results: How do you back up your web­sites? — CSS-Tricks said:

[…] this poll was to tap into your col­lec­tive knowl­edge and find some sweet auto­mated backup solu­tions. Christina’s solu­tion is absolutely awe­some and fits into my sit­u­a­tion per­fectly (being on Media Tem­ple, and hav­ing an S3 […]

Posted on: October 8, 2008 at 8:45 amQuote this Comment
Tim Pea­cock - Gravatar

Tim Pea­cock said:

Christina, really appre­ci­ate the help with this. A cou­ple of errors, but got these ironed out, and now work­ing per­fectly. Saved me so much time !!

Posted on: October 13, 2008 at 6:25 amQuote this Comment
Ced - Gravatar

Ced said:

How to choose the data­base to save ? (and not all the data­bases) Because with more than 5–6 db, it uses too much ressources : “(mt) Media Temple’s auto­mated MySQL mon­i­tor­ing sys­tems have detected an increase in data­base activ­ity for your Grid-Service”

Posted on: October 15, 2008 at 3:22 amQuote this Comment
Matt - Gravatar

Matt said:

Any word on when this will be ready? I could def­i­nitely use it; my back­ups fail con­sis­tently because of one site that is around 2GB in size.

Christina said: Peter, Mat, and any­one else receiv­ing the con­nec­tion reset mes­sage: It IS size related. I’m work­ing on a new post that …
Posted on: October 15, 2008 at 10:49 pmQuote this Comment
Christina - Gravatar

Christina said:

Matt, It will be up by the end of the week. I had work com­mit­ments and then went on vaca­tion and this week has been crazy busy!

Posted on: October 15, 2008 at 10:57 pmQuote this Comment
Matt - Gravatar

Matt said:

Any luck on the size lim­i­ta­tion issue? Thanks!

Posted on: October 27, 2008 at 7:04 amQuote this Comment
Week in Review (Octo­ber 20 through Octo­ber 26) | Grand­mas­terB dot com - Gravatar

Week in Review (Octo­ber 20 through Octo­ber 26) | Grand­mas­terB dot com said:

[…] I also setup an account on the Ama­zon AWS to remotely backup my sites. After read­ing a tuto­r­ial by Christina War­ren, I setup the nec­es­sary soft­ware and a shell script to do auto­matic back­ups of my site daily. The […]

Posted on: October 27, 2008 at 4:45 pmQuote this Comment
creede - Gravatar

creede said:

I get a con­nec­tion reset too, but it seems to go through even­tu­ally. Let me know if you get rolling back­ups going. Thanks again for the great tool!

Posted on: November 12, 2008 at 6:30 pmQuote this Comment

Ama­zon S3 for Back­ups — Effec­tive Programming said:

[…] Christina Warren […]

Mikkel - Gravatar

Mikkel said:

I get the same error as Matt when I test the script:

Con­nec­tion reset: Con­nec­tion reset by peer

99 retries left, sleep­ing for 30 seconds

How­ever when I look in my bucket (using trans­mit), I see the files are turn­ing up just fine, so I guess I should just ignore the error. But it would be nice to have it run­ning smoothly.

An yes, I’m using a EU bucket (sit­ting in Denmark).

Another thing I’ve noticed; It backs up all of my datab­ses, but only the domain direc­to­ries I choose in the “backup_server.sh” file. Is there a way to exlude data­bases? I have a lot of test sites (with db’s), that arent realy that impor­tant to backup.

Posted on: December 8, 2008 at 7:32 amQuote this Comment
Samm - Gravatar

Samm said:

First, Christina, thank you for this arti­cle — I am using mt (gs) and S3 and am hav­ing prob­lem. Prob­a­bly a mis­take on my part, but look­ing for some help. I have com­pleted the setup, and when I run the com­mand ./backup_server.sh, I get the mes­sage: “bad inter­preter: No such file or direc­tory”. The file is clearly there, I can see it via FTP and it shows with the dir com­mand in Putty. Any ideas?

Posted on: December 10, 2008 at 12:26 pmQuote this Comment
Michael - Gravatar

Michael said:

Hey Christina,

Any news on the recur­sive script?

Thanks! –Michael

Posted on: December 10, 2008 at 2:08 pmQuote this Comment
trs21219 - Gravatar

trs21219 said:

i got this run­ning fine but i want the back­ups to go into sub­di­rec­to­ries in my s3 bucket for each day that it backs up. i got it doing it for my server but how do i make it make the fold­ers on S3?

im adding this to the end of the direc­tory path at the top and bot­tom of the backup.sh

directorypath........./date +%m-%d-%y/

Posted on: December 16, 2008 at 2:29 amQuote this Comment
trs21219 - Gravatar

trs21219 said:

trs21219 said: i got this run­ning fine but i want the back­ups to go into sub­di­rec­to­ries in my s3 bucket for each …

nev­er­mind i got it ! here is my backup_server.sh file

!/bin/sh

A list of web­site direc­to­ries to back up

websites=“domains.com domain2.com”

The des­ti­na­tion direc­tory to backup the files to

destdir=/home/xxxxx/users/.home/s3sync/s3backup/date +%m-%d-%y

The direc­tory where all web­site domain direc­to­ries reside

domaindir=/home/xxxx/users/.home/domains

The MySQL data­base hostname

dbhost=internal-db.sxxxxx.gridserver.com

The MySQL data­base user­name — requires read access to databases

dbuser=xxxxxx

The MySQL data­base password

dbpassword=xxxxxxx

mkdir $dest­dir rm $destdir/*.tar.gz

echo date “: Begin­ning backup process…” > $destdir/backup.log

backup data­bases

for dbname in echo 'show databases;' | /usr/bin/mysql -h $dbhost -u$dbuser -p$dbpassword do if [ $dbname != “Data­base” ]; then echo date “: Back­ing up data­base $dbname…” » $destdir/backup.log /usr/bin/mysqldump –opt –h $dbhost –u$dbuser –p$dbpassword $dbname > $destdir/$dbname.sql tar –czf $destdir/$dbname.sql.tar.gz $destdir/$dbname.sql rm $destdir/$dbname.sql fi done

backup web content

echo date “: Back­ing up web con­tent…” » $destdir/backup.log for web­site in $web­sites do echo date “: Back­ing up web­site $web­site…” » $destdir/backup.log tar –czf $destdir/$website.tar.gz $domaindir/$website done

echo date “: Backup process com­plete.” » $destdir/backup.log

echo date “: Com­pact­ing Pack­age Total…” » $destdir/backup.log tar –czf $destdir.tar.gz $destdir

rm –r –f $destdir

The direc­tory where s3sync is installed

s3syncdir=/home/xxxxx/users/.home/s3sync

The direc­tory where the backup archives are stored

backupdir=/home/xxxxxx/users/.home/s3sync/s3backup/

The S3 bucket a.k.a. direc­tory to upload the back­ups into

s3bucket=BUCKET NAME HERE

cd $s3syncdir ./s3sync.rb $backupdir/ $s3bucket:

hope this helps some peo­ple… i got it from trail and error… thanks again christina for the awe­some script!

Posted on: December 16, 2008 at 3:06 amQuote this Comment
Adam - Gravatar

Adam said:

I thought I did every­thing right, but got this after run­ning the script manually:

com­mand not found line 4:

com­mand not found line 8:
com­mand not found line 10:
com­mand not found line 14:
com­mand not found line 16:
com­mand not found line 20:
com­mand not found line 22:
com­mand not found line 26:
com­mand not found line 28:
com­mand not found line 32:
com­mand not found line 34:
com­mand not found line 40: ./backup_server.sh: line 42: /backup.log: Read-only file system
com­mand not found line 44: rm: can­not remove /*.tar.gz': No such file or directory rm: cannot remove\r’: No such file or directory
com­mand not found line 50: ’/backup_server.sh: line 70: syn­tax error near unex­pected token '/backup_server.sh: line 70:fi

What did I do wrong?

Posted on: January 7, 2009 at 4:45 pmQuote this Comment
Adam - Gravatar

Adam said:

Adam said: I thought I did every­thing right, but got this after run­ning the script man­u­ally: : com­mand not found line 4: : command …

Nev­er­mind. My bucket name had a typo. Sorry.

Does any­one know of a way to fil­ter out cer­tain direc­to­ries (for exam­ple: “cache” fold­ers)? I also have drop­box fold­ers that I don’t need to backup, etc. I’d love to exclude them.

Posted on: January 8, 2009 at 5:18 pmQuote this Comment
A - Gravatar

A said:

Proper back­ups are like eat­ing your veg­eta­bles — we all say we’ll do it and that it is a good idea, but it is so much eas­ier NOT to do it and eat Oreo cook­ies instead”

Its refresh­ing to see analo­gies like these..once in a while atleast..web would be so much greener, warmer & health­ier if even guy geeks start giv­ing such analogies..i wish..

Posted on: January 17, 2009 at 5:31 amQuote this Comment
Adam - Gravatar

Adam said:

Adam said: Does any­one know of a way to fil­ter out cer­tain direc­to­ries (for exam­ple: “cache” fold­ers)? I also have drop­box fold­ers that I don’t need to backup, etc. I’d love to exclude them.

In case any­one else is inter­ested, I made two changes to the script that I’m really happy with. I changed the com­pres­sion type to bzip2 (bz2) instead of gzip (gz), and my file sizes shrunk dra­mat­i­cally! Addi­tion­ally, I excluded some files and direc­to­ries that I didn’t need backed up and knew were full of tons on files. This saves a lot of time and end file size. Here’s the main change, but a cou­ple oth­ers need to be made for this to fully work:

tar –jcf $destdir/$website.tar.bz2 –exclude ‘/cache’ –exclude ‘/dropbox’ –exclude ‘*.zip’ $domaindir/$website

Posted on: January 17, 2009 at 1:09 pmQuote this Comment
Johnny - Gravatar

Johnny said:

hi!

Is there any way to backup the e-mails as well from the grid-server?

(Thanks for the tuto­r­ial, it’s work­ing fine on my side.)

Posted on: January 18, 2009 at 7:59 amQuote this Comment
jota - Gravatar

jota said:

Hello,

It’s been a few weeks since the script stopped work­ing for me.

Now I’m get­ting this error

Read from remote host domain.com: Con­nec­tion reset by peer Con­nec­tion to domain.com closed.

Any clue what might be happening?

Posted on: June 6, 2009 at 8:25 amQuote this Comment
Matt - Gravatar

Matt said:

Hello Christina,

Excel­lent script! It seems to work fine for me all the files and dbs get backed up but I get this error for some reason:

/usr/bin/mysqldump: Got error: 1044: Access denied for user ‘db62917’@’%’ to data­base ‘information_schema’ when using LOCK TABLES

Not quite sure why this is, any ideas?

Matt

Posted on: June 23, 2009 at 4:38 amQuote this Comment
Pietro - Gravatar

Pietro said:

Hi Christina, thanks a lot! Your tuto­r­ial is fantastic!

I’m get­ting the same error as Matt, though: /usr/bin/mysqldump: Got error: 1044: Access denied for user ‘dbxxxxx’@’%’ to data­base ‘information_schema’ when using LOCK TABLES

Any­way, the script is working…

Pietro

Posted on: July 20, 2009 at 4:21 amQuote this Comment
Karl Gech­lik - Gravatar

Karl Gech­lik said:

I am get­ting errors, can you help me out? Any help would be greatly appreciated.

tar: Remov­ing lead­ing /' from member names S3 command failed: list_bucket prefix max-keys 200 delimiter / With result 404 Not Found S3 ERROR: # ./s3sync.rb:290:in+’: can’t con­vert nil into Array (Type­Er­ror) from ./s3sync.rb:290:in s3TreeRecurse' from ./s3sync.rb:346:inmain’ from ./thread_generator.rb:79:in call' from ./thread_generator.rb:79:inini­tial­ize’ from ./thread_generator.rb:76:in new' from ./thread_generator.rb:76:inini­tial­ize’ from ./s3sync.rb:267:in new' from ./s3sync.rb:267:inmain’ from ./s3sync.rb:735

Posted on: August 4, 2009 at 10:20 amQuote this Comment
Christina - Gravatar

Christina said:

Hey guys, OK — I’m upload­ing new copies of the screen­cast now and will have those links switched in about 15 min­utes. Karl, give me a lit­tle bit to look over the code, I need to do a follow-up post (13 months later, wow!) any­way :-)

Posted on: August 4, 2009 at 10:25 amQuote this Comment
Karl Gech­lik - Gravatar

Karl Gech­lik said:

Thank you so much Christina I am sit­ting on the edge of my seat pulling my hair out for a few days now (more like a week!) I can gen­er­ate the backup file but I have to man­u­ally xfer it to s3. I am won­der­ing if it is my bucket name it has a — in it. Any assis­tance is REALLY REALLY appre­ci­ated! You rock.

Posted on: August 4, 2009 at 10:34 amQuote this Comment
MattO - Gravatar

MattO said:

SOME HELP FOR NEWBIES ON HOW TO DO THIS AND SOME PROBLEMS I RAN INTO:

Hi Christina,

You Rock! This is a huge help — you have no idea how hard this stuff can be to get done when your not a pro.

I just want to doc­u­ment a few prob­lems that I encoun­tered when try­ing this so oth­ers can hope­fully skip my frus­tra­tions on Sat­ur­day Morning : )

1) If you run into the prob­lem of not being able to see the s3sync direc­tory that you create.

  • In Christina’s video it shows up as a direc­tory next to your domain folder, how­ever, in my expe­ri­ence it wasn’t any­where to be found. I knew it existed because I could see it in SSH, but in FTP and my Account Cen­ter it was invisible.

  • To rec­tify this you can go into your FTP set­ting and show hid­den files. In filezilla this is under the Server Menu and is called “Force show­ing hid­den files”

–Then, your not done yet — to actu­ally see the s3sync direc­tory — go into your users direc­tory and you should now see a direc­tory called “.home” — you’ll find it in there.

2) Put your server_backup.sh script in the data direc­tory — this is the main data direc­tory in the same hier­ar­chi­cal level as your domains folder and etc folder

3) When you fig­ure all this out and you actu­ally run the script you’ll get some errors — the first ones I hit were related to buck­ets in AWS.

–If your new at all this you need to cre­ate a bucket, but this can be kinda confusing.

–I finally found this sim­ple way to do it: http://it.toolbox.com/blogs/oracle-guide/cloud-studio-for-aws-ec2-and-s3-28117

–Essen­tially you down­load the cloud stu­dio free­ware (FYI — I’m on MAC), open it, go into the set­tings and enter your account info, don’t worry about all the RED text (that has to do with another amazaon image tool), go down toward the bot­tom of the screen and add a new bucket. That should do the trick — now go to the server_backup.sh script and put the name of your bucket there like she tells you to do.

4) You might still be get­ting an error that has this in it: “s3sync/s3backup/*.tar.gz’: No such file or directory”

–This is because the way that the ama­zon direc­to­ries are down­loaded is a bit dif­fer­ent than the script is writ­ten (I think).

–To solve this sim­ply go into your FTP and sim­ply move the S3backup direc­tory inside of the s3sync direc­tory where it is look­ing for it.

OK! I know it’s the clear­est explaina­tion, but I hope it saves some­one a few hours!

Thanks again for the detailed tutorial!

–MattO

http://mattoden.com

Posted on: August 22, 2009 at 3:25 pmQuote this Comment
MattO - Gravatar

MattO said:

Hi Christina,

Quick Ques­tion:

Is there a way to mod­ify the script so that you don’t over­write the back­ups on S3?

For Exam­ple — I would like to run a daily backup and on S3 I would like the html direc­to­ries and the sql data­bases to append the date to each ver­sion. Day one: Example.com_8_25_09.tar.gz, Example.com_8_26_09.tar.gz, etc.

That way, if there is a prob­lem, I can roll back to the last fully func­tion­ing date.

Thanks,

MattO

Posted on: August 22, 2009 at 3:35 pmQuote this Comment
Christina - Gravatar

Christina said:

Matt, Amaz­ing, amaz­ing — I’ll be updat­ing the post this week (I’ve been say­ing that for a year but I actu­ally mean it this time) and will be using your cor­rec­tions — the for­mat­ting has changed a bit in the last year and I need to update the script.

I’ll also include a way for you to write a new file each time (with a date appended). I’m sure we can fig­ure out a way to have a rule to only keep the last X or what­ever too.

Posted on: August 23, 2009 at 2:03 pmQuote this Comment
Coa­lesce - Gravatar

Coa­lesce said:

Thanks for the help­ful tuto­r­ial. We’re going to be await­ing your revised script!

Posted on: August 27, 2009 at 12:34 pmQuote this Comment
Andrew - Gravatar

Andrew said:

Just imple­mented this on my gs server, thanks for the great script. Ear­gly antic­i­pat­ing the revised script with the dated backups.

Posted on: August 28, 2009 at 11:24 pmQuote this Comment
zxr - Gravatar

zxr said:

Hi guys!

I still have a prob­lem when work­ing with a big backup file. One of my sites is around 1gb of data and this method does not work for It. I run the script but It says comthing like the time has expired…

Any solu­tiono for this?

Thanks!

Posted on: August 29, 2009 at 12:33 pmQuote this Comment
Mikkel Hansen - Gravatar

Mikkel Hansen said:

Any news on the update? Would realy love to be able to keep the last x days of back­ups on the server!

Posted on: September 8, 2009 at 5:50 pmQuote this Comment
Andrew - Gravatar

Andrew said:

I was won­der­ing if you had an ETA for a way to keep mul­ti­ple dated back­ups, I needed that today. Thanks for the script though, it was a lifesaver.

Posted on: September 15, 2009 at 12:06 amQuote this Comment
Brett Wilcox - Gravatar

Brett Wilcox said:

Hello Christina!

I just wanted to share with you and every­one here that I have cre­ated a VERY robust backup script that keeps mul­ti­ple rotat­ing back­ups and such.

I will be releas­ing more infor­ma­tion soon but I have posted the code at http://www.brettwilcox.com/2009/09/20/amazon-s3-backup-script/ for any­one interested.

Shoot me an email if any­one has any questions!

brett@brettwilcox.com

Posted on: September 21, 2009 at 1:21 amQuote this Comment
Anthony Abraira - Gravatar

Anthony Abraira said:

This is prob­a­bly a stu­pid ques­tion, but is there a means of doing some­thing where this can instead zip up to one file that goes to a des­ig­nated folder on the site. That way one can just go in and down­load the zip file and have their back up with­out need­ing an addi­tional server stor­age space.

If this has been answered I am sorry, can some­body point me in the right direction?

Posted on: September 21, 2009 at 2:49 pmQuote this Comment
Brett Wilcox - Gravatar

Brett Wilcox said:

Hello Anthony,

I have cre­ated that very script that you are talk­ing about.

http://www.brettwilcox.com/2009/09/20/amazon-s3-backup-script/

Just dis­able the ama­zon S3 Sync option and it will cre­ate mul­ti­ple rotat­ing dated back­ups with­out the need for ama­zon S3.

Posted on: September 21, 2009 at 2:55 pmQuote this Comment
Brett Wilcox - Gravatar

Brett Wilcox said:

I have released a newer ver­sion of the script and pub­lished it to google code — http://code.google.com/p/ziplinebackup/

Posted on: September 30, 2009 at 5:09 pmQuote this Comment
Hugh Esco - Gravatar

Hugh Esco said:

A cou­ple of thoughts I have not seen addressed above:

I would be care­ful about delet­ing the pre­vi­ous backup prior to cre­at­ing the new one.

If mem­ory serves, an S3 bucket is lim­ited to a giga­byte or so. This may be the issue folks are encoun­ter­ing back­ing up larger sites.

Essen­tial touch to time stamp back­ups and keep the last sev­eral around. I once had to roll back sev­eral days to find an uncor­rupted data­base backup. That was when I gave up email (for scp and rsync) as a trans­port for files which mattered.

if you can add some sort of log rotate func­tion to this script you’ll have a full fea­tured backup sys­tem in place.

Back­ing up to S3 is a fine way to store pub­lic data acces­si­ble on the web any­way, but for folks deal­ing with pro­pri­etary data, inhouse file stores, or at least encrypted S3 buck­ets are prob­a­bly a more appro­pri­ate way to go.

And one last impor­tant step, as you add new sites and new resources which ought to be backed up, it impor­tant to update your backup script to account for those.

Thanks for the arti­cle. Was unaware of this ruby tool.

– Hugh

Posted on: October 20, 2009 at 1:27 amQuote this Comment
irms - Gravatar

irms said:

This was awe­some. Thanks!

Posted on: December 14, 2009 at 8:16 amQuote this Comment
Clay­ton - Gravatar

Clay­ton said:

Thanks for the tuto­r­ial, Christina. This post, along with your video, were a huge help for me.

Posted on: January 5, 2010 at 8:06 pmQuote this Comment
Clay­ton - Gravatar

Clay­ton said:

This appears to be work­ing, but I am get­ting an error mes­sage at the top of my Cron job noti­fi­ca­tion email: “Got error: 1044: Access denied for user ‘dbXXXXX’@’%’ to data­base ‘information_schema’ when using LOCK TABLES

Any­one else get this?

Posted on: January 8, 2010 at 10:07 amQuote this Comment
Christina War­ren - Gravatar

Christina War­ren said:

Clay­ton, Yes — a user was nice enough to put up a new script on github — the link is in the com­ments (a few posts before yours) — I des­per­ately need to update this blog post with that info — thanks for the reminder!

Posted on: January 8, 2010 at 10:09 amQuote this Comment
Karl L. Gech­lik | AskTheAdmin.com - Gravatar

Karl L. Gech­lik | AskTheAdmin.com said:

Thanks for your help. I rewrote the script to cre­ate cus­tom file­names with the date in them. I then croned another script to remove files after 7 days keep­ing 7 back­ups. I could not get your MySql part to work so I cronned out MySql dumps and put them in the tar. Finally I reduced the file size with com­pres­sion and fil­ter­ing out spe­cific file types I didnt need backed up. I will get it doc­u­mented and up on http://www.askTheAdmin.com as soon as I can.

Posted on: January 8, 2010 at 10:14 amQuote this Comment
Jesse - Gravatar

Jesse said:

Thank you so much for spelling all this out for us! This was extremely help­ful! I am using Ama­zon S3 now, it’s kind of tricky to use but this helped me save much time :)

Posted on: February 24, 2010 at 5:12 pmQuote this Comment

Media Tem­ple back­ups naar Ama­zon S3 met S3Sync said:

[…] GB per maand + upload en down­load fees. Er is geen ver­plichte afname van het aan­tal GBs. Kudos voor Christina War­ren en haar blog­post die de basis vormt voor deze Ned­er­landse ver­tal­ing en kudos voor Brett Wilcox, […]

Andre Kreft - Gravatar

Andre Kreft said:

Hi Christina, love the script and has been work­ing for a while now. Lately I got some strange error’s I cant seem to fix.

Like: No such file or direc­tory in my Cron Dea­mon emails and when I try to kick off my cron job through SSH I can’t kick off ./server_backup.sh in my data folder. It keep say­ing: –bash: /backup_server.sh: No such file or directory

Any­thing changed on MT side? or am I miss­ing some­thing. Hope you can help.

Posted on: March 17, 2010 at 4:41 amQuote this Comment
the Blog Butler - Gravatar

the Blog Butler said:

Thanks for writ­ing this up Cristina, was a great help and got me off to a good start. I have every­thing work­ing fine when I run it from an SSH ses­sion but when the cron kicks the job off it will back up the site but will NOT trans­fer it to my S3 bucket. It does it just fine when I run it man­u­ally but not from cron.

Any­one have any clue on this?

Posted on: June 29, 2010 at 6:40 pmQuote this Comment
Tan­ner Hobin - Gravatar

Tan­ner Hobin said:

Thank you, thank you, thank you!

So I tried to make things a lit­tle eas­ier on myself by mak­ing the “list of web­site direc­to­ries to backup” /domains and “the direc­tory where all web­site domain direc­to­ries reside” /home/XXXXX/users/.home and the result­ing .tar.gz file only had fold­ers in it for home>XXXXX>users. No domains backed-up.

Sim­ply put, I was hop­ing to skip hav­ing to list all domains and the need to update the script every time I add/remove a domain. Any idea how I might be able to do that?

Again, thank you.

Posted on: July 16, 2010 at 5:47 pmQuote this Comment
Brett - Gravatar

Brett said:

Any idea on how to cor­rect these errors in the zipline scripte?

======================================================================

Data­base Backup Start Time Fri Oct 8 01:33:30 PDT 2010

Back­ing up data­bases from /home/86415/data/database_backups tar: Remov­ing lead­ing ‘/’ from mem­ber names tar: /home/86415/data/database_backups: Can­not stat: No such file or direc­tory tar: Error exit delayed from pre­vi­ous errors Complete

======================================================================

Data­base Backup End Time Fri Oct 8 01:33:30 PDT 2010

======================================================================

Start­ing Trans­fer to Online Stor­age Backup Fri Oct 8 01:33:30 PDT 2010

Now trans­fer­ing Back­ups to Ama­zon S3 S3 com­mand failed: list_bucket max-keys 200 pre­fix delim­iter / With result 404 Not Found S3 ERROR: # ./s3sync.rb:290:in +': can't convert nil into Array (TypeError) from ./s3sync.rb:290:ins3TreeRecurse’ from ./s3sync.rb:346:in main' from ./thread_generator.rb:79:incall’ from ./thread_generator.rb:79:in initialize' from ./thread_generator.rb:76:innew’ from ./thread_generator.rb:76:in initialize' from ./s3sync.rb:267:innew’ from ./s3sync.rb:267:in ‘main’ from ./s3sync.rb:735

Com­plete

Posted on: October 8, 2010 at 4:36 amQuote this Comment
Brett - Gravatar

Brett said:

Oh crap this is the wrong blog to ask that ques­tion, I should really not mess with this sort of thing this late at night. ><

Posted on: October 8, 2010 at 4:38 amQuote this Comment
Bar­ton - Gravatar

Bar­ton said:

I also get the error:

Access denied for user ‘dbxxxxx_s3sync’@’%’ to data­base ‘yyyyy’ when using LOCK TABLES

Posted on: October 18, 2010 at 2:25 pmQuote this Comment
Bar­ton - Gravatar

Bar­ton said:

Okay, to all peo­ple with the “Access denied” … “using LOCK TABLES” error:

The backup is work­ing, its just fail­ing when it tries to backup the information_schema, which isn’t required.

Posted on: October 18, 2010 at 2:30 pmQuote this Comment
nosaukums - Gravatar

nosaukums said:

Some ideas, I defined in the begin­ning of backup script: DATESTAMP=date +%Y-%m-%d

and mod­i­fied to include it in all file­names i.e.:

backup data­bases

for dbname in echo 'show databases;' | /usr/bin/mysql -h $dbhost -u$dbuser -p$dbpassword do if [ $dbname != “Data­base” ]; then echo date “: Back­ing up data­base $dbname…” » $destdir/backup.log /usr/bin/mysqldump –h $dbhost –opt –skip-lock-tables –u$dbuser –p$dbpassword $dbname > $destdir/$DATESTAMP.$dbname.sql tar –czf $destdir/$DATESTAMP.$dbname.sql.tar.gz $destdir/$DATESTAMP.$dbname.sql rm $destdir/$DATESTAMP.$dbname.sql fi done

backup web content

echo date “: Back­ing up web con­tent…” » $destdir/$DATESTAMP.backup.log for web­site in $web­sites do echo date “: Back­ing up web­site $web­site…” » $destdir/backup.log tar –czf $destdir/$DATESTAMP.$website.tar.gz $domaindir/$website done

also use­ful thing — exclude a spe­cific direc­tory (in my case cash, save some pre­cious band­width ;) ) tar -czf $destdir/$DATESTAMP.$website.tar.gz $domaindir/$website --exclude 'notthis/cache/*'

also couldnt fig­ure out data­base lock­ing issue so i added “–skip-lock-tables” to com­mand. so far so good.

thanks so much fur such a great tutorial.

Posted on: May 9, 2011 at 4:15 amQuote this Comment
nosaukums - Gravatar

nosaukums said:

ouch, for­mat­ting screwed up.

any­way.. just add “$DATESTAMP.” to all “$dbname.sql.tar.gz” and to first “$website.tar.gz”

tadaa..

and not tested yet, but I replaced the orig­i­nal “rm $destdir/*.tar.gz” (remove all tar.gz files) with “find /$destdir –type f –mtime +10 –exec rm {} \;” (delete all files older then 10 days.)

Posted on: May 9, 2011 at 4:20 amQuote this Comment
hum­bert - Gravatar

hum­bert said:

mmmmm cook­ies.…..!

Posted on: October 26, 2011 at 10:56 pmQuote this Comment
Rosamunda - Gravatar

Rosamunda said:

Thanks Christina! I was so help­less about what to do to backup remotely my sites in my brand new MT account!! THANK YOU VERY MUCH!!! Gra­cias! Rosamunda from Buenos Aires, Argentina

Posted on: August 19, 2012 at 1:04 pmQuote this Comment
Dave F - Gravatar

Dave F said:

Thanks so much for post­ing this. It was EXACTLY what I was look­ing for. Per­fectly writ­ten and com­mented. THANK YOU!. I did run into one snag. I keep get­ting this error when try­ing to backup my domain archive. Broken pipe: Broken pipe 99 retries left, sleeping for 30 seconds Broken pipe: Broken pipe 98 retries left, sleeping for 30 seconds

The sql archive uploaded to s3 per­fectly but it fails with this file for some rea­son. Any idea why? My site is VERY image depen­dant (for a mod­el­ing agency). The site archive is about 10GB. It cre­ated the archive but its fail­ing to upload it from my MT Grid Server.

Any help is greatly appreciated.

Thanks,

Dave

Posted on: May 6, 2013 at 1:10 amQuote this Comment
Dave F - Gravatar

Dave F said:

Just a fol­low up. I split the very large tar file into 950MB parts and it seems to solve the Bro­ken Pipe prob­lem. Hope this helps someone.

Posted on: May 6, 2013 at 10:04 amQuote this Comment