how to copy your Wordpress files to CloudFront efficiently
Following is my bash script for copying Wordpress blog files, such as downloads, images, attachments in general to CloudFront. Written with efficiency in mind it will actually synchronize files thus sparing you massive uploads and lets you use CloudFront as CDN.
CSS and JS files will be compressed before uploading, if necessary, and the appropriate headers will be set correctly.
I’ve been inspired by the script on Mudy’s blog, which I found being inefficient as it yields many requests agains S3 (resulting in a slightly higher monthly bill, I guess) and which utilizes a Ruby script. But mine depends on s3cmd, which is written in Python, as I believe more people have Python installed than Ruby. Plus s3cmd has a native ’sync’ command.
I believe you can easily alter my script to fit other blogging systems or even arbitrary folder structures. You’re welcome to let the public know whether you succeeded in this.
So, here is the script:
#!/bin/bash
# Synchronizer script of static Wordpress files to Amazon S3.
# by W-Mark Kubacki; wmark@hurrikane.de
# http://mark.ossdl.de/2009/09/how-to-copy-your-wordpress-files-to-cloudfront-efficiently
#
# Utilizes s3cmd, which can be found here:
# http://s3tools.org/s3cmd
#
# Licensed under RPL for private and academia use,
# and only so for commercial use and blogs with advertisements if
# a paypal donation is made to the author.
#
# version 2009-09-11-r2
#
# the bucket files will be synced with; e.g. s3://my-blog/
BUCKET=s3://your-blog/
# temporary directory for CSS and JS file structure, will be created
TMPDIR=/tmp/your-blog
# all files except JS and CSS
for DIR in ./; do
s3cmd sync --exclude 'wp-admin/**' --exclude 'wp-content/cache/**' --exclude '**.svn**' \
--exclude '*.php' --exclude '*.js' --exclude '*.css' --exclude '.htaccess' \
--exclude '*.orig' \
--acl-public --guess-mime-type --add-header=Cache-Control:max-age=604800 \
--no-preserve --recursive "${DIR}" "${BUCKET}"
done
# JS and CSS files; being compressed if necessary, then synchronized
test -e $TMPDIR && rm -r $TMPDIR
for F in $(find -type f -readable -name '*.js' -o -name '*.css' ! -path '**.svn**' ! -path 'wp-content/cache/**' | grep -v 'wp-admin/' | cut -b 3-); do
mkdir -p "${TMPDIR}/${F%/*}"
file "${F}" | grep -q gzip \
&& cp -a "${F}" "${TMPDIR}/${F}" \
|| gzip -9 -c "${F}" > "${TMPDIR}/${F}"
done
cd $TMPDIR
s3cmd sync --exclude '*.*' --include '*.js' --include '*.css' \
--add-header=Content-Encoding:gzip \
--acl-public --guess-mime-type --add-header="Cache-Control:max-age=604800, public" \
--no-preserve --recursive ./ "${BUCKET}"
cd -
test -e $TMPDIR && rm -r $TMPDIR
You can also download my own latest version here.
Follow me on Twitter
[...] CSS, JS, images and the such by a tool, e.g. to Amazon S3 and let it there be served by CloudFront. This article describes how, and this one how you can directly upload your [...]
Generally I do not post on blogs, but I would like to say that this post really forced me to do so, Excellent post!
I don’t know If I said it already but …Excellent site, keep up the good work. I read a lot of blogs on a daily basis and for the most part, people lack substance but, I just wanted to make a quick comment to say I’m glad I found your blog. Thanks,
A definite great read..Jim Bean
[...] available on Amazon Cloudfront, for example. In our case we used some kind of script described on Mark’s blog. It’s a great script which even gzips content and adds custom [...]
Thanks a lot Mark, saved me quite some time
Thanks for this script! Spent some time optimising my blog and this script made it pretty much effortless!
Great work!
i just downloaded the latest version and currently get this output on a cpanel based server
bash-3.2$ ./s3sync.sh js
Will sync JS and CSS…
find: invalid predicate `-readable’
./s3sync.sh: line 42: cd: /tmp/antsomerset.co.uk: No such file or directory
./s3sync.sh: line 47: cd: OLDPWD not set
it otherwise looks like its uploading files fine and they have the correct content encoding headers etc, but its not compressing or uploading compressed versions of the files, its the same issue with the script in the article or the latest version
any thoughts?
Anthony, I run the script on several blogs at least once a week. Therefore I have no doubt it works correctly – in my environment. So let’s determine where that differs from yours.
My BASH is version 4.0_p37, findutils version 4.4.2. What are yours?
GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
GNU find version 4.2.27
Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION SELINUX
this is latest centos 5.5 running cpanel
got it working by adding a create tmp dir command and removing the -readable flag for my setup.
how do you reccomend triggering the sync command and how often
Anthony, every time you have written an article, prior to clicking on preview or publish.