Mark's Blog

Amazon and Rackspace both provide virtualized servers billed per hour. That makes deploying scalable applications, especially with distributed key/value stores, very handy: You could launch new servers just as load, customer count or requests increase.

Today I’ve run the Redis benchmark on the SheevaPlug, Amazon EC2 and Rackspace Cloudservers to see where a small Redis database will perform the best.

test environment

Benchmarks have been conducted under Gentoo in order to have an identical test environment in terms of packages and versions. Prior to lauching any instances I have created a snapshot of the latest portage tree and my set of overlays, set portage binhosts (if available) and made sure every machine had the same configuration:

mount --bind /dev/shm /tmp

for D in portage-main-latest.tar.lzma bootstrap-latest.tar.lzma overlays-latest.tar.lzma; do
  wget http://.../distfiles/$D -O - | tar --lzma -xp -C /
done

cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime
sed -ie 's:#TIMEZONE="Factory":TIMEZONE="Europe/Berlin":g' /etc/conf.d/clock

# The following is already integrated into amdfam10-slave branch of portage.git:
# add binhost and FEATURES+=getbinpkg to make.conf
# mirrorselect -D -s3 -b10 -o

emerge -C sys-fs/e2fsprogs \
&& emerge -Kn --keep-going --deep system  \
&& emerge -K --keep-going glibc gcc binutils \
&& gcc-config x86_64-pc-linux-gnu-4.4.1 \
&& source /etc/profile \
&& emerge -K --keep-going --deep world \
&& rc-update del rsyslog default && emerge -C sys-process/vixie-cron app-admin/rsyslog \
&& emerge -1Kn --keep-going ccache nano eix colordiff app-portage/mirrorselect \
          srv-electron dev-util/git dev-db/tokyocabinet dev-db/redis dev-cpp/asio \
&& eix-update

ntpdate 130.75.1.32

cd cd /usr/local/overlays/
rm -r ossdl-overlay
git clone http://git.ossdl.de/r/ossdl-overlay.git

git clone -b amdfam10-slave http://git.ossdl.de/r/portage.git \
|| git clone -b amdfam10 http://git.ossdl.de/r/portage.git
eix-sync

emerge --deep world

Rackspace provides the current Gentoo 10.1, but Amazon confused me with Gentoo images from 2008 (!) with Python 2.4. That really takes the biscuit! I will comment on that later.

So, for Amazon EC2 I had to update Portage and Python twice, re-emerge the whole system and had still trouble installing Nginx at all. (Therefore I won’t benchmark Nginx today.) Here’s a rough outline of what had to be done prior to the above:

alias dir='ls -al --color'
echo app-arch/lzma >> /etc/portage/package.keywords
echo sys-devel/autoconf >> /etc/portage/package.keywords
echo "=sys-devel/autoconf-2.6*" >> /etc/portage/package.keywords
echo "=sys-devel/autoconf-2.6* **" >> /etc/portage/package.keywords
echo "sys-devel/autoconf-wrapper" >> /etc/portage/package.keywords

# adjust settings for the CPU in make.conf

emerge -1n lzma
wget http://binhost.ossdl.de/distfiles/overlays-latest.tar.lzma
lzma d overlays-latest.tar.lzma - | tar -xvp -C /
rm -r /usr/portage
wget http://binhost.ossdl.de/distfiles/portage-main-latest.tar.lzma
lzma d portage-main-latest.tar.lzma - | tar -xvp -C /

emerge -1n eix mirrorselect
mirrorselect -s3 -b10 >> /etc/make.conf
echo PORTDIR_OVERLAY="/usr/local/overlays/ossdl-overlay" >> /etc/make.conf
echo 'USE_EXPAND="${USE_EXPAND} NGINX_MODULES"' >> /etc/make.conf
echo 'NGINX_MODULES="static rewrite autoindex auth_basic access limit_zone limit_req geo geoip map referer redis rewrite proxy upstream_ip_hash accept_language'" >> /etc/make.conf

eselect profile set 1
emerge -1 portage
emerge -1 =dev-lang/python-2.6*
eselect python set python2.6 && python-updater
emerge -1 portage
emerge redis nginx

As you can see from the applications list above I have benchmarked Tokyo Cabinet and some other software, too.
Redis is version 1.2.1 (without debugging symbols).

hardware configuration

benchmark commands

Redis already ships with a benchmark program. For comparability I have used it instead of writing my own. If requested I could do the last and benchmark with different key or value sizes.

/usr/sbin/redis-benchmark -n 1000000 -q

Yes, that is a million.

results

summary

The smallest Rackspace instance does not have much memory, but given that it costs only about 1/10th the price Amazon EC2 bills for its equivalent it is a bargain. Rackspace has done everything right: No matter what machine you request, the architecture (AMD Opteron, amd64) will be the same. And, you can grow (and shrink) its capacities on-the-fly to a larger configuration.

Amazon should really update their Gentoo images. At the hourly rate they bill it is inacceptable to lose about two to three hours for something which you shouldn’t care about. Yes, Gentoo 10.1 is out, you need not necessarily have the latest packages pre-installed, but at least updates every six months are a must – for every distribution!

EC2 instances are not homogeneous, Rackspace’s are. Although “smallest” and “large” share the same CPU type, you get the one in 32bit and the other in 64bit. There is no small instance in 64bit with EC2. Additionally, the processor families and even architectures vary amongst the same class. About a year ago I had been provided with Xeon Nocomas.

On the other side, Amazon gives more memory and space, Queues and so on. The 20GB 10GB HDD of Rackspace smallest are… pre-historic. As a rule of thumb I feel it is reasonable to say, the smallest instances should have at least as much HDD space as the cheapest 2.5″ (or whatever the prevailing size is) drive on the end-user market. Which is 80GB as of today.

Rackspace claims their instances are run on RAID 10, which is nice, and that the latter storage ispersistent. In case of a systems failure the virtual machine gets migrated to an other host. That is IMHO a big plus.

Bottom line, I will stick with Rackspace.

Although they lost my data on first registration. (Are you reading this guys? I still didn’t get my CloudFiles API key and customer number…)

See also:

• Redis Benchmarking on Amazon EC2, Flexiscale, and Slicehost at porteightyeight

Bookmark to:

Tags: Amazon EC2, benchmark, cloud, Rackspace CloudServers, Redis, SheevaPlug