distributed key/value stores shootout (part 1)
For a project of mine I have been looking today on key/value stores. Redis, Tokyo Cabinet and Memcache are the popular ones, but I have been interested in benchmarking the distributed species of its kind (“DKVS”). Those are needed in high scalable systems as replacement for traditional databases, which happen not to scale well.
These are the criteria which the DKVS must meet:
- distribute data amongst one to four nodes (and the more the better)
- nodes must be able to be added on-the-fly,
- … and removed on-the-fly
- data has to be held redundantly, even after removal of a node
- data access has to be transparent, i.e. every node shall be equal
- scale with additional nodes – for every additional node there should be an increase of GET and SET operations per second up to the maximum dictated by the hardware and network
Optionally:
- if a node runs out of storage (memory or HDD) its present or new data shall be redirected/migrated to other nodes
- support of atomic operations, such as increment and decrement
- list data type
- backups and restores from another machine
- fully asynchronous
- support a well-known protocol (e.g., memcache or redis)
- be not drastically slower than the popular non-distributed KVS
One Master with multiple slaves is no architecture which falls under my requirements.
So, here are the first two DKVS:
Plurk’s LightCloud
Written in Python, LightCloud is divided into a storage engine launcher (currently for Redis and Tokyo Tyrant) and a access library. It does not qualify for benchmarks, due to first of these points:
- Is not able to add nodes on-the-fly. Every node has to added in a configuration file on every node by hand.
- Thus administration efford grows quadratically.
- Configuration of more than a pair of master/slaves is not documented well enough. I.e., I had four servers and wanted to launch eight redis instances.
- If a node fails, say A from [A A'] (A’ is its slave), then it does not copy data redundantly to another node (say B).
- If a node appears back, it does not synchronize its potentially old data with new one.
TSnosql
Written in Python, primary dependencies (excluding transitive):
net-zope/zope-interface dev-python/axiom dev-python/twisted-web dev-python/ll-core dev-python/which
First of all, the documentation is not coherent. Links are missing, dependencies not properly documented. I had trouble obtaining the source code from Subversion as the link was broken. The source code from SVN has not been the same as in the TSnosql 0.4 tarball.
The SVN version has given me no more than a socket.gaierror: [Errno -2] Name or service not known. For the tarball version I had to stop Redis, the benchmark of the latter yield these results which you can compare to the one of my previous post:
SET: 5170.63 requests per second GET: 13680.30 requests per second INCR: 9345.79 requests per second ** aborted, took too long **
To my surprise the author has written that TSnosql is a distributed key/value store. I didn’t find any option to configure that.
- Currently not a distributed k/v-store.
Bottom Line
Disappointing. I really hope the next products will be usable.
Follow me on Twitter
I may be off base, but have you looked into CouchDB?