Designing a highly availabile directory service

This reading is a chapter from Sun's Java System Directory Service manual about how to set up a replication topology for reliability. To understand it fully, a bit of background is needed from the other chapters:

1. The role of a directory service is to store key-value pairs within some directory of suffixes, for a read-mostly workload. The keys and values are both small.
2. Sun's directory service can run in three modes: master, hub and consumer. A consumer is read-only cache and is updated by another server. A hub is similar to a consumer but its purpose is to push updates from a master to multiple consumers. Finally, a master performs writes as well as reads.

The article starts with a description of backup mechanisms available in Directory Service. There is a fast coarse-grained binary backup that only works on identical machines, and a structured data (LDIF) backup that can be restricted to parts of a data set and can be loaded on any machine. This shows what might be a fundamental tradeoff between performance and flexibility of backup mechanisms: binary snapshots are very easy to take, but keep data in a machine-dependent, non-portable and difficult-to-subdivide format. Standard formats are more portable and future-proof but require more work to write. The guide recommends using both types of backups (having a binary one locally and on similar machines only).

The article then goes into some sample topologies for various scenarios. The single-datacenter scaled for read performance is the first complex one, and it is very interesting because it contains a detailed analysis of possible faults, including link faults. We see why having hubs and multiple redundant links (e.g. a link from each master to each hub) is useful for dealing with the various failures. The two-datacenter example in addition mentions that you want to have two independent links between data centers, as well as "backup agreements" that are disabled for the case when more than one master fails. Finally, when we get up to five data centers, we see an interesting limitation of the software: the software supports only up to 4 masters. Presumably this is because they need to run a consensus protocol, and consensus protocols don't scale very well beyond this level. This is an interesting disadvantage of consensus protocols that we haven't seen in many cloud computing papers and might represent a fundamental tradeoff between performance and availability due to latency of network links, etc.

Overall this reading is especially interesting because it comes from a highly available enterprise system, not from the web company perspective behind many cloud papers. Enterprises are serious about availability and it shows in this detailed user guide. Failure scenarios are carefully examined and best practices are described. Equally interesting is the fact that Directory Service is a product which external administrators need to be trained to use through a manual (and courses presumably), so it needs to not only be reliable but be administrable by someone who is not a designer of the original system. Cloud software (Hadoop, Ruby on Rails, etc) rarely comes with instructions this good often and requires experienced personnel to manage it.