hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "ZooKeeper/SoC2010Ideas" by HenryRobinson
Date Fri, 12 Mar 2010 23:55:46 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "ZooKeeper/SoC2010Ideas" page has been changed by HenryRobinson.
http://wiki.apache.org/hadoop/ZooKeeper/SoC2010Ideas?action=diff&rev1=2&rev2=3

--------------------------------------------------

  
  More details about Google's Summer of Code can be found here: http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2010/faqs
- the deadline for our applications is Friday, so get those ideas coming in!
  
+ (Note that the mentorship assignment is tentative in some cases - every project can be mentored,
however!)
+ 
- ----
+ ------
  === Optimizations for WAN Deployments ===
  ==== Possible Mentor ====
  Henry Robinson (henry at apache dot org)
  ==== Requirements ====
  Java, some networking familiarity
  ==== Description ====
- ZK 3.3.0 added ''observers'' which are non-voting members of a ZK ensemble. One use case
for observers is as a proxy to a remote voting ensemble, say in a different data center. Since
observers do not need to vote, there are less strict latency requirements on the delivery
of messages to them. WAN traffic is also expensive. This project would investigate and implement
batching of messages to observers, and potential mechanisms for decreasing the number of messages
that need to be sent. For example, a destructive update to a znode twice in a row does not
theoretically need to be sent twice - although making this work correctly with ZAB will be
a challenge.  
+ ZK 3.3.0 added [[http://www.cloudera.com/blog/2009/12/observers-making-zookeeper-scale-even-further/|''observers'']
which are non-voting members of a ZK ensemble. One use case for observers is as a proxy to
a remote voting ensemble, say in a different data center. Since observers do not need to vote,
there are less strict latency requirements on the delivery of messages to them. WAN traffic
is also expensive. This project would investigate and implement batching of messages to observers,
and potential mechanisms for decreasing the number of messages that need to be sent. For example,
a destructive update to a znode twice in a row does not theoretically need to be sent twice
- although making this work correctly with ZAB will be a challenge.  
+ ----
  === FUSE module for BookKeeper ===
  ==== Possible Mentor ====
  Ben Reed (breed at apache dot org) & Patrick Hunt (phunt at apache dot org)
@@ -27, +30 @@

  BookKeeper is a distributed write ahead log with client & server written in Java. BookKeeper
client & server also use ZooKeeper. There is a BookKeeper API that clients can use to
integrate write ahead logging into their application. It would be a lot easier if applications
could use BK without changes to the client application through use of a file system api (FUSE).
The project would involve implementing a C interface for BookKeeper (Java already exists)
and implementing the FUSE module.
  
  Example use: the write ahead log in mysql, called binlogs are typically written to the local
filesystem using the std filesystem api. We could modify mysql to use BooKeeper, however if
we had a BK FUSE module we could run it (mysql) w/o any modification and get the performance/reliability
of a distributed write ahead log.
+ ----
+ === Web-based Administrative Interface ===
+ ==== Possible Mentor ====
+ Henry Robinson (henry at apache dot org)
+ ==== Requirements ====
+ Modern web platform - e.g. Django. Some design or UI skills would help. Java for adding
methods to ZooKeeper. 
+ ==== Description ====
+ ZooKeeper is a complex distributed system. Understanding how well it is running is tremendously
important. Patrick Hunt has created a Django-based dashboard (see http://github.com/phunt/zookeeper_dashboard#readme)
that allows some insight into how ZooKeeper is running. This is a great foundation on which
to build; however there are improvements that could be made! This project would capture much
more information from ZooKeeper, adding hooks to retrieve it where necessary and visualise
it in a appealing and useful way. Integration with [[http://ganglia.sourceforge.net/|Ganglia]]
would be a definite plus. 
  
+ ----
+ === Failure Detector Module ===
+ ==== Possible Mentor ====
+ Henry Robinson (henry at apache dot org)
+ ==== Requirements ====
+ Java, some distributed systems knowledge, comfort implementing distributed systems protocols
+ ==== Description ====
+ ZooKeeper servers detects the failure of other servers and clients by counting the number
of 'ticks' for which it doesn't get a heartbeat from other machines. This is the 'timeout'
method of failure detection and works very well; however it is possible that it is too aggressive
and not easily tuned for some more unusual ZooKeeper installations (such as in a wide-area
network, or even in a mobile ad-hoc network).
+ 
+ This project would abstract the notion of failure detection to a dedicated Java module,
and implement several failure detectors to compare and contrast their appropriateness for
ZooKeeper. For example, [[http://incubator.apache.org/cassandra/|Apache Cassandra]] uses a
phi-accrual failure detector (http://ddsg.jaist.ac.jp/pub/HDY+04.pdf) which is much more tunable
and has some very interesting properties. This is a great project if you are interested in
distributed algorithms, or want to help re-factor some of ZooKeeper's internal code. 
+ 
+ ----
+ === Concurrent Primitives Library ===
+ ==== Possible Mentor ====
+ Henry Robinson (henry at apache dot org)
+ ==== Requirements ====
+ Java / C / Python / Ruby
+ ==== Description ====
+ ZooKeeper is very powerful, but sometimes a bit hard to use. This project will create a
library of concurrent primitives such as locks in many varieties (RW, MRSW, MRMW), concurrent
queues, barriers, latches and semaphores that application developers can use to more easily
take advantage of ZooKeeper's power. Writing distributed systems code is hard, and therefore
should be done at most once!
+ 
+ This project would contribute a solid library in at least one of our supported languages.
It is possible that improvements to the client library will be required - e.g. wrapper code
to retry failed RPC calls. That falls under the scope of this project!
+ 
+ This would be a very interesting project to work on, as it will directly influence the evolution
of the ZooKeeper API when you discover things that the current API makes needlessly difficult.

+ 
+ See http://www.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/
for a detailed example of how to build a distributed queue. 
+ 
+ ----
+ === ZooKeeper DNS Server ===
+ ==== Possible Mentor ====
+ Henry Robinson (henry at apache dot org)
+ ==== Requirements ====
+ Java or Python or C
+ ==== Description ====
+ Although ZooKeeper is primarily used for co-ordination of distributed processes, its consistency
semantics means that it's a good candidate for serving small (key,value) records as well.
The Domain Name Service has similar requirements, raising the interesting question of whether
ZooKeeper would be a capable DNS server for your local network. One intriguing possibility
is having ''versioned'' DNS records, such that known-good configurations can be stored and
rolled back to in the case of an issue.  If this versioning primitive proves to be useful,
it's easy to imagine other types of configuration that could be stored.
+ 
+ This project would involve designing and building an RFC-1035 compliant DNS server and performing
a detailed performance study against an already existant simple DNS server like tinydns. 
+ 
+ ----
+ === Read-only mode ===
+ ==== Possible Mentor ====
+ Henry Robinson (henry at apache dot org)
+ ==== Requirements ====
+ Java and TCP/IP networking
+ ==== Description ====
+ When a ZooKeeper server loses contact with over half of the other servers in an ensemble
('loses a quorum'), it stops responding to client requests because it cannot guarantee that
writes will get processed correctly. For some applications, it would be beneficial if a server
still responded to read requests when the quorum is lost, but caused an error condition when
a write request was attempted. 
+ 
+ This project would implement a 'read-only' mode for ZooKeeper servers (maybe only for Observers)
that allowed read requests to be served as long as the client can contact a server.
+ 
+ This is a great project for getting really hands-on with the internals of ZooKeeper - you
must be comfortable with Java and networking otherwise you'll have a hard time coming up to
speed. 
+ 

Mime
View raw message