hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject [DISCUSS]Helix for global synchronization in Hama
Date Wed, 08 May 2013 22:01:05 GMT
Hello,

I am starting a discussion thread on potential pros/cons of using Helix in
Hama. I dont know the internal details of Hama, so please correct me if
something does not make sense.

My source of information is http://wiki.apache.org/hama/Architecture and a
brief chat with Suraj at ApacheCon where he described the need for barriers
between super steps.

Please read about Apache Helix here http://helix.incubator.apache.org/.

Architecture wise Helix maps pretty well with the components in Hama.
HelixController can be wrapped inside BSPMaster and GroomServer is the
PARTICIPANT in Helix terminology that wraps Helix Agent.

The partitioning and assigning tasks to GroomServers can be done via Helix
Apis, it basically boils down to setting the idealstate for a particular
stage. Starting of the next step which basically depends on all tasks in
previous step being completed can be done by watching the ExternalView.

In the architecture wiki, I see that there is plan to integrate with
Zookeeper for fault tolerance. Helix internally uses Zookeeper to store the
cluster state. So it might make it easier to make the tasks fault tolerant
and probably restartable as well at a task level instead of job/stage level.

We recently added a recipe in Helix to demonstrate the concept of
dependency between resources.

http://helix.incubator.apache.org/recipes/task_dag_execution.html
Code:
https://github.com/apache/incubator-helix/tree/master/recipes/task-execution/src/main/java/org/apache/helix/taskexecution

Let me know your thoughts.

thanks,
Kishore G

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message