hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HedWig/TopicManagement" by ErwinTam
Date Tue, 11 Jan 2011 19:18:45 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HedWig/TopicManagement" page has been changed by ErwinTam.
http://wiki.apache.org/hadoop/HedWig/TopicManagement?action=diff&rev1=3&rev2=4

--------------------------------------------------

     * '''hedwig''' is the root. If the !ZooKeeper instance is shared between regions, then
there would be a region-specific root.
        * '''regionname''' is the region this Hedwig cluster lives in.
           * '''topics''' is the root for the topics subtree. All topics that have been created
live under this root.
-             * '''T1, T2 ... ''' are nodes for each topic. The nodes are named by the topic
name. The fact that a topic has a node under the <B>Topics</B> node means that
the topic exists. If we had any other topic metadata (like permissions) that we wanted to
store, we could store them as the content of the node.
+             * '''T1, T2 ... ''' are nodes for each topic. The nodes are named by the topic
name. The fact that a topic has a node under the '''Topics''' node means that the topic exists.
If we had any other topic metadata (like permissions) that we wanted to store, we could store
them as the content of the node.
                 * '''Ti.hub''' is the hub that is currently assigned to the topic. This is
an ephemeral node, so that if the hub fails, another hub can take over. The name of the node
is "Hub" and the content is the hostname of the hub.
                 * '''hubscribers''' is the root of the tree of subscribers to the topic.

                    * '''Sub1, Sub2 ...''' are the subscribers. Each node is named with the
subscriber ID (which should be descriptive enough for the hub to be able to deliver messages
to the subscriber.) The content of the node includes the subscriber's current consume mark
for the topic.
@@ -84, +84 @@

        * If a hub exists, but is a different hub than the one the client contacted (e.g.
H1!=H2), then the hub returns a ''redirect'' message, requesting that the client retry its
publish at H2.
        * If no hub exists, H1 will check the flag of the ''publish'' call to see if the client
was redirected.
           * If false (the client has not yet been redirected) H1 will choose a random hub
H3 (possibly itself) to manage the topic. 
-             * If H1 chooses itself (H1==H3), then H1 will try to create an ephemeral node
under <B>T</B> called <B>Hub</B> with its own hostname (e.g. H1) as
the content). This creation should be done using test-and-set, so that if a <B>Hub</B>
node already exists, the creation fails.
+             * If H1 chooses itself (H1==H3), then H1 will try to create an ephemeral node
under '''T''' called '''Hub''' with its own hostname (e.g. H1) as the content). This creation
should be done using test-and-set, so that if a '''Hub''' node already exists, the creation
fails.
                 * If the ephemeral node creation succeeds, then H1 will set up its internal
bookkeeping to begin publishing messages on T, and will accept and publish C's message.
-                * Otherwise (ephemeral node creation fails) then H1 will read the hostname
(e.g. H4) of the hub assigned to the topic from the <B>T.Hub</B> node, and return
to the client a <I>redirect</I> message, requesting that the client retry its
publish at H4.
+                * Otherwise (ephemeral node creation fails) then H1 will read the hostname
(e.g. H4) of the hub assigned to the topic from the '''T.Hub''' node, and return to the client
a ''redirect'' message, requesting that the client retry its publish at H4.
-             * Otherwise (H1!=H3), then H1 will return to the client a <I>redirect</I>
message, requesting that the client retry its subscription at H3.
+             * Otherwise (H1!=H3), then H1 will return to the client a ''redirect'' message,
requesting that the client retry its subscription at H3.
           * If true (the client has been redirected), then H1 will try to become the owner
of the topic.
-             * H1 will try to create an ephemeral node under <B>T</B> called
<B>Hub</B> with its own hostname (e.g. H1) as the content. This creation should
be done using test-and-set, so that if a <B>Hub</B> node already exists, the creation
fails.
+             * H1 will try to create an ephemeral node under '''T''' called '''Hub''' with
its own hostname (e.g. H1) as the content. This creation should be done using test-and-set,
so that if a '''Hub''' node already exists, the creation fails.
                 * If the ephemeral node creation succeeds, then H1 will set up its internal
bookkeeping to begin publishing messages on T, and will accept and publish C's message.
-                * Otherwise (ephemeral node creation fails) then H1 will read the hostname
(e.g. H3) of the hub assigned to the topic from the <B>T.Hub</B> node, and return
to the client a <I>redirect</I> message, requesting that the client retry its
publish at H3.
+                * Otherwise (ephemeral node creation fails) then H1 will read the hostname
(e.g. H3) of the hub assigned to the topic from the '''T.Hub''' node, and return to the client
a ''redirect'' message, requesting that the client retry its publish at H3.
  
- Notes:
+ ==== Notes: ====
   
-    * We may want to optimize this procedure so that the hub does not have to contact !ZooKeeper
on every publish. This could be done using leases, or by depending on the disconnect exception...the
proper way to do this is an open question.
+    * We may want to optimize this procedure so that the hub does not have to contact ZooKeeper
on every publish. This could be done using leases, or by depending on the disconnect exception...the
proper way to do this is an open question.
     * Instead of redirecting the client, we could forward the published message to the correct
hub. But redirecting seems cleaner, since we would really like the publisher to be directly
connected to the correct hub.
  
  === Topic redistribution ===
  
- Occasionally, we should shuffle topics between hubs to ensure load balancing. For example,
when a new hub joins, we want topics to be assigned to it. Similarly, if some topics are hotter
than others, the hub should be able to shed load. Since all of the persistent state about
a topic is in !ZooKeeper or !BookKeeper, shuffling a single topic can be easy: the hub just
stops accepting publishes and deletes its ephemeral node. The next time a client tries to
subscribe or publish to the topic, it will get assigned to a random hub.
+ Occasionally, we should shuffle topics between hubs to ensure load balancing. For example,
when a new hub joins, we want topics to be assigned to it. Similarly, if some topics are hotter
than others, the hub should be able to shed load. Since all of the persistent state about
a topic is in ZooKeeper or BookKeeper, shuffling a single topic can be easy: the hub just
stops accepting publishes and deletes its ephemeral node. The next time a client tries to
subscribe or publish to the topic, it will get assigned to a random hub.
  
  When should a hub abandon a topic? It should do so at least under the following conditions:
  
-    * If the hub is overloaded, compared to the other hubs in the system (determined by load
statistics stored by the hubs in their !ZooKeeper <B>hedwig.regionname.hosts.Si</B>
node.)
+    * If the hub is overloaded, compared to the other hubs in the system (determined by load
statistics stored by the hubs in their ZooKeeper '''hedwig.regionname.hosts.Si''' node.)
-    * Periodically (e.g. when the hub is closing the !BookKeeper log for the topic)
+    * Periodically (e.g. when the hub is closing the BookKeeper log for the topic)
  
  The constant shuffling of topics should help to keep load evenly balanced across hubs, without
human intervention. Moreover, lazily abandoning topics will help the shuffling to occur in
an incremental fashion spread out over time.
  

Mime
View raw message