helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Terence Yim <cht...@gmail.com>
Subject Re: Question about participant dropping a subset of assigned partitions
Date Sat, 02 Feb 2013 05:44:42 GMT

What do mean by "fake" live instance that you mentioned? I think the Java
proxy could simply creates one HelixManager participant per C++ instance
(hence there are N HelixManager instances in the Java proxy) and disconnect
them accordingly based on the liveness of the C++ process.


On Fri, Feb 1, 2013 at 7:19 PM, Abhishek Rai <abhishekrai@gmail.com> wrote:

> Thanks for the quick and thoughtful response Kishore!  Comments inline.
> On Fri, Feb 1, 2013 at 6:33 PM, kishore g <g.kishore@gmail.com> wrote:
>> Hi Abhishek,
>> Thanks for the good question. We have two options(listed later in the
>> email) for allowing a partition to drop the partitions.  However, It works
>> only in two modes (auto, custom) of three mode(auto_rebalance, auto,
>> custom) the controller can support.
>> More info about modes here
>> http://helix.incubator.apache.org/Features.html
>> Can you let me know which mode you are running it in?.
> We are planning to use CUSTOM mode since we have some specific
> requirements about (1) the desired state for each partition, and (2)
> scheduling partitions to instances.  Our requirements for (1) are not
> expressible in the FSM framework.
> Also is it sufficient if the disabled partitions are re-assigned uniformly
>> to other nodes or you want to other partitions from other nodes to be
>> assigned to this node.
> Once a participant disables some partitions, it's alright for the default
> rebalancing logic to kick in.
>> Also it will help us if you can tell the use case when you need this
>> feature.
> Sure, I'm still trying to hash things out but here is a summary.  The DDS
> nodes are C++ processes, which is the crux of the problem.  AFAIK Helix
> does not have a C++ library, so I'm planning to use a participant written
> in Java, which runs as a separate process on the same node, receives state
> transitions from the controller, and proxies them to the C++ process via an
> RPC interface.  The problem is that the C++ process and the Java-Helix
> proxy can fail independently.  I'm not worried about the Java-Helix proxy
> crashing since that would knock of all partitions in the C++ process from
> Helix view, which does not affect correctness.
> But when the C++ process crashes, the Java-Helix proxy needs to let the
> controller know asap, so the Helix "external view" can be updated,
> rebalancing can start, etc.  One alternative is to invoke
> "manager.disconnect()" from the Helix proxy.  But this would knock off all
> partitions managed by the proxy (I want to retain the ability for the proxy
> to manage multiple C++ programs).  Hence the question about selectively
> dropping certain partitions, viz., the ones in a crashed C++ program.
>> To summarize, you can achieve this in AUTO and CUSTOM but not in
>> AUTO_REBALANCE mode because the goal of controller is always to assign the
>> partitions evenly among nodes. But you bring up a good use case, depending
>> the  behavior we might be able to support it easily.
>> 1. Disable a partition on a given node: Disabling a partition on a
>> particular node should automatically trigger rebalancing. This can be done
>> either by admin using command line tool
>> helix-admin.sh --zkSvr <ZookeeperServerAddress(Required)>
>> --enablePartition <clusterName instanceName resourceName partitionName
>> true/false>
>> or programmatically if you have the access to manager, you can invoke this
>> manager.getClusterManagementTool().enablePartition(enabled,
>> clusterName,instanceName,resourceName,partitionNames);
>> This can be done in auto and custom.
> I am not sure this will have the right effect in the scenario described
> above.  Specifically, the Java proxy would need to disable all the crashed
> partitions, and then re-enable them when the C++ DDS process reboots
> successfully.  If the disable-enable transitions happen too quickly, could
> the controller possibly miss the transition for some partition and not do
> anything?
>> 2. The other option is to change the mapping of partition --> node in the
>> ideal state. ( You can do this programmatically in custom modes and in some
>> what in auto mode as well). Doing this will send transitions to the node to
>> drop the partitions and reassign it to other nodes.
> Yes, this seems like the most logical thing.  The Java proxy will probably
> need to send a message to the controller to trigger this change in the
> ideal states of all crashed partitions.  The messaging API would probably
> be useful here.
> Another alternative I'm considering is for the Java proxy to add a "fake"
> instance for each C++ process that it spawns locally.  The custom
> rebalancer (that I'd write inside the controller) would then schedule the
> C++ DDS partitions on to these "fake" live instances.  When the C++ process
> crashes, the Java proxy would simply disconnect the corresponding fake
> instance's manager.  Does this make sense to you?  Or do you have any other
> thoughts?
> Thanks again for your thoughtful feedback!
> Abhishek
>> Thanks,
>> Kishore G
>> On Fri, Feb 1, 2013 at 5:44 PM, Abhishek Rai <abhishekrai@gmail.com>wrote:
>>> Hi Helix users!
>>> I'm a Helix newbie and need some advice about a use case.  I'm using
>>> Helix to manage a storage system which fits the description of a DDS
>>> ("distributed data service" as defined in the Helix SOCC paper).  Each
>>> participant hosts a bunch of partitions of a resource, as assigned by the
>>> controller.  The set of partitions assigned to a participant changes
>>> dynamically as the controller rebalances partitions, nodes join or leave,
>>> etc.
>>> Additionally, I need the ability for a participant to "drop" a subset of
>>> partitions currently assigned to it.  When a partition is dropped by a
>>> participant, Helix would remove the partition from the current state of the
>>> instance, update the external view, and make the partition available for
>>> rebalancing by the controller.  Does the Java API provide a way of
>>> accomplishing this?  If not, are there any workarounds?  Or, was there a
>>> design rationale to disallow such actions from the participant?
>>> Thanks,
>>> Abhishek

View raw message