helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Rai <abhishek...@gmail.com>
Subject Re: Question about participant dropping a subset of assigned partitions
Date Sat, 02 Feb 2013 03:19:14 GMT
Thanks for the quick and thoughtful response Kishore!  Comments inline.

On Fri, Feb 1, 2013 at 6:33 PM, kishore g <g.kishore@gmail.com> wrote:

> Hi Abhishek,
> Thanks for the good question. We have two options(listed later in the
> email) for allowing a partition to drop the partitions.  However, It works
> only in two modes (auto, custom) of three mode(auto_rebalance, auto,
> custom) the controller can support.
> More info about modes here http://helix.incubator.apache.org/Features.html
> Can you let me know which mode you are running it in?.

We are planning to use CUSTOM mode since we have some specific requirements
about (1) the desired state for each partition, and (2) scheduling
partitions to instances.  Our requirements for (1) are not expressible in
the FSM framework.

Also is it sufficient if the disabled partitions are re-assigned uniformly
> to other nodes or you want to other partitions from other nodes to be
> assigned to this node.

Once a participant disables some partitions, it's alright for the default
rebalancing logic to kick in.

> Also it will help us if you can tell the use case when you need this
> feature.

Sure, I'm still trying to hash things out but here is a summary.  The DDS
nodes are C++ processes, which is the crux of the problem.  AFAIK Helix
does not have a C++ library, so I'm planning to use a participant written
in Java, which runs as a separate process on the same node, receives state
transitions from the controller, and proxies them to the C++ process via an
RPC interface.  The problem is that the C++ process and the Java-Helix
proxy can fail independently.  I'm not worried about the Java-Helix proxy
crashing since that would knock of all partitions in the C++ process from
Helix view, which does not affect correctness.

But when the C++ process crashes, the Java-Helix proxy needs to let the
controller know asap, so the Helix "external view" can be updated,
rebalancing can start, etc.  One alternative is to invoke
"manager.disconnect()" from the Helix proxy.  But this would knock off all
partitions managed by the proxy (I want to retain the ability for the proxy
to manage multiple C++ programs).  Hence the question about selectively
dropping certain partitions, viz., the ones in a crashed C++ program.

> To summarize, you can achieve this in AUTO and CUSTOM but not in
> AUTO_REBALANCE mode because the goal of controller is always to assign the
> partitions evenly among nodes. But you bring up a good use case, depending
> the  behavior we might be able to support it easily.
> 1. Disable a partition on a given node: Disabling a partition on a
> particular node should automatically trigger rebalancing. This can be done
> either by admin using command line tool
> helix-admin.sh --zkSvr <ZookeeperServerAddress(Required)>
> --enablePartition <clusterName instanceName resourceName partitionName
> true/false>
> or programmatically if you have the access to manager, you can invoke this
> manager.getClusterManagementTool().enablePartition(enabled,
> clusterName,instanceName,resourceName,partitionNames);
> This can be done in auto and custom.
I am not sure this will have the right effect in the scenario described
above.  Specifically, the Java proxy would need to disable all the crashed
partitions, and then re-enable them when the C++ DDS process reboots
successfully.  If the disable-enable transitions happen too quickly, could
the controller possibly miss the transition for some partition and not do

> 2. The other option is to change the mapping of partition --> node in the
> ideal state. ( You can do this programmatically in custom modes and in some
> what in auto mode as well). Doing this will send transitions to the node to
> drop the partitions and reassign it to other nodes.

Yes, this seems like the most logical thing.  The Java proxy will probably
need to send a message to the controller to trigger this change in the
ideal states of all crashed partitions.  The messaging API would probably
be useful here.

Another alternative I'm considering is for the Java proxy to add a "fake"
instance for each C++ process that it spawns locally.  The custom
rebalancer (that I'd write inside the controller) would then schedule the
C++ DDS partitions on to these "fake" live instances.  When the C++ process
crashes, the Java proxy would simply disconnect the corresponding fake
instance's manager.  Does this make sense to you?  Or do you have any other

Thanks again for your thoughtful feedback!

> Thanks,
> Kishore G
> On Fri, Feb 1, 2013 at 5:44 PM, Abhishek Rai <abhishekrai@gmail.com>wrote:
>> Hi Helix users!
>> I'm a Helix newbie and need some advice about a use case.  I'm using
>> Helix to manage a storage system which fits the description of a DDS
>> ("distributed data service" as defined in the Helix SOCC paper).  Each
>> participant hosts a bunch of partitions of a resource, as assigned by the
>> controller.  The set of partitions assigned to a participant changes
>> dynamically as the controller rebalances partitions, nodes join or leave,
>> etc.
>> Additionally, I need the ability for a participant to "drop" a subset of
>> partitions currently assigned to it.  When a partition is dropped by a
>> participant, Helix would remove the partition from the current state of the
>> instance, update the external view, and make the partition available for
>> rebalancing by the controller.  Does the Java API provide a way of
>> accomplishing this?  If not, are there any workarounds?  Or, was there a
>> design rationale to disallow such actions from the participant?
>> Thanks,
>> Abhishek

View raw message