Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 49643EA8E for ; Sun, 3 Mar 2013 18:34:36 +0000 (UTC) Received: (qmail 36965 invoked by uid 500); 3 Mar 2013 18:34:36 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 36919 invoked by uid 500); 3 Mar 2013 18:34:36 -0000 Mailing-List: contact user-help@helix.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.incubator.apache.org Delivered-To: mailing list user@helix.incubator.apache.org Received: (qmail 36912 invoked by uid 99); 3 Mar 2013 18:34:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Mar 2013 18:34:36 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mingfang@mac.com designates 17.172.204.239 as permitted sender) Received: from [17.172.204.239] (HELO st11p01mm-asmtp004.mac.com) (17.172.204.239) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Mar 2013 18:34:28 +0000 Received: from [192.168.1.168] (pool-96-246-103-18.nycmny.fios.verizon.net [96.246.103.18]) by st11p01mm-asmtp004.mac.com (Oracle Communications Messaging Server 7u4-24.01(7.0.4.24.0) 64bit (built Jan 3 2012)) with ESMTPSA id <0MJ300K41KWQON70@st11p01mm-asmtp004.mac.com> for user@helix.incubator.apache.org; Sun, 03 Mar 2013 18:34:03 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.9.8327,1.0.431,0.0.0000 definitions=2013-03-03_06:2013-03-01,2013-03-03,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=6.0.2-1212290000 definitions=main-1303030189 From: Ming Fang Content-type: multipart/alternative; boundary="Apple-Mail=_18C32BC7-0F6C-4BF7-876D-07EE15D745BE" Message-id: <893A02E0-A7A2-4061-A6B7-388BBC671607@mac.com> MIME-version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Failure detection time Date: Sun, 03 Mar 2013 13:34:02 -0500 References: To: user@helix.incubator.apache.org In-reply-to: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_18C32BC7-0F6C-4BF7-876D-07EE15D745BE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 I've tried setting zk.session.timeout property from my participants but = I don't think it's working. Looking at org.apache.helix.manager.zk.ZKHelixManager line 155, it seems = the session timeout is made same value as = helixmanager.flappingTimeWindow. That looks like a bug since these two values are for different purposes. As a temporary workaround, this is a hack that works manager =3D = HelixManagerFactory.getZKHelixManager(CLUSTER_NAME, instanceName, = InstanceType.PARTICIPANT, ZK_ADDRESS); { //hack to set sessionTimeout Field sessionTimeout =3D = ZKHelixManager.class.getDeclaredField("_sessionTimeout"); sessionTimeout.setAccessible(true); sessionTimeout.setInt(manager, 1000); } Also on the Zookeeper side I made the tickTime =3D500 and = minSessionTimeout =3D 1000. On Mar 3, 2013, at 1:53 AM, kishore g wrote: > There are two kinds of fail over planned( during software upgrade) = unplanned( node crash etc).=20 >=20 > For planned, you should add a jvm shutdownhook from which will you = invoke helixmanager.disconnect() and then invoke kill . This will = allow Helix to detect the failure immediately like 5-15 milli seconds. >=20 > For unplanned, it is determined by zookeeper session timeout, this is = by default set to 30 seconds. You can change this to be more aggressive = like 5,10 or 15 seconds. Recommended value 15 seconds. You can change = this by setting system property "zk.session.timeout"=3D 15*1000. >=20 > helixmanager.flappingTimeWindow and = helixmanager.maxDisconnectThreshold can be tuned in case you have bad = network situations and excessive GC's. You probably dont need to tune = this, but let me know if you need additional info on this. >=20 > Fail over depends on number of partitions, nodes, resources etc in the = system. For a 1000 partition system with 10 nodes, failover time for = one node might be 200-300 milliseconds.=20 >=20 > Jason has done lot of performance improvements on another branch that = might improve this time further.=20 >=20 > thanks, > Kishore G >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > On Sat, Mar 2, 2013 at 9:53 PM, Ming Fang wrote: > How can I tune the amount of time it takes for detecting a failed = node, e.g. kill -9? > Is it by setting "helixmanager.flappingTimeWindow"? >=20 > What is the fastest possible time for a failover? >=20 --Apple-Mail=_18C32BC7-0F6C-4BF7-876D-07EE15D745BE Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 I've = tried setting zk.session.timeout property from my participants but I = don't think it's working.
Looking at = org.apache.helix.manager.zk.ZKHelixManager line 155, it seems the = session timeout is made same value = as helixmanager.flappingTimeWindow.
That looks like a bug = since these two values are for different = purposes.

As a temporary workaround, this is a = hack that works

      =       manager =3D = HelixManagerFactory.getZKHelixManager(CLUSTER_NAME, instanceName, = InstanceType.PARTICIPANT, ZK_ADDRESS);
      =       {
          =       //hack to set sessionTimeout
  =               Field sessionTimeout =3D = ZKHelixManager.class.getDeclaredField("_sessionTimeout");
 =               = sessionTimeout.setAccessible(true);
      =           sessionTimeout.setInt(manager, = 1000);
            = }

Also on the Zookeeper side I made = the tickTime =3D500 and minSessionTimeout =3D = 1000.

On Mar 3, 2013, at 1:53 AM, kishore = g <g.kishore@gmail.com> = wrote:

There are two kinds of fail over planned( = during software upgrade) unplanned( node crash = etc). 

For planned, you should add a jvm = shutdownhook from which will you invoke helixmanager.disconnect() and = then invoke kill <pid>. This will allow Helix to detect the = failure immediately like 5-15 milli seconds.

For unplanned, it is determined by zookeeper session = timeout, this is by default set to 30 seconds. You can change this to be = more aggressive like 5,10 or 15 seconds. Recommended value 15 seconds. = You can change this by setting system property "zk.session.timeout"=3D = 15*1000.

helixmanager.flappingTimeWindow = and helixmanager.maxDisconnectThreshold can be tuned in case you = have bad network situations and excessive GC's. You = probably dont need to tune this, but let me know if you need additional = info on this.

Fail over depends on number of = partitions, nodes, resources etc in the system.  For a 1000 = partition system with 10 nodes, failover time for one node might be = 200-300 milliseconds. 

Jason has done lot of performance improvements = on another branch that might improve this time further. 

thanks,
Kishore = G








On Sat, Mar 2, 2013 at 9:53 PM, Ming = Fang <mingfang@mac.com> wrote:
How can I tune the amount of time it takes for detecting a failed node, = e.g. kill -9?
Is it by setting "helixmanager.flappingTimeWindow"?

What is the fastest possible time for a = failover?


= --Apple-Mail=_18C32BC7-0F6C-4BF7-876D-07EE15D745BE--