Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 75E8710115 for ; Thu, 4 Dec 2014 21:52:27 +0000 (UTC) Received: (qmail 43103 invoked by uid 500); 4 Dec 2014 21:52:20 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 42980 invoked by uid 500); 4 Dec 2014 21:52:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 42969 invoked by uid 99); 4 Dec 2014 21:52:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Dec 2014 21:52:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rchiang@cloudera.com designates 209.85.213.174 as permitted sender) Received: from [209.85.213.174] (HELO mail-ig0-f174.google.com) (209.85.213.174) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Dec 2014 21:52:15 +0000 Received: by mail-ig0-f174.google.com with SMTP id hn15so19485536igb.13 for ; Thu, 04 Dec 2014 13:50:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=TlRSZ//Pm5Oz7qF7kuSwVosns9mq3IujKdh2PGkERUg=; b=hRknwXeOclTwEV48jfJQ7jwIOUIJsP+fz3w2IONCPDiZOlzBtaDneWZl1lVOgngiTL g+tZ18fV9rzn1cHF7bRa/QdupzCA0J4vj1Mo5MDDWNEoNAYDMwqtUh1ucQMrUhWKbFpV PkNzcqg2P0vrFUX0ucAEYozlpMnWcV/2Vzp27P10uL8ds2iNkDp+TBAsYKsncQajdLDW fLOi6cqHHL6w2WSJq6j1RYI4N09EbtOAZbkzpJXXvkvKbnY5qxhB3DZDGe4LlMgOvxE/ vLWTjBIAkaCS8fJiv67ErAF1YlGrrWHzujntpUtypGH1StBWZ2v3BfCwRInUA8HUoXEf tCMg== X-Gm-Message-State: ALoCoQnloPs6QNuXr1djbtZ6gwhH0hW+0kLPN/3zPaBjJAzdz4u0Z9LMp40E70LzOIkkBE8NiuZ8 MIME-Version: 1.0 X-Received: by 10.50.110.1 with SMTP id hw1mr349084igb.13.1417729825289; Thu, 04 Dec 2014 13:50:25 -0800 (PST) Received: by 10.107.36.137 with HTTP; Thu, 4 Dec 2014 13:50:25 -0800 (PST) In-Reply-To: <4DDB6FBC-8B36-4F08-895E-A195F4D95E0A@gmail.com> References: <1B699136-87C4-4E37-B752-9F707A9422E8@gmail.com> <29C5FDF4-3A09-47BF-8536-7BF0FD1C6F20@gmail.com> <383ECC5E-0010-4B23-9DE6-0DD3D7916971@gmail.com> <4DDB6FBC-8B36-4F08-895E-A195F4D95E0A@gmail.com> Date: Thu, 4 Dec 2014 13:50:25 -0800 Message-ID: Subject: Re: Question about the QJM HA namenode From: Ray Chiang To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e012946509fd07205096af2f0 X-Virus-Checked: Checked by ClamAV on apache.org --089e012946509fd07205096af2f0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable It looks like that's tied to the ipc.client.connect.* properties. You can adjust retries & timeout values to something shorter and see if that works for you. Offhand, I'm not certain if that will affect other services besides HDFS. -Ray On Wed, Dec 3, 2014 at 2:51 AM, mail list wrote: > hadoop-2.3.0-cdh5.1.0 > > hi, i move QJM from the l-hbase1.dba.dev.cn0 to another machine, and the > down time reduced to > 5 mins, and the log on the l-hbase2.dba.dev.cn0 like below: > > {log} > 2014-12-03 15:55:51,306 INFO > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 197 edits > starting from txid 6599 > 2014-12-03 15:55:51,306 INFO > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Marking al= l > datandoes as stale > 2014-12-03 15:55:51,307 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Reprocessing > replication and invalidation queues > 2014-12-03 15:55:51,307 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing > replication queues > 2014-12-03 15:55:51,307 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Will take over writi= ng > edit logs at txnid 6797 > 2014-12-03 15:55:51,313 INFO > org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at > 6797 > 2014-12-03 15:55:51,373 INFO > org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions:= 1 > Total time for transactions(ms): 0 Number of transactions batched in Sync= s: > 0 Number of syncs: 0 SyncTimes(ms): 0 9 > 2014-12-03 15:55:51,385 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Starting CacheReplicationMonitor with interval 30000 milliseconds > 2014-12-03 15:55:51,385 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning because of pending operations > 2014-12-03 15:55:51,678 INFO org.apache.hadoop.fs.TrashPolicyDefault: > Namenode trash configuration: Deletion interval =3D 1440 minutes, Emptier > interval =3D 0 minutes. > 2014-12-03 15:55:51,679 INFO org.apache.hadoop.fs.TrashPolicyDefault: The > configured checkpoint interval is 0 minutes. Using an interval of 1440 > minutes that is used for deletion instead > 2014-12-03 15:55:51,693 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number = of > blocks =3D 179 > 2014-12-03 15:55:51,693 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of > invalid blocks =3D 0 > 2014-12-03 15:55:51,693 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of > under-replicated blocks =3D 0 > 2014-12-03 15:55:51,693 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of > over-replicated blocks =3D 0 > 2014-12-03 15:55:51,693 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of > blocks being written =3D 4 > 2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.StateChange: STATE* > Replication Queue initialization scan for invalid, over- and > under-replicated blocks completed in 386 msec > 2014-12-03 15:55:51,693 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 308 millisecond(s). > 2014-12-03 15:56:21,385 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30000 milliseconds > 2014-12-03 15:56:21,386 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-12-03 15:56:51,386 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-12-03 15:56:51,386 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-12-03 15:57:21,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30000 milliseconds > 2014-12-03 15:57:21,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). > 2014-12-03 15:57:51,386 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30000 milliseconds > 2014-12-03 15:57:51,386 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-12-03 15:58:21,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30000 milliseconds > 2014-12-03 15:58:21,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). > 2014-12-03 15:58:51,386 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30000 milliseconds > 2014-12-03 15:58:51,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-12-03 15:59:21,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-12-03 15:59:21,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-12-03 15:59:51,387 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30000 milliseconds > 2014-12-03 15:59:51,388 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-12-03 16:00:14,295 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* > allocateBlock: caught retry for allocation of a new block in > /hbase/testnn/WALs/l-hbase3.dba.dev.cn0.qunar.com,60020,1417585992012/ > l-hbase3.dba.dev.cn0.qunar.com%2C60020%2C1417585992012.1417593301483. > Returning previously allocated block > blk_1073743458_2634{blockUCState=3DUNDER_CONSTRUCTION, primaryNodeIndex= =3D-1, > replicas=3D[]} > {log} > > > It seems the from 15:55:51 to 16:00:14 , all is > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor, > what is hadoop doing? how can i reduce the time cause 5 mins is too long! > > > > On Dec 3, 2014, at 16:31, Harsh J wrote: > > > What is your Hadoop version? > > > > On Wed, Dec 3, 2014 at 12:55 PM, mail list > wrote: > >> hi all, > >> > >> Attach log again! > >> > >> The failover happened at about time: 2014-12-03 12:01: > >> > >> > >> > >> > >> > >> On Dec 3, 2014, at 14:55, mail list wrote: > >> > >>> Sorry forget the log, the failover time at about 2014-12-03 12:01: > >>> > >>> > >>> On Dec 3, 2014, at 14:48, mail list wrote: > >>> > >>>> Hi all, > >>>> > >>>> I deploy the hadoop with 3 machines: > >>>> > >>>> l-hbase1.dba.dev.cn0 (namenode active and QJM) > >>>> l-hbase2.dba.dev.cn0 (namenode standby and datanode and QJM) > >>>> l-hbase3.dba.dev.cn0 (datanode and QJM) > >>>> > >>>> Above the hadoop, i deploy a hbase: > >>>> l-hbase1.dba.dev.cn0 (HMaster active) > >>>> l-hbase2.dba.dev.cn0 (HMaster standby) > >>>> l-hbase3.dba.dev.cn0 (RegionServer) > >>>> > >>>> > >>>> I write a program which put data into hbase one row every seconds in > a loop. > >>>> Then I use iptables to simulate l-hbase1.dba.dev.cn0 offline=EF=BC= =8Cand > after that , the program hang and can not > >>>> write to hbase. After about 15 mins, the program can write again. > >>>> > >>>> The time 15mins for the HA failover is too long for me! > >>>> And I=E2=80=99ve no idea about the reason. > >>>> > >>>> Then I check the l-hbase2.dba.dev.cn0 namenode logs, and find many > retry like below: > >>>> {code} > >>>> 2014-12-03 12:13:35,165 INFO org.apache.hadoop.ipc.Client: Retrying > connect to server: l-hbase1.dba.dev.cn0/10.86.36.217:8485. Already tried > 1 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sleepTime=3D1000 > MILLISECONDS) > >>>> {code} > >>>> > >>>> I have the QJM on l-hbase1.dba.dev.cn0, does it matter? > >>>> > >>>> I am a newbie, Any idea will be appreciated!! > >>> > >> > >> > > > > > > > > -- > > Harsh J > > --089e012946509fd07205096af2f0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
It looks like that's tied to the ipc.client.connect.* = properties.=C2=A0 You can adjust retries & timeout values to something = shorter and see if that works for you.

Offhand, I'm = not certain if that will affect other services besides HDFS.

=
-Ray


On Wed, Dec 3, 2014 at 2:51 AM, mail list <= louis.hust.ml@gmail.com> wrote:
hadoop-2.3.0-cdh5.1.0

hi, i move QJM from the=C2=A0 l-hbase1.dba.dev.cn0 to another machine, and = the down time reduced to
5 mins, and the log on the l-hbase2.dba.dev.cn0 like below:

{log}
2014-12-03 15:55:51,306 INFO org.apache.hadoop.hdfs.server.namenode.ha.Edit= LogTailer: Loaded 197 edits starting from txid 6599
2014-12-03 15:55:51,306 INFO org.apache.hadoop.hdfs.server.blockmanagement.= DatanodeManager: Marking all datandoes as stale
2014-12-03 15:55:51,307 INFO org.apache.hadoop.hdfs.server.namenode.FSNames= ystem: Reprocessing replication and invalidation queues
2014-12-03 15:55:51,307 INFO org.apache.hadoop.hdfs.server.namenode.FSNames= ystem: initializing replication queues
2014-12-03 15:55:51,307 INFO org.apache.hadoop.hdfs.server.namenode.FSNames= ystem: Will take over writing edit logs at txnid 6797
2014-12-03 15:55:51,313 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Starting log segment at 6797
2014-12-03 15:55:51,373 INFO org.apache.hadoop.hdfs.server.namenode.FSEditL= og: Number of transactions: 1 Total time for transactions(ms): 0 Number of = transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 9
2014-12-03 15:55:51,385 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Starting CacheReplicationMonitor with interval 300= 00 milliseconds
2014-12-03 15:55:51,385 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning because of pending operations
2014-12-03 15:55:51,678 INFO org.apache.hadoop.fs.TrashPolicyDefault: Namen= ode trash configuration: Deletion interval =3D 1440 minutes, Emptier interv= al =3D 0 minutes.
2014-12-03 15:55:51,679 INFO org.apache.hadoop.fs.TrashPolicyDefault: The c= onfigured checkpoint interval is 0 minutes. Using an interval of 1440 minut= es that is used for deletion instead
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.server.blockmanagement.= BlockManager: Total number of blocks=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =3D 179
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.server.blockmanagement.= BlockManager: Number of invalid blocks=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =3D 0
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.server.blockmanagement.= BlockManager: Number of under-replicated blocks =3D 0
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.server.blockmanagement.= BlockManager: Number of=C2=A0 over-replicated blocks =3D 0
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.server.blockmanagement.= BlockManager: Number of blocks being written=C2=A0 =C2=A0 =3D 4
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.StateChange: STATE* Rep= lication Queue initialization scan for invalid, over- and under-replicated = blocks completed in 386 msec
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 308 milli= second(s).
2014-12-03 15:56:21,385 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30000 milliseconds
2014-12-03 15:56:21,386 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millise= cond(s).
2014-12-03 15:56:51,386 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30001 milliseconds
2014-12-03 15:56:51,386 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millise= cond(s).
2014-12-03 15:57:21,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30000 milliseconds
2014-12-03 15:57:21,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millise= cond(s).
2014-12-03 15:57:51,386 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30000 milliseconds
2014-12-03 15:57:51,386 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millise= cond(s).
2014-12-03 15:58:21,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30000 milliseconds
2014-12-03 15:58:21,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millise= cond(s).
2014-12-03 15:58:51,386 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30000 milliseconds
2014-12-03 15:58:51,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millise= cond(s).
2014-12-03 15:59:21,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30001 milliseconds
2014-12-03 15:59:21,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millise= cond(s).
2014-12-03 15:59:51,387 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Rescanning after 30000 milliseconds
2014-12-03 15:59:51,388 INFO org.apache.hadoop.hdfs.server.blockmanagement.= CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millise= cond(s).
2014-12-03 16:00:14,295 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* all= ocateBlock: caught retry for allocation of a new block in /hbase/testnn/WAL= s/l-hba= se3.dba.dev.cn0.qunar.com,60020,1417585992012/l-hbase3.dba.dev.cn0.qunar.com%2C60020%2C1417585992012.1417593301483. Returning previously allocated bl= ock blk_1073743458_2634{blockUCState=3DUNDER_CONSTRUCTION, primaryNodeIndex= =3D-1, replicas=3D[]}
{log}


It seems the from 15:55:51 to 16:00:14 , all is org.apache.hadoop.hdfs.serv= er.blockmanagement.CacheReplicationMonitor,
what is hadoop doing? how can i reduce the time cause 5 mins is too long!



On Dec 3, 2014, at 16:31, Harsh J <harsh@cloudera.com> wrote:

> What is your Hadoop version?
>
> On Wed, Dec 3, 2014 at 12:55 PM, mail list <louis.hust.ml@gmail.com> wrote:
>> hi all,
>>
>> Attach log again!
>>
>> The failover happened at about time: 2014-12-03 12:01:
>>
>>
>>
>>
>>
>> On Dec 3, 2014, at 14:55, mail list <louis.hust.ml@gmail.com> wrote:
>>
>>> Sorry forget the log, the failover time at about 2014-12-03 12= :01:
>>>
>>> <hadoop-hadoop-namenode-l-hbase2.dba.dev.cn0.log.tar.gz>=
>>> On Dec 3, 2014, at 14:48, mail list <louis.hust.ml@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I deploy the hadoop with 3 machines:
>>>>
>>>> l-hbase1.dba.dev.cn0 (namenode active and QJM)
>>>> l-hbase2.dba.dev.cn0 (namenode standby and datanode and QJ= M)
>>>> l-hbase3.dba.dev.cn0 (datanode and QJM)
>>>>
>>>> Above the hadoop, i deploy a hbase:
>>>> l-hbase1.dba.dev.cn0 (HMaster active)
>>>> l-hbase2.dba.dev.cn0 (HMaster standby)
>>>> l-hbase3.dba.dev.cn0 (RegionServer)
>>>>
>>>>
>>>> I write a program which put data into hbase one row every = seconds in a loop.
>>>> Then I use iptables to=C2=A0 simulate l-hbase1.dba.dev.cn0= offline=EF=BC=8Cand after that , the program hang and can not
>>>> write to hbase. After about 15 mins, the program can write= again.
>>>>
>>>> The time 15mins for the HA failover is too long for me! >>>> And I=E2=80=99ve no idea about the reason.
>>>>
>>>> Then I check the l-hbase2.dba.dev.cn0 namenode logs, and f= ind many retry like below:
>>>> {code}
>>>> 2014-12-03 12:13:35,165 INFO org.apache.hadoop.ipc.Client:= Retrying connect to server: l-hbase1.dba.dev.cn0/10.86.36.217:8485. Already tried 1 time(s= ); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sle= epTime=3D1000 MILLISECONDS)
>>>> {code}
>>>>
>>>> I have the QJM on l-hbase1.dba.dev.cn0, does it matter? >>>>
>>>> I am a newbie, Any idea will be appreciated!!
>>>
>>
>>
>
>
>
> --
> Harsh J


--089e012946509fd07205096af2f0--