Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <1EFFC0EDF85B044A99A02C925FDA375520224C7F@HQITEXCH07.pclc0.merkle.local>
References: <1EFFC0EDF85B044A99A02C925FDA375520223A10@HQITEXCH07.pclc0.merkle.local>
 <CALPvCiDBV=xY=mB72OvmXcYm989ywsnjnoPe=6MPgcW9rhjmDA@mail.gmail.com>
 <1EFFC0EDF85B044A99A02C925FDA375520223A60@HQITEXCH07.pclc0.merkle.local>
 <CALPvCiC7EUTo3A9cZd5mQvWq1SUXDoLJ-YNN6DEKC2SpSAML1g@mail.gmail.com> <1EFFC0EDF85B044A99A02C925FDA375520224C7F@HQITEXCH07.pclc0.merkle.local>
From: Esteban Gutierrez <esteban@cloudera.com>
Date: Fri, 9 Sep 2016 10:22:55 -0700
Message-ID: <CALPvCiAUXVNxXJQa08vbL1vR3ceG=3QRd_kuOdB3mXqzuVeuCQ@mail.gmail.com>
Subject: Re: CopyTable fails on copying between two secured clusters
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a114022ba937f24053c1666d4
archived-at: Fri, 09 Sep 2016 17:23:21 -0000

--001a114022ba937f24053c1666d4
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Frank,

doesn't looks like the you are pointing the znode base to /hbase-secure,
see the arguments that you provided initially:

"--peer.adr=3Dzookeeper1, zookeeper2:2181:/hbase",
"--new.name=3DTargetTable",
"SourceTable"

if the destination cluster has the base znode under /hbase-secure then you
need to point to the right base znode in --peer.adr, e.g. something
like: --peer.adr=3Dzookeeper1, zookeeper2:2181:/hbase-secure

or is there something different you have as the arguments for CopyTable?

esteban.


--
Cloudera, Inc.


On Fri, Sep 9, 2016 at 9:40 AM, Frank Luo <jluo@merkleinc.com> wrote:

> I think I know the cause now.
>
> The code tries to get "baseZNode" from the config. and the latter is
> obtained from Connection#getConfiguration(). Now we have two connections,
> one from local hbase, the other remote. The local hbase's connection has
> the configuration set perfectly, while the one on the remote connection
> barely has anything, hence not able to get a value of "baseZNode".
>
> So based on this theory, CopyTable will never work if the remote is a
> secured cluster, is that a right assessment? Does anyone have luck to get
> it work?
>
> -----Original Message-----
> From: Frank Luo
> Sent: Thursday, September 08, 2016 6:45 PM
> To: user@hbase.apache.org
> Subject: RE: CopyTable fails on copying between two secured clusters
>
> I don't think they are pointing to different locations. Both of them
> should be /hbase-secure.
>
> However, the debugger shows that ConnectionManager#retrieveClusterId are
> called twice, the first time regards to the source cluster, which works
> fine, and watcher.clusterIdZNode=3D/hbase-secure/hbaseid, and it is corre=
ct.
>
> The second time for the remote cluster, watcher.clusterIdZNode=3D/hbase/h=
baseid,
> which should be incorrect.
>
> What I am suspecting is ZooKeeperWatcher, method setNodeNames. It reads:
>
>   private void setNodeNames(Configuration conf) {
>     baseZNode =3D conf.get(HConstants.ZOOKEEPER_ZNODE_PARENT,
>         HConstants.DEFAULT_ZOOKEEPER_ZNODE_PARENT);
>
> I am not sure the conf is corrected fetched from the remote cluster. If
> not, the default value is given, which is /hbase and incorrect.
>
> By the way, below is the root znodes for zookeepers:
>
> The source cluster:
>  [hbase-secure, hiveserver2, hive, hbase-unsecure, templeton-hadoop,
> hadoop-ha, zookeeper]
>
> The target cluster:
> [hbase-secure, hive, hiveserver2, hbase-unsecure, hadoop-ha, zookeeper]
>
> -----Original Message-----
> From: Esteban Gutierrez [mailto:esteban@cloudera.com]
> Sent: Thursday, September 08, 2016 1:02 PM
> To: user@hbase.apache.org
> Subject: Re: CopyTable fails on copying between two secured clusters
>
> Is it possible that in your destination cluster zookeeper.znode.parent
> points to a different location than /hbase ? If both clusters are under t=
he
> same kerberos realm then there is no need to worry about
> zookeeper.security.auth_to_local.
>
>
>
> --
> Cloudera, Inc.
>
>
> On Thu, Sep 8, 2016 at 10:50 AM, Frank Luo <jluo@merkleinc.com> wrote:
>
> > Thanks Esteban for replying.
> >
> > The Kerberos realm is shared between the two clusters.
> >
> >  I searched zookeeper config and couldn't find the rule, so where it
> > is set?
> >
> > Having said that, I looked at parameters passed to getData call, and
> > it doesn't look like security related.
> >
> > PS. I am using hbase 1.1.2.
> >
> > Here is the log:
> >
> > com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest,testCopyT
> > able Connected to the target VM, address: '127.0.0.1:50669',
> > transport:
> > 'socket'
> > 0    [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> > hbase.regionserver.global.memstore.upperLimit is deprecated by
> > hbase.regionserver.global.memstore.size
> > 247  [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to
> > load native-hadoop library for your platform... using builtin-java
> > classes where applicable
> > 1492 [main] INFO  org.apache.hadoop.security.UserGroupInformation  -
> > Login successful for user tidsvc_exphbase@PCLC0.MERKLE.LOCAL using
> > keytab file \Users\jluo\tidsvc_exphbase.keytab
> > 11462 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> > - hbase.regionserver.global.memstore.upperLimit is deprecated by
> > hbase.regionserver.global.memstore.size
> > 164493 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> > - hbase.regionserver.global.memstore.upperLimit is deprecated by
> > hbase.regionserver.global.memstore.size
> > 167720 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> > - hbase.regionserver.global.memstore.upperLimit is deprecated by
> > hbase.regionserver.global.memstore.size
> > 172108 [main] WARN
> > org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory
> > - The short-circuit local reads feature cannot be used because UNIX
> > Domain sockets are not available on Windows.
> > 224735 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> > - hbase.regionserver.global.memstore.upperLimit is deprecated by
> > hbase.regionserver.global.memstore.size
> > 225165 [main] INFO
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> > - Process identifier=3Dhconnection-0x4dd5d6bb connecting to ZooKeeper
> > ensemble=3Dhqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> > 225190 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:zookeeper.version=3D3.4.6-1569965, built on 02/20/2014 09:0=
9
> > GMT
> > 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environmen=
t:
> > host.name=3DJLUOWIN7.pclc0.merkle.local
> > 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.version=3D1.7.0_51
> > 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.vendor=3DOracle Corporation
> > 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.home=3DC:\work\jdk1.7.0_51\jre
> > 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.io.tmpdir=3DC:\Users\jluo\AppData\Local\Temp\
> > 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.compiler=3D<NA>
> > 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environmen=
t:
> > os.name=3DWindows 7
> > 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:os.arch=3Damd64
> > 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:os.version=3D6.1
> > 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environmen=
t:
> > user.name=3Djluo
> > 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:user.home=3DC:\Users\jluo
> > 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> > environment:user.dir=3DC:\work\project\c2\trunk\misc\hbase_maintenance
> > 225197 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> > client connection, connectString=3Dhqhd02nm01.pclc0.merkle.local:2181,
> > hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> > sessionTimeout=3D120000 watcher=3Dhconnection-0x4dd5d6bb0x0,
> > quorum=3Dhqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> > baseZNode=3D/hbase-secure
> > 225442 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 225490 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Socket connection established to
> > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> > 225544 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =3D
> > 0x357067c5442410e, negotiated timeout =3D 120000
> > 231758 [main] INFO
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> > - Process identifier=3DTokenUtil-getAuthToken connecting to ZooKeeper
> > ensemble=3Dhqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> > 231758 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> > client connection, connectString=3Dhqhd02nm01.pclc0.merkle.local:2181,
> > hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> > sessionTimeout=3D120000 watcher=3DTokenUtil-getAuthToken0x0,
> > quorum=3Dhqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> > baseZNode=3D/hbase-secure
> > 231876 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 253405 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Socket connection established to
> > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> > 257914 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =3D
> > 0x357067c54424116, negotiated timeout =3D 120000
> > 374820 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> > heard from server in 96993ms for sessionid 0x357067c5442410e, closing
> > socket connection and attempting reconnect
> > 374923 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> > 0x357067c54424116 closed
> > 374924 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> > EventThread shut down
> > 375053 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 375095 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Socket connection established to
> > hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, initiating session
> > 375136 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> > server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, sessionid =3D
> > 0x357067c5442410e, negotiated timeout =3D 120000
> > 376538 [main] INFO
> > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa
> > tion
> > - Closing zookeeper sessionid=3D0x357067c5442410e
> > 376581 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> > 0x357067c5442410e closed
> > 376581 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> > EventThread shut down
> > 376746 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> > - hbase.regionserver.global.memstore.upperLimit is deprecated by
> > hbase.regionserver.global.memstore.size
> > 376749 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation
> > - io.bytes.per.checksum is deprecated. Instead, use
> > dfs.bytes-per-checksum
> > 376839 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> > - hbase.regionserver.global.memstore.upperLimit is deprecated by
> > hbase.regionserver.global.memstore.size
> > 376841 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation
> > - io.bytes.per.checksum is deprecated. Instead, use
> > dfs.bytes-per-checksum
> > 376843 [main] INFO
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> > - Process identifier=3Dhconnection-0x613cf3c5 connecting to ZooKeeper
> > ensemble=3Dhqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.
> > local:2181
> > 376843 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> > client connection,
> > connectString=3Dhqetghdpadm91.pclc0.merkle.local:2181,
> > hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=3D120000
> > watcher=3Dhconnection-0x613cf3c50x0, quorum=3Dhqetghdpadm91.pclc0.
> > merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181,
> > baseZNode=3D/hbase
> > 377008 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181. Will not attempt
> > to authenticate using SASL (unknown error)
> > 377057 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Socket connection established to
> > hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, initiating session
> > 377098 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> > server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, sessionid =
=3D
> > 0x2565c552d296982, negotiated timeout =3D 40000
> > 377140 [main] INFO  org.apache.hadoop.hbase.client.ZooKeeperRegistry
> > - ClusterId read in ZooKeeper is null
> > 377141 [main] INFO
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> > - Process identifier=3DTokenUtil-getAuthToken connecting to ZooKeeper
> > ensemble=3Dhqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.
> > local:2181
> > 377141 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> > client connection,
> > connectString=3Dhqetghdpadm91.pclc0.merkle.local:2181,
> > hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=3D120000
> > watcher=3DTokenUtil-getAuthToken0x0, quorum=3Dhqetghdpadm91.pclc0.
> > merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181,
> > baseZNode=3D/hbase
> > 377189 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt
> > to authenticate using SASL (unknown error)
> > 399494 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Socket connection established to
> > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> > 399541 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> > server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, sessionid =
=3D
> > 0x1565c552cfd6b03, negotiated timeout =3D 40000
> > 1036441 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> > heard from server in 636835ms for sessionid 0x1565c552cfd6b03, closing
> > socket connection and attempting reconnect
> > 1036441 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> > heard from server in 636900ms for sessionid 0x2565c552d296982, closing
> > socket connection and attempting reconnect
> > 1236709 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt
> > to authenticate using SASL (unknown error)
> > 1236761 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> > org.apache.zookeeper.ClientCnxn  - Socket connection established to
> > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> > 1236762 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> > 0x1565c552cfd6b03 closed
> > 1236762 [main] INFO
> > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa
> > tion
> > - Closing zookeeper sessionid=3D0x2565c552d296982
> > 1236762 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> > EventThread shut down
> > 1236908 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> > 0x2565c552d296982 closed
> > 1236909 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> > EventThread shut down
> >
> > java.io.IOException: Failed to get cluster ID
> >
> >         at org.apache.hadoop.hbase.security.token.TokenUtil.
> > getAuthToken(TokenUtil.java:365)
> >         at org.apache.hadoop.hbase.security.token.TokenUtil.
> > addTokenForJob(TokenUtil.java:327)
> >         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> > initCredentials(TableMapReduceUtil.java:451)
> >         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> > initTableReducerJob(TableMapReduceUtil.java:658)
> >         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> > initTableReducerJob(TableMapReduceUtil.java:591)
> >         at org.apache.hadoop.hbase.mapreduce.CopyTable.
> > createSubmittableJob(CopyTable.java:168)
> >         at org.apache.hadoop.hbase.mapreduce.CopyTable.run(
> > CopyTable.java:348)
> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >         at org.apache.hadoop.hbase.mapreduce.CopyTable.main(
> > CopyTable.java:341)
> >         at com.merkleinc.cr.hbase_maintenance.tableexport.
> > CopyTableTest.export(CopyTableTest.java:44)
> >         at com.merkleinc.cr.hbase_maintenance.tableexport.
> > CopyTableTest.testCopyTable(CopyTableTest.java:25)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:57)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at
> > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > FrameworkMethod.java:50)
> >         at org.junit.internal.runners.model.ReflectiveCallable.run(
> > ReflectiveCallable.java:12)
> >         at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > FrameworkMethod.java:47)
> >         at org.junit.internal.runners.statements.InvokeMethod.
> > evaluate(InvokeMethod.java:17)
> >         at com.intellij.junit4.JUnit4TestRunnerUtil$
> > IgnoreIgnoredTestJUnit4ClassRunner.runChild(
> JUnit4TestRunnerUtil.java:349)
> >         at org.junit.runners.BlockJUnit4ClassRunner.runChild(
> > BlockJUnit4ClassRunner.java:57)
> >         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> >         at org.junit.runners.ParentRunner$1.schedule(
> ParentRunner.java:71)
> >         at org.junit.runners.ParentRunner.runChildren(
> > ParentRunner.java:288)
> >         at org.junit.runners.ParentRunner.access$000(
> ParentRunner.java:58)
> >         at org.junit.runners.ParentRunner$2.evaluate(
> > ParentRunner.java:268)
> >         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> >         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> >         at
> > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(
> > JUnit4IdeaTestRunner.java:69)
> >         at com.intellij.rt.execution.junit.JUnitStarter.
> > prepareStreamsAndStart(JUnitStarter.java:234)
> >         at com.intellij.rt.execution.junit.JUnitStarter.main(
> > JUnitStarter.java:74)
> >
> > Disconnected from the target VM, address: '127.0.0.1:50669', transport:
> > 'socket'
> >
> > Process finished with exit code -1
> >
> > -----Original Message-----
> > From: Esteban Gutierrez [mailto:esteban@cloudera.com]
> > Sent: Thursday, September 08, 2016 12:37 PM
> > To: user@hbase.apache.org
> > Subject: Re: CopyTable fails on copying between two secured clusters
> >
> > Frank,
> >
> > Do you have a cross realm setup? if yes, do you have ZK to use the
> > proper rules via the -Dzookeeper.security.auth_to_local flag? If you
> > could share additional logs that would be helpful for us.
> >
> > Thanks!
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Thu, Sep 8, 2016 at 10:32 AM, Frank Luo <jluo@merkleinc.com> wrote:
> >
> > > I couldn=E2=80=99t manage to get the CopyTable to work between two se=
cured
> > > clusters and hope someone can shed some light.
> > >
> > > I have a table created on both clusters, and I am running the
> > > CopyTable command from the source cluster.
> > >
> > > The parameters are the following:
> > > "--peer.adr=3Dzookeeper1, zookeeper2:2181:/hbase",
> > > "--new.name=3DTargetTable", "SourceTable"
> > >
> > > I have ensured that the user has all permissions on both tables.
> > > When I run it, I got:
> > >
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > java.io.IOException: Failed to get cluster ID
> > >
> > >                at org.apache.hadoop.hbase.security.token.TokenUtil.
> > > getAuthToken(TokenUtil.java:365)
> > >                at org.apache.hadoop.hbase.security.token.TokenUtil.
> > > addTokenForJob(TokenUtil.java:327)
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > >
> > > Debugging into it, I found that it is trying to get the clusterId
> > > from my target cluster using address of zookeeper1/2, and the code
> > > below returns
> > > null:
> > >
> > >
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > >
> > >         ZKUtil.getData(watcher, watcher.clusterIdZNode);
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > >
> > > Any hints please?
> > >
> > >
> > >
> > > Access the Q2 2016 Digital Marketing Report for a fresh set of
> > > trends and benchmarks in digital
> > > marketing<http://www2.merkleinc.com/l/47252/2016-07-
> > > 26/47gt7c>
> > >
> > > Download our latest report titled =E2=80=9CThe Case for Change: Explo=
ring
> > > the Myths of Customer-Centric Transformation=E2=80=9D
> > > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c>
> > >
> > > This email and any attachments transmitted with it are intended for
> > > use by the intended recipient(s) only. If you have received this
> > > email in error, please notify the sender immediately and then delete
> > > it. If you are not the intended recipient, you must not keep, use,
> > > disclose, copy or distribute this email without the author=E2=80=99s =
prior
> permission.
> > > We take precautions to minimize the risk of transmitting software
> > > viruses, but we advise you to perform your own virus checks on any
> > > attachment to this message. We cannot accept liability for any loss
> > > or damage caused by software viruses. The information contained in
> > > this communication may be confidential and may be subject to the
> > attorney-client privilege.
> > >
> > Access the Q2 2016 Digital Marketing Report for a fresh set of trends
> > and benchmarks in digital
> > marketing<http://www2.merkleinc.com/l/47252/2016-07-
> > 26/47gt7c>
> >
> > Download our latest report titled =E2=80=9CThe Case for Change: Explori=
ng the
> > Myths of Customer-Centric Transformation=E2=80=9D
> > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> This email and
> > any attachments transmitted with it are intended for use by the
> > intended recipient(s) only. If you have received this email in error,
> > please notify the sender immediately and then delete it. If you are
> > not the intended recipient, you must not keep, use, disclose, copy or
> > distribute this email without the author=E2=80=99s prior permission. We=
 take
> > precautions to minimize the risk of transmitting software viruses, but
> > we advise you to perform your own virus checks on any attachment to
> > this message. We cannot accept liability for any loss or damage caused
> > by software viruses. The information contained in this communication
> > may be confidential and may be subject to the attorney-client privilege=
.
> >
> Access the Q2 2016 Digital Marketing Report for a fresh set of trends and
> benchmarks in digital marketing<http://www2.merkleinc.com/l/47252/2016-07=
-
> 26/47gt7c>
>
> Download our latest report titled =E2=80=9CThe Case for Change: Exploring=
 the
> Myths of Customer-Centric Transformation=E2=80=9D <http://www2.merkleinc.=
com/l/
> 47252/2016-08-04/4b9p7c>
> This email and any attachments transmitted with it are intended for use b=
y
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not t=
he
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author=E2=80=99s prior permission. We take precaut=
ions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We canno=
t
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may b=
e
> subject to the attorney-client privilege.
>

--001a114022ba937f24053c1666d4--