hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: CopyTable fails on copying between two secured clusters
Date Thu, 08 Sep 2016 18:02:14 GMT
Is it possible that in your destination cluster zookeeper.znode.parent
points to a different location than /hbase ? If both clusters are under the
same kerberos realm then there is no need to worry about
zookeeper.security.auth_to_local.



--
Cloudera, Inc.


On Thu, Sep 8, 2016 at 10:50 AM, Frank Luo <jluo@merkleinc.com> wrote:

> Thanks Esteban for replying.
>
> The Kerberos realm is shared between the two clusters.
>
>  I searched zookeeper config and couldn't find the rule, so where it is
> set?
>
> Having said that, I looked at parameters passed to getData call, and it
> doesn't look like security related.
>
> PS. I am using hbase 1.1.2.
>
> Here is the log:
>
> com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest,testCopyTable
> Connected to the target VM, address: '127.0.0.1:50669', transport:
> 'socket'
> 0    [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 247  [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to
> load native-hadoop library for your platform... using builtin-java classes
> where applicable
> 1492 [main] INFO  org.apache.hadoop.security.UserGroupInformation  -
> Login successful for user tidsvc_exphbase@PCLC0.MERKLE.LOCAL using keytab
> file \Users\jluo\tidsvc_exphbase.keytab
> 11462 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 164493 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 167720 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 172108 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory
> - The short-circuit local reads feature cannot be used because UNIX Domain
> sockets are not available on Windows.
> 224735 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 225165 [main] INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=hconnection-0x4dd5d6bb connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 225190 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:
> host.name=JLUOWIN7.pclc0.merkle.local
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.version=1.7.0_51
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.vendor=Oracle Corporation
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.home=C:\work\jdk1.7.0_51\jre
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.io.tmpdir=C:\Users\jluo\AppData\Local\Temp\
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.compiler=<NA>
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:
> os.name=Windows 7
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:os.arch=amd64
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:os.version=6.1
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:
> user.name=jluo
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:user.home=C:\Users\jluo
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:user.dir=C:\work\project\c2\trunk\misc\hbase_maintenance
> 225197 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating client
> connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,
> hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> sessionTimeout=120000 watcher=hconnection-0x4dd5d6bb0x0,
> quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> baseZNode=/hbase-secure
> 225442 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 225490 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> 225544 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =
> 0x357067c5442410e, negotiated timeout = 120000
> 231758 [main] INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 231758 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating client
> connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,
> hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> sessionTimeout=120000 watcher=TokenUtil-getAuthToken0x0,
> quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> baseZNode=/hbase-secure
> 231876 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 253405 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> 257914 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =
> 0x357067c54424116, negotiated timeout = 120000
> 374820 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> heard from server in 96993ms for sessionid 0x357067c5442410e, closing
> socket connection and attempting reconnect
> 374923 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x357067c54424116 closed
> 374924 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
> 375053 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 375095 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, initiating session
> 375136 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, sessionid =
> 0x357067c5442410e, negotiated timeout = 120000
> 376538 [main] INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation
> - Closing zookeeper sessionid=0x357067c5442410e
> 376581 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x357067c5442410e closed
> 376581 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
> 376746 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 376749 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
> io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
> 376839 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 376841 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
> io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
> 376843 [main] INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=hconnection-0x613cf3c5 connecting to ZooKeeper
> ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.
> local:2181
> 376843 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating client
> connection, connectString=hqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=hconnection-0x613cf3c50x0, quorum=hqetghdpadm91.pclc0.
> merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, baseZNode=/hbase
> 377008 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 377057 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, initiating session
> 377098 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, sessionid =
> 0x2565c552d296982, negotiated timeout = 40000
> 377140 [main] INFO  org.apache.hadoop.hbase.client.ZooKeeperRegistry  -
> ClusterId read in ZooKeeper is null
> 377141 [main] INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper
> ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.
> local:2181
> 377141 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating client
> connection, connectString=hqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=TokenUtil-getAuthToken0x0, quorum=hqetghdpadm91.pclc0.
> merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, baseZNode=/hbase
> 377189 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 399494 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> 399541 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, sessionid =
> 0x1565c552cfd6b03, negotiated timeout = 40000
> 1036441 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> heard from server in 636835ms for sessionid 0x1565c552cfd6b03, closing
> socket connection and attempting reconnect
> 1036441 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> heard from server in 636900ms for sessionid 0x2565c552d296982, closing
> socket connection and attempting reconnect
> 1236709 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 1236761 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> 1236762 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x1565c552cfd6b03 closed
> 1236762 [main] INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation
> - Closing zookeeper sessionid=0x2565c552d296982
> 1236762 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
> 1236908 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x2565c552d296982 closed
> 1236909 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
>
> java.io.IOException: Failed to get cluster ID
>
>         at org.apache.hadoop.hbase.security.token.TokenUtil.
> getAuthToken(TokenUtil.java:365)
>         at org.apache.hadoop.hbase.security.token.TokenUtil.
> addTokenForJob(TokenUtil.java:327)
>         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initCredentials(TableMapReduceUtil.java:451)
>         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initTableReducerJob(TableMapReduceUtil.java:658)
>         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initTableReducerJob(TableMapReduceUtil.java:591)
>         at org.apache.hadoop.hbase.mapreduce.CopyTable.
> createSubmittableJob(CopyTable.java:168)
>         at org.apache.hadoop.hbase.mapreduce.CopyTable.run(
> CopyTable.java:348)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>         at org.apache.hadoop.hbase.mapreduce.CopyTable.main(
> CopyTable.java:341)
>         at com.merkleinc.cr.hbase_maintenance.tableexport.
> CopyTableTest.export(CopyTableTest.java:44)
>         at com.merkleinc.cr.hbase_maintenance.tableexport.
> CopyTableTest.testCopyTable(CopyTableTest.java:25)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> FrameworkMethod.java:50)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(
> ReflectiveCallable.java:12)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> FrameworkMethod.java:47)
>         at org.junit.internal.runners.statements.InvokeMethod.
> evaluate(InvokeMethod.java:17)
>         at com.intellij.junit4.JUnit4TestRunnerUtil$
> IgnoreIgnoredTestJUnit4ClassRunner.runChild(JUnit4TestRunnerUtil.java:349)
>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(
> BlockJUnit4ClassRunner.java:57)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>         at org.junit.runners.ParentRunner.runChildren(
> ParentRunner.java:288)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>         at org.junit.runners.ParentRunner$2.evaluate(
> ParentRunner.java:268)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>         at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(
> JUnit4IdeaTestRunner.java:69)
>         at com.intellij.rt.execution.junit.JUnitStarter.
> prepareStreamsAndStart(JUnitStarter.java:234)
>         at com.intellij.rt.execution.junit.JUnitStarter.main(
> JUnitStarter.java:74)
>
> Disconnected from the target VM, address: '127.0.0.1:50669', transport:
> 'socket'
>
> Process finished with exit code -1
>
> -----Original Message-----
> From: Esteban Gutierrez [mailto:esteban@cloudera.com]
> Sent: Thursday, September 08, 2016 12:37 PM
> To: user@hbase.apache.org
> Subject: Re: CopyTable fails on copying between two secured clusters
>
> Frank,
>
> Do you have a cross realm setup? if yes, do you have ZK to use the proper
> rules via the -Dzookeeper.security.auth_to_local flag? If you could share
> additional logs that would be helpful for us.
>
> Thanks!
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Thu, Sep 8, 2016 at 10:32 AM, Frank Luo <jluo@merkleinc.com> wrote:
>
> > I couldn’t manage to get the CopyTable to work between two secured
> > clusters and hope someone can shed some light.
> >
> > I have a table created on both clusters, and I am running the
> > CopyTable command from the source cluster.
> >
> > The parameters are the following:
> > "--peer.adr=zookeeper1, zookeeper2:2181:/hbase",
> > "--new.name=TargetTable", "SourceTable"
> >
> > I have ensured that the user has all permissions on both tables. When
> > I run it, I got:
> >
> > =====================
> > java.io.IOException: Failed to get cluster ID
> >
> >                at org.apache.hadoop.hbase.security.token.TokenUtil.
> > getAuthToken(TokenUtil.java:365)
> >                at org.apache.hadoop.hbase.security.token.TokenUtil.
> > addTokenForJob(TokenUtil.java:327)
> > =====================
> >
> > Debugging into it, I found that it is trying to get the clusterId from
> > my target cluster using address of zookeeper1/2, and the code below
> > returns
> > null:
> >
> >
> > =====================
> >
> >         ZKUtil.getData(watcher, watcher.clusterIdZNode);
> > =====================
> >
> > Any hints please?
> >
> >
> >
> > Access the Q2 2016 Digital Marketing Report for a fresh set of trends
> > and benchmarks in digital
> > marketing<http://www2.merkleinc.com/l/47252/2016-07-
> > 26/47gt7c>
> >
> > Download our latest report titled “The Case for Change: Exploring the
> > Myths of Customer-Centric Transformation”
> > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c>
> >
> > This email and any attachments transmitted with it are intended for
> > use by the intended recipient(s) only. If you have received this email
> > in error, please notify the sender immediately and then delete it. If
> > you are not the intended recipient, you must not keep, use, disclose,
> > copy or distribute this email without the author’s prior permission.
> > We take precautions to minimize the risk of transmitting software
> > viruses, but we advise you to perform your own virus checks on any
> > attachment to this message. We cannot accept liability for any loss or
> > damage caused by software viruses. The information contained in this
> > communication may be confidential and may be subject to the
> attorney-client privilege.
> >
> Access the Q2 2016 Digital Marketing Report for a fresh set of trends and
> benchmarks in digital marketing<http://www2.merkleinc.com/l/47252/2016-07-
> 26/47gt7c>
>
> Download our latest report titled “The Case for Change: Exploring the
> Myths of Customer-Centric Transformation” <http://www2.merkleinc.com/l/
> 47252/2016-08-04/4b9p7c>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message