kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Nagy <andras.istvan.n...@gmail.com>
Subject Issue when recreating EMR cluster with HBase data on S3
Date Wed, 26 Jun 2019 14:04:28 GMT
Greetings,

I'm testing a setup where HBase is running on AWS EMR and HBase data is
stored on S3. It's working fine so far, but when I terminate the EMR
cluster and recreate it with the same S3 location for HBase, HBase won't
start up properly. Before shutting down, I did execute the
disable_all_tables.sh script to flush HBase state to S3.

Actually the issue is that RegionServers don't start up. Maybe I'm missing
something in the EMR setup and not in Kylin setup, but the exceptions I get
in the RegionServer's log point at Kylin's CubeVisitService coprocessor,
which is still trying to connect to the old HBase master on the old EMR
cluster's master node and fails with: "coprocessor.CoprocessorHost: The
coprocessor
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService
threw java.net.NoRouteToHostException: No Route to Host from
 ip-172-35-5-11/172.35.5.11 to
ip-172-35-7-125.us-west-2.compute.internal:8020 failed on socket timeout
exception: java.net.NoRouteToHostException: No route to host; "

(Here, ip-172-35-7-125 was the old clusters' master node.)

Does anyone have any idea what I'm doing wrong here?
The HBase master node's address seems to be cached somewhere, and when
starting up HBase on the new cluster with the same S3 location for HFiles,
this old address is used still.
Is there anything specific I have missed to get this scenario to work
properly?

This is the full stacktrace:

2019-06-26 12:33:53,352 ERROR [RS_OPEN_REGION-ip-172-35-5-11:16020-1]
coprocessor.CoprocessorHost: The coprocessor
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService
threw java.net.NoRouteToHostException: No Route to Host from
 ip-172-35-5-11/172.35.5.11 to
ip-172-35-7-125.us-west-2.compute.internal:8020 failed on socket timeout
exception: java.net.NoRouteToHostException: No route to host; For more
details see:  http://wiki.apache.org/hadoop/NoRouteToHost
java.net.NoRouteToHostException: No Route to Host from  ip-172-35-5-11/
172.35.5.11 to ip-172-35-7-125.us-west-2.compute.internal:8020 failed on
socket timeout exception: java.net.NoRouteToHostException: No route to
host; For more details see:  http://wiki.apache.org/hadoop/NoRouteToHost
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:758)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
at org.apache.hadoop.ipc.Client.call(Client.java:1345)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy36.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:796)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
at com.sun.proxy.$Proxy37.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1649)
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1440)
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1437)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1452)
at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1466)
at
org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:264)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:214)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:188)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:376)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:238)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:802)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:710)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6716)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7020)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6992)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6948)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6899)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550)
at org.apache.hadoop.ipc.Client.call(Client.java:1381)
... 43 more

Many thanks,
Andras

Mime
View raw message