kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3035) How to use Kylin on EMR with S3 as hbase storage
Date Mon, 13 Nov 2017 11:42:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249433#comment-16249433
] 

Shawn Wang commented on KYLIN-3035:
-----------------------------------

Hi, I tried the steps.

My kylin.properties:
{noformat}
kylin.env.hdfs-working-dir=s3://kylin-emrfs/kylin-working-dir
kylin.server.mode=query
kylin.storage.hbase.cluster-fs=s3://kylin-emrfs
{noformat}

And in my hbase-site.xml:
{noformat}
<property>
    <name>hbase.rootdir</name>
    <value>s3://kylin-emrfs/hbase-production</value>
</property>
{noformat}

Redeploy coprocessor gives error:
{noformat}
[hadoop@ip-172-31-1-10 apache-kylin-2.2.0-bin]$ $KYLIN_HOME/bin/kylin.sh org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI
default all
Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/kylin/apache-kylin-2.2.0-bin
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Retrieving Spark dependency...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/kylin/apache-kylin-2.2.0-bin/tool/kylin-tool-2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/kylin/apache-kylin-2.2.0-bin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-11-13 11:30:13,085 INFO  [main] common.KylinConfig:315 : Loading kylin-defaults.properties
from file:/usr/local/kylin/apache-kylin-2.2.0-bin/tool/kylin-tool-2.2.0.jar!/kylin-defaults.properties
2017-11-13 11:30:13,123 DEBUG [main] common.KylinConfig:274 : KYLIN_CONF property was not
set, will seek KYLIN_HOME env variable
2017-11-13 11:30:13,129 INFO  [main] common.KylinConfig:95 : Initialized a new KylinConfig
from getInstanceFromEnv : 1285524499
2017-11-13 11:30:17,375 INFO  [main] hbase.HBaseConnection:230 : connection is null or closed,
creating a new one
2017-11-13 11:30:17,504 INFO  [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x74170687
connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:30:17,512 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:zookeeper.version=3.4.10-0338e6621a7a828d8224b10e83a4d72395c41c29,
built on 04/14/2017 23:18 GMT
2017-11-13 11:30:17,512 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:host.name=ip-172-31-1-10.cn-north-1.compute.internal
2017-11-13 11:30:17,512 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.version=1.8.0_141
2017-11-13 11:30:17,512 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.vendor=Oracle
Corporation
2017-11-13 11:30:17,513 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.32.amzn1.x86_64/jre
2017-11-13 11:30:17,513 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.class.path=/
 ......
2017-11-13 11:30:17,516 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native
2017-11-13 11:30:17,516 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.io.tmpdir=/tmp
2017-11-13 11:30:17,517 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.compiler=<NA>
2017-11-13 11:30:17,517 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:os.name=Linux
2017-11-13 11:30:17,517 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:os.arch=amd64
2017-11-13 11:30:17,518 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:os.version=4.4.35-33.55.amzn1.x86_64
2017-11-13 11:30:17,519 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:user.name=hadoop
2017-11-13 11:30:17,519 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:user.home=/home/hadoop
2017-11-13 11:30:17,519 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:user.dir=/usr/local/kylin/apache-kylin-2.2.0-bin
2017-11-13 11:30:17,520 INFO  [main] zookeeper.ZooKeeper:438 : Initiating client connection,
connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@3e3315d9
2017-11-13 11:30:17,544 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1032 : Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181.
Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:30:17,553 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:876 : Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181,
initiating session
2017-11-13 11:30:17,561 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181,
sessionid = 0x15fb4173c1006a0, negotiated timeout = 40000
2017-11-13 11:30:17,713 INFO  [main] util.DeployCoprocessorCLI:100 : Identify coprocessor
jar /usr/local/kylin/apache-kylin-2.2.0-bin/lib/kylin-coprocessor-2.2.0.jar
2017-11-13 11:30:17,718 INFO  [main] cube.CubeManager:140 : Initializing CubeManager with
config kylin_metadata@hbase
2017-11-13 11:30:17,720 INFO  [main] persistence.ResourceStore:79 : Using metadata url kylin_metadata@hbase
for resource store
2017-11-13 11:30:17,800 INFO  [main] hbase.HBaseConnection:230 : connection is null or closed,
creating a new one
2017-11-13 11:30:17,801 INFO  [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x7b32b129
connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:30:17,801 INFO  [main] zookeeper.ZooKeeper:438 : Initiating client connection,
connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@439e3cb4
2017-11-13 11:30:17,803 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1032 : Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181.
Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:30:17,803 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:876 : Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181,
initiating session
2017-11-13 11:30:17,805 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181,
sessionid = 0x15fb4173c1006a1, negotiated timeout = 40000
2017-11-13 11:30:18,085 DEBUG [main] hbase.HBaseConnection:279 : HTable 'kylin_metadata' already
exists
2017-11-13 11:30:18,788 INFO  [main] cube.CubeManager:744 : Loading Cube from folder kylin_metadata(key='/cube')@kylin_metadata@hbase
2017-11-13 11:30:18,799 ERROR [main] cube.CubeManager:800 : Error during load cube instance,
skipping : /cube/test_goods_cube.json
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Mon Nov 13 11:30:18 UTC 2017, RpcRetryingCaller{globalStartTime=1510572618788, pause=100,
retries=1}, org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)


	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
	at org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
	at org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
	at org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:273)
	at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:154)
	at org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:765)
	at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:749)
	at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145)
	at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:108)
	at org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.getHTableNames(DeployCoprocessorCLI.java:465)
	at org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:102)
Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:860)
	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:847)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
	... 12 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException):
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521
aborting
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:35372)
	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:856)
	... 14 more
2017-11-13 11:30:18,806 ERROR [main] cube.CubeManager:800 : Error during load cube instance,
skipping : /cube/tt_test.json
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Mon Nov 13 11:30:18 UTC 2017, RpcRetryingCaller{globalStartTime=1510572618801, pause=100,
retries=1}, org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)


	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
	at org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
	at org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
	at org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:273)
	at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:154)
	at org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:765)
	at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:749)
	at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145)
	at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:108)
	at org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.getHTableNames(DeployCoprocessorCLI.java:465)
	at org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:102)
Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:860)
	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:847)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
	... 12 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException):
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521
aborting
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:35372)
	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:856)
	... 14 more
2017-11-13 11:30:18,807 INFO  [main] cube.CubeManager:757 : Loaded 0 cubes, fail on 2 cubes
2017-11-13 11:30:18,809 DEBUG [main] cachesync.Broadcaster:122 : 1 nodes in the cluster: [localhost:7070]
2017-11-13 11:30:18,810 INFO  [main] util.DeployCoprocessorCLI:103 : Identify tables []
2017-11-13 11:30:18,810 INFO  [main] util.DeployCoprocessorCLI:116 : Will execute tables []
2017-11-13 11:30:18,811 INFO  [main] util.DeployCoprocessorCLI:120 : Old coprocessor jar:
[]
2017-11-13 11:30:19,496 INFO  [main] util.DeployCoprocessorCLI:123 : New coprocessor jar:
s3://kylin-emrfs/kylin-working-dir/kylin_metadata/coprocessor/kylin-coprocessor-2.2.0-0.jar
2017-11-13 11:30:19,496 INFO  [main] util.DeployCoprocessorCLI:132 : Processed time: 685
2017-11-13 11:30:19,496 INFO  [main] util.DeployCoprocessorCLI:133 : Processed tables count:
0
2017-11-13 11:30:19,496 INFO  [main] util.DeployCoprocessorCLI:134 : Processed tables: []
2017-11-13 11:30:19,496 INFO  [main] util.DeployCoprocessorCLI:135 : Active coprocessor jar:
s3://kylin-emrfs/kylin-working-dir/kylin_metadata/coprocessor/kylin-coprocessor-2.2.0-0.jar
2017-11-13 11:30:19,498 INFO  [Thread-0] client.ConnectionManager$HConnectionImplementation:1718
: Closing zookeeper sessionid=0x15fb4173c1006a0
2017-11-13 11:30:19,499 INFO  [main-EventThread] zookeeper.ClientCnxn:519 : EventThread shut
down for session: 0x15fb4173c1006a0
2017-11-13 11:30:19,499 INFO  [Thread-0] zookeeper.ZooKeeper:684 : Session: 0x15fb4173c1006a0
closed
2017-11-13 11:30:19,500 INFO  [Thread-0] client.ConnectionManager$HConnectionImplementation:2180
: Closing master protocol: MasterService
2017-11-13 11:30:19,501 INFO  [Thread-0] client.ConnectionManager$HConnectionImplementation:1718
: Closing zookeeper sessionid=0x15fb4173c1006a1
2017-11-13 11:30:19,502 INFO  [Thread-0] zookeeper.ZooKeeper:684 : Session: 0x15fb4173c1006a1
closed
2017-11-13 11:30:19,502 INFO  [main-EventThread] zookeeper.ClientCnxn:519 : EventThread shut
down for session: 0x15fb4173c1006a1
{noformat}

hbase hbck says:
{noformat}
2017-11-13 11:31:37,786 INFO  [main] util.HBaseFsck: Checking and fixing region consistency
ERROR: Region { meta => kylin_metadata,,1510301827569.2dcbceee014fe3d3b55d987f54ce9632.,
hdfs => s3://kylin-emrfs/hbase-production/data/default/kylin_metadata/2dcbceee014fe3d3b55d987f54ce9632,
deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => KYLIN_WSFX70K3YE,,1510308663611.c6506cd949d382880483dff35b236bad.,
hdfs => s3://kylin-emrfs/hbase-production/data/default/KYLIN_WSFX70K3YE/c6506cd949d382880483dff35b236bad,
deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => hbase:namespace,,1510301118352.f4593b0b8af80a90788fea3a96c25750.,
hdfs => s3://kylin-emrfs/hbase-production/data/hbase/namespace/f4593b0b8af80a90788fea3a96c25750,
deployed => , replicaId => 0 } not deployed on any region server.
2017-11-13 11:31:38,449 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table KYLIN_WSFX70K3YE
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table hbase:namespace
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table kylin_metadata
2017-11-13 11:31:38,455 INFO  [main] util.HBaseFsck: Computing mapping of all store files
......
Summary:
Table hbase:meta is okay.
    Number of regions: 1
    Deployed on:  ip-172-31-1-10.cn-north-1.compute.internal,16000,1510554962161
Table KYLIN_WSFX70K3YE is okay.
    Number of regions: 0
    Deployed on:
Table hbase:namespace is okay.
    Number of regions: 0
    Deployed on:
Table kylin_metadata is okay.
    Number of regions: 0
    Deployed on:
6 inconsistencies detected.
Status: INCONSISTENT
{noformat}

If you need the complete hbase hbck output, I will post it below.

> How to use Kylin on EMR with S3 as hbase storage
> ------------------------------------------------
>
>                 Key: KYLIN-3035
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3035
>             Project: Kylin
>          Issue Type: Bug
>          Components: Metadata
>    Affects Versions: v2.2.0
>         Environment: EMR 5.5.0, Kylin 2.2.0
>            Reporter: Shawn Wang
>            Assignee: Shaofeng SHI
>
> Can somebody give an example of how to use kylin on EMR with S3 as hbase storage, which
support reuse the previously built cube on new EMR after the original EMR has been terminated.
> My purpose is simple:
> 1. use transient EMR cluster to build cubes
> 2. use a persistent cluster to handle query requests
> Of course, the culsters should share same hbase storage, so I setup the cluster to use
S3 as hbase storage, after 2.2.0 fix the "HFile not written to S3" issue, I have been built
a sample cube successfully, using configurations:
> EMR:
> {noformat}
> [
> 	{
> 		"Classification": "hbase-site",
> 		"Properties": {
> 			"hbase.rootdir": "s3://kylin-emrfs/hbase-production"
> 		}
> 	},
> 	{
> 		"Classification": "hbase",
> 		"Properties": {
> 			"hbase.emr.storageMode": "s3"
> 		}
> 	},
> 	{
> 		"Classification": "emrfs-site",
> 		"Properties": {
> 			"fs.s3.consistent": "true",
> 			"fs.s3.consistent.metadata.tableName": "KylinEmrFSMetadata"
> 		}
> 	}
> ]
> {noformat}
> kylin.propertities:
> {noformat}
> kylin.env.hdfs-working-dir=s3://kylin-emrfs/kylin-working-dir
> kylin.server.mode=all
> {noformat}
> Then I create a new cluster with same EMR configuration and query mode for kylin, kylin
just can't startup with errors:
> {noformat}
> 2017-11-13 07:33:44,415 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:876 : Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181,
initiating session
> 2017-11-13 07:33:44,422 INFO  [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181,
sessionid = 0x15fb4173c100156, negotiated timeout = 40000
> 2017-11-13 07:33:48,380 DEBUG [main] hbase.HBaseConnection:279 : HTable 'kylin_metadata'
already exists
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata
store by url: kylin_metadata@hbase
> 	at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
> 	at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
> 	at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
> 	at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1,
exceptions:
> Mon Nov 13 07:33:48 UTC 2017, RpcRetryingCaller{globalStartTime=1510558428667, pause=100,
retries=1}, java.net.ConnectException: 拒绝连接
> 	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
> 	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
> 	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
> 	at org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
> 	at org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
> 	at org.apache.kylin.storage.hbase.HBaseResourceStore.existsImpl(HBaseResourceStore.java:116)
> 	at org.apache.kylin.common.persistence.ResourceStore.exists(ResourceStore.java:144)
> 	at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:84)
> 	... 3 more
> Caused by: java.net.ConnectException: 拒绝连接
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> 	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
> 	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
> 	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:909)
> 	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> 	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1244)
> 	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> 	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> 	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:35372)
> 	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:856)
> 	at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:847)
> 	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
> 	... 10 more
> 2017-11-13 07:33:48,709 INFO  [Thread-1] client.ConnectionManager$HConnectionImplementation:2180
: Closing master protocol: MasterService
> 2017-11-13 07:33:48,710 INFO  [Thread-1] client.ConnectionManager$HConnectionImplementation:1718
: Closing zookeeper sessionid=0x15fb4173c100156
> 2017-11-13 07:33:48,712 INFO  [Thread-1] zookeeper.ZooKeeper:684 : Session: 0x15fb4173c100156
closed
> 2017-11-13 07:33:48,712 INFO  [main-EventThread] zookeeper.ClientCnxn:519 : EventThread
shut down for session: 0x15fb4173c100156
> ERROR: Unknown error. Please check full log.
> {noformat}
> And if I change the Kylin server mode to all, Kylin can startup, but the page on port
7070 can not be opened, with similar errors as above.
> I am wondering If there are some other configuration I have missed, or I am just in the
wrong way.
> It'll be many thanks if someone can give an complete example to show how to get this
work done!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message