kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: A problem in cube building with SPARK
Date Tue, 23 May 2017 09:27:04 GMT
The spark executor doesn't have the kerberos token to access HBase. You can
check this JIRA: https://issues.apache.org/jira/browse/SPARK-12279

There is a solution in the last comment, it worth a try (although I doubt
it).

The best solution is letting Spark executors read the Cube metadata from
files (local or HDFS), instead of from HBase, that will ease the deployment.



2017-05-23 14:57 GMT+08:00 周备华 <zhoubeihua@fosun.com>:

> Hi  all,
>
>          Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when
> I use MapReduce engine. However,
>
> when I try to use spark engine to build cube, it fails at step 7: Build
> Cube with Spark. Here
>
> is the log info:
>
> 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34):
>
> java.lang.IllegalArgumentException: Failed to find metadata store by url:
> kylin_metadata@hbase
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>
>                     at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>
>                     at org.apache.kylin.cube.CubeDescManager.
> reloadAllCubeDesc(CubeDescManager.java:298)
>
>                     at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>
>                     at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>
>                     at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>
>                     at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>
>                     at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>
>                     at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>
>                     at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.
> createInstance(AbstractRowKeyEncoder.java:48)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at scala.collection.Iterator$$
> anon$11.next(Iterator.scala:328)
>
>                     at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>
>                     at org.apache.spark.shuffle.sort.
> SortShuffleWriter.write(SortShuffleWriter.scala:64)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>
>                     at org.apache.spark.scheduler.Task.run(Task.scala:89)
>
>                     at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>
>                     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
>                     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>                     at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>
> Here , we find the spark history log, that is
>
> 17/05/17 17:24:35 WARN ipc.RpcClientImpl: Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
>
> 17/05/17 17:24:35 ERROR ipc.RpcClientImpl: SASL authentication failed. The
> most likely cause is missing or invalid credentials. Consider 'kinit'.
>
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
>
>          at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(
> GssKrb5Client.java:211)
>
>          at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.
> saslConnect(HBaseSaslRpcClient.java:181)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.
> setupSaslConnection(RpcClientImpl.java:617)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.
> access$700(RpcClientImpl.java:162)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.
> run(RpcClientImpl.java:743)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.
> run(RpcClientImpl.java:740)
>
>          at java.security.AccessController.doPrivileged(Native Method)
>
>          at javax.security.auth.Subject.doAs(Subject.java:422)
>
>          at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1628)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.
> setupIOstreams(RpcClientImpl.java:740)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.
> writeRequest(RpcClientImpl.java:906)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.
> tracedWriteRequest(RpcClientImpl.java:873)
>
>          at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(
> RpcClientImpl.java:1242)
>
>          at org.apache.hadoop.hbase.ipc.AbstractRpcClient.
> callBlockingMethod(AbstractRpcClient.java:227)
>
>          at org.apache.hadoop.hbase.ipc.AbstractRpcClient$
> BlockingRpcChannelImplementation.callBlockingMethod(
> AbstractRpcClient.java:336)
>
>          at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
>
>          at org.apache.hadoop.hbase.client.ScannerCallable.
> openScanner(ScannerCallable.java:394)
>
>          at org.apache.hadoop.hbase.client.ScannerCallable.call(
> ScannerCallable.java:203)
>
>          at org.apache.hadoop.hbase.client.ScannerCallable.call(
> ScannerCallable.java:64)
>
>          at org.apache.hadoop.hbase.client.RpcRetryingCaller.
> callWithoutRetries(RpcRetryingCaller.java:200)
>
>          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$
> RetryingRPC.call(ScannerCallableWithReplicas.java:381)
>
>          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$
> RetryingRPC.call(ScannerCallableWithReplicas.java:355)
>
>          at org.apache.hadoop.hbase.client.RpcRetryingCaller.
> callWithRetries(RpcRetryingCaller.java:126)
>
>          at org.apache.hadoop.hbase.client.ResultBoundedCompletionService
> $QueueingFuture.run(ResultBoundedCompletionService.java:80)
>
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>          at java.lang.Thread.run(Thread.java:745)
>
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
>
>          at sun.security.jgss.krb5.Krb5InitCredential.getInstance(
> Krb5InitCredential.java:147)
>
>          at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(
> Krb5MechFactory.java:122)
>
>          at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(
> Krb5MechFactory.java:187)
>
>          at sun.security.jgss.GSSManagerImpl.getMechanismContext(
> GSSManagerImpl.java:224)
>
>          at sun.security.jgss.GSSContextImpl.initSecContext(
> GSSContextImpl.java:212)
>
>          at sun.security.jgss.GSSContextImpl.initSecContext(
> GSSContextImpl.java:179)
>
>          at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(
> GssKrb5Client.java:192)
>
>          ... 26 more
>
>
>
> Our HDFS,HBASE has use Kerberos security.
>
> How can we use kylin2.0 to visit the hbase Kerberos enrivoment.
>
>
>
> Any suggestions would help.
>
>
>
> Thanks
>
>
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message