kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3666) MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
Date Mon, 19 Nov 2018 14:32:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691769#comment-16691769
] 

ASF GitHub Bot commented on KYLIN-3666:
---------------------------------------

shaofengshi commented on a change in pull request #347: KYLIN-3666 HDFS metadata url not be
recognized
URL: https://github.com/apache/kylin/pull/347#discussion_r234639228
 
 

 ##########
 File path: engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
 ##########
 @@ -494,6 +494,17 @@ public static KylinConfig loadKylinConfigFromHdfs(SerializableConfiguration
conf
     }
 
     public static KylinConfig loadKylinConfigFromHdfs(String uri) {
+        KylinConfig config = loadKylinConfigFromHdfsNotThreadLocal(uri);
+        
+        // This is a bad example where the thread local KylinConfig cannot be auto-closed
due to 
+        // limitation of MR API. It works because MR task runs its own process. Do not copy.
+        @SuppressWarnings("unused")
+        SetAndUnsetThreadLocalConfig shouldAutoClose = KylinConfig.setAndUnsetThreadLocalConfig(config);
+
+        return config;
+    }
+
+    public static KylinConfig loadKylinConfigFromHdfsNotThreadLocal(String uri) {
 
 Review comment:
   The method name is confusing. I think the root cause is the bad sample in AbstractHadoopJob.loadKylinConfigFromHdfs(),
we should fix that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
> ---------------------------------------------------------------------
>
>                 Key: KYLIN-3666
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3666
>             Project: Kylin
>          Issue Type: Bug
>          Components: Job Engine
>    Affects Versions: v2.5.0
>         Environment: cdh5.11.0
>            Reporter: Xi Chen
>            Assignee: Chao Long
>            Priority: Major
>             Fix For: v2.5.3
>
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>  
> {code:java}
> java.lang.IllegalStateException: Metadata uri : kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
> at org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Metadata uri : kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
looks like a file but it's neither a file nor a directory
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
> ... 9 more
> {code}
> The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
> Thus throwing such exception.
> Both MR and Spark engine are affected.
> Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message