kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Victor <joelsvic...@gmail.com>
Subject Fact tables with complex data types.
Date Mon, 06 Jun 2016 11:30:53 GMT
Hi,

I am using Kylin 1.5.2 with HDP 2.2
Currently my fact table contains multiple columns with type
array<string>. Kylin
won't allow me to sync this table since it has complex datatypes. I don't
need these complex data types in my cube builds but I do require them for
other jobs.

The table is partitioned on date has 2 buckets and is stored in ORC format.

I tried creating a view over it but it seems Kylin doesn't support views as
a fact table.

Another approach that I came up with is moving all the columns with complex
data types from the original table to a separate table and use the original
table as my fact table for building cubes.

Is there any other way to go about this scenario ?

I get the following error when I sync the view:
java.lang.RuntimeException: java.io.IOException:
java.lang.NullPointerException
        at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:86)
        at
org.apache.kylin.source.hive.cardinality.HiveColumnCardinalityJob.run(HiveColumnCardinalityJob.java:89)
        at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
        at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
        at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
        at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
        at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
        at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.NullPointerException
        at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
        at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
        at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
        ... 10 more
Caused by: java.lang.NullPointerException
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:191)
        at
org.apache.hive.hcatalog.mapreduce.FosterStorageHandler.<init>(FosterStorageHandler.java:59)
        at
org.apache.hive.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:417)
        at
org.apache.hive.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:380)
        at
org.apache.hive.hcatalog.mapreduce.InitializeInput.extractPartInfo(InitializeInput.java:158)
        at
org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:137)
        at
org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
        at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)


Thanks,
Joel

Mime
View raw message