kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Can not build sample cube
Date Sun, 05 Feb 2017 13:38:08 GMT
No, from the output of "hdfs fs -ls" the file does exist on HDFS; In hive's
context its default fs is HDFS, so it works as expected. The behavior of
Kylin is wrong as it goes to "RawLocalFileSystem". Usually this is because
the core-site.xml is wrongly configured or it is absent from Kylin's
classpath I think. A quick workaround can be manually finding the right
core-site.xml and then copy it to KYLIN_HOME/conf, restart and then resume
the job. Please take a try and let us know whether it can solve. (we
haven't tried HDP 2.5)

2017-02-05 21:22 GMT+08:00 磊 王 <biolearning@hotmail.com>:

> Thanks for your reply.
> I see the log shows it firstly moved that file from hdfs to local, and
> then read from local. But the issue is the file to move in hdfs does not
> exist at the step of move, so the local file is not available to read at
> the step of read.
> Is my understanding correct?
>
> 发自我的 iPhone
>
> > 在 2017年2月5日,20:54,ShaoFeng Shi <shaofengshi@apache.org> 写道:
> >
> > From the error it tried to seek the file from local file system, instead
> of
> > HDFS. Please check whethere "fs.DefaultFS" in your environment was set to
> > local file by mistake.
> >
> >
> > 2017-02-04 22:58 GMT+08:00 ? ? <BioLearning@hotmail.com>:
> >
> >> Hi Sir,
> >>
> >> When I built the sample cube (and my own cube), I met the error as
> below.
> >> It seems the issue is at the moving step, because I confirmed there was
> >> ‘hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/
> >> kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0’, but
> Kylin
> >> was trying to move is hdfs://sandbox.hortonworks.
> >> com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-
> >> 1ce44302c394/row_count/.hive-staging_hive_2017-02-04_09-38-
> >> 21_841_1217104325959051054-5/-ext-10000.
> >>
> >> [root@ip-10-9-255-49 ec2-user]# hdfs dfs  -ls
> hdfs://sandbox.hortonworks.
> >> com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-
> >> 1ce44302c394/row_count
> >> Found 1 items
> >> -rwxr-xr-x   1 hive hdfs          3 2017-02-04 09:38 hdfs://
> >> sandbox.hortonworks.com:8020/kylin/kylin_metadata/
> >> kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0
> >>
> >> Environment:
> >> Kylin 1.6.0 + HDP 2.5
> >> I am not sure if HDP 2.5 is a too high version, because I only see HDP
> 2.4
> >> is referred in Kylin doc.
> >>
> >>
> >> Build error:
> >> 2017-02-04 09:38:17,947 INFO  [pool-5-thread-2]
> >> manager.ExecutableManager:292 : job id:6a392cfd-a903-4763-89cf-
> 1ce44302c394-01
> >> from READY to RUNNING
> >> 2017-02-04 09:38:17,947 INFO  [pool-5-thread-2] hive.HiveCmdBuilder:81 :
> >> The statements to execute in beeline:
> >> USE default;
> >> SET hive.exec.compress.output=true;
> >> SET hive.auto.convert.join.noconditionaltask=true;
> >> SET hive.auto.convert.join.noconditionaltask.size=100000000;
> >> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
> >> SET mapreduce.job.split.metainfo.maxsize=-1;
> >>
> >> set hive.exec.compress.output=false;
> >>
> >> set hive.exec.compress.output=false;
> >> INSERT OVERWRITE DIRECTORY '/kylin/kylin_metadata/kylin-
> >> 6a392cfd-a903-4763-89cf-1ce44302c394/row_count' SELECT count(*) FROM
> >> kylin_intermediate_kylin_sales_cube_desc_0eb9faec_6a22_
> >> 4b49_889f_c283d82d72dd;
> >>
> >>
> >> 2017-02-04 09:38:17,947 DEBUG [pool-5-thread-2] hive.HiveCmdBuilder:83 :
> >> The SQL to execute in beeline:
> >>
> >> 2017-02-04 09:38:17,947 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : Compute row count of flat hive table,
> >> cmd:
> >> 2017-02-04 09:38:17,947 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : beeline -n root -u 'jdbc:hive2://
> >> sandbox.hortonworks.com:2181/;serviceDiscoveryMode=
> >> zooKeeper;zooKeeperNamespace=hiveserver2' -f
> >> /root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_
> 3013188987573907903.hql;rm
> >> -f /root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_
> >> 3013188987573907903.hql
> >> 2017-02-04 09:38:19,832 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch
> YARN
> >> applications.
> >> 2017-02-04 09:38:20,342 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : Connecting to jdbc:hive2://sandbox.
> >> hortonworks.com:2181/;serviceDiscoveryMode=
> zooKeeper;zooKeeperNamespace=
> >> hiveserver2
> >> 2017-02-04 09:38:21,629 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : Connected to: Apache Hive (version
> >> 1.2.1000.2.5.0.0-1245)
> >> 2017-02-04 09:38:21,630 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : Driver: Hive JDBC (version
> >> 1.2.1.2.3.2.0-2950)
> >> 2017-02-04 09:38:21,630 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : Transaction isolation:
> >> TRANSACTION_REPEATABLE_READ
> >> 2017-02-04 09:38:21,679 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> USE default;
> >> 2017-02-04 09:38:21,732 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.051 seconds)
> >> 2017-02-04 09:38:21,746 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> SET hive.exec.compress.output=true;
> >> 2017-02-04 09:38:21,757 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.011 seconds)
> >> 2017-02-04 09:38:21,766 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> SET hive.auto.convert.join.
> noconditionaltask=true;
> >> 2017-02-04 09:38:21,768 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.002 seconds)
> >> 2017-02-04 09:38:21,777 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> SET hive.auto.convert.join.
> noconditionaltask.size=
> >> 100000000;
> >> 2017-02-04 09:38:21,779 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.002 seconds)
> >> 2017-02-04 09:38:21,787 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> SET mapreduce.output.fileoutputformat.compress.
> >> type=BLOCK;
> >> 2017-02-04 09:38:21,790 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.003 seconds)
> >> 2017-02-04 09:38:21,796 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> SET mapreduce.job.split.metainfo.maxsize=-1;
> >> 2017-02-04 09:38:21,801 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.005 seconds)
> >> 2017-02-04 09:38:21,804 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/>
> >> 2017-02-04 09:38:21,808 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> set hive.exec.compress.output=false;
> >> 2017-02-04 09:38:21,811 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.003 seconds)
> >> 2017-02-04 09:38:21,812 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/>
> >> 2017-02-04 09:38:21,816 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> set hive.exec.compress.output=false;
> >> 2017-02-04 09:38:21,818 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (0.002 seconds)
> >> 2017-02-04 09:38:21,833 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/> INSERT OVERWRITE DIRECTORY
> >> '/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1c
> >> e44302c394/row_count' SELECT count(*) FROM k
> >> 2017-02-04 09:38:21,840 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : ylin_intermediate_kylin_sales_
> >> cube_desc_0eb9faec_6a22_4b49_889f_c283d82d72dd;
> >> 2017-02-04 09:38:30,452 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Tez session hasn't been
> created
> >> yet. Opening session
> >> 2017-02-04 09:38:30,452 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Dag name: INSERT OVERWRITE
> >> DIRE...49_889f_c283d82d72dd(Stage-1)
> >> 2017-02-04 09:38:30,452 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  :
> >> 2017-02-04 09:38:30,452 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 :
> >> 2017-02-04 09:38:30,452 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Status: Running (Executing on
> >> YARN cluster with App id application_1486198519344_0004)
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 :
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Map 1: -/-    Reducer 2: 0/1
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Map 1: 0/1    Reducer 2: 0/1
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Map 1: 0(+1)/1        Reducer
> >> 2: 0/1
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Map 1: 1/1    Reducer 2: 0/1
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Map 1: 1/1    Reducer 2:
> 0(+1)/1
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Map 1: 1/1    Reducer 2: 1/1
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : INFO  : Moving data to directory
> >> /kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-
> 1ce44302c394/row_count
> >> from hdfs://sandbox.h
> >> ortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd-
> >> a903-4763-89cf-1ce44302c394/row_count/.hive-staging_hive_
> >> 2017-02-04_09-38-21_841_1217104325959051054-5/-ext-10000
> >> 2017-02-04 09:38:30,453 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : No rows affected (8.613 seconds)
> >> 2017-02-04 09:38:30,457 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/>
> >> 2017-02-04 09:38:30,457 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/>
> >> 2017-02-04 09:38:30,458 INFO  [pool-5-thread-2]
> >> execution.AbstractExecutable:36 : Closing: 0: jdbc:hive2://sandbox.
> >> hortonworks.com:2181/;serviceDiscoveryMode=
> zooKeeper;zooKeeperNamespace=
> >> hiveserver2
> >> 2017-02-04 09:38:30,573 ERROR [pool-5-thread-2]
> >> execution.AbstractExecutable:370 : job:6a392cfd-a903-4763-89cf-
> 1ce44302c394-01
> >> execute finished with exception
> >> java.io.FileNotFoundException: File /kylin/kylin_metadata/kylin-
> >> 6a392cfd-a903-4763-89cf-1ce44302c394/row_count does not exist
> >>        at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(
> >> RawLocalFileSystem.java:429)
> >>        at org.apache.hadoop.fs.FileSystem.listStatus(
> >> FileSystem.java:1515)
> >>        at org.apache.hadoop.fs.FileSystem.listStatus(
> >> FileSystem.java:1555)
> >>        at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(
> >> ChecksumFileSystem.java:574)
> >>        at org.apache.kylin.source.hive.HiveMRInput$
> >> RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:338)
> >>        at org.apache.kylin.job.execution.AbstractExecutable.
> >> execute(AbstractExecutable.java:113)
> >>        at org.apache.kylin.job.execution.DefaultChainedExecutable.
> doWork(
> >> DefaultChainedExecutable.java:57)
> >>        at org.apache.kylin.job.execution.AbstractExecutable.
> >> execute(AbstractExecutable.java:113)
> >>        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> >> JobRunner.run(DefaultScheduler.java:136)
> >>
> >>
> >>
> >> Thx
> >> Lei Wang
> >>
> >>
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message