kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Yang <liy...@apache.org>
Subject Re: errors when run StorageCleanupJob
Date Fri, 30 Jun 2017 02:54:20 GMT
A known issue, is fixed on master branch. Thanks for reporting!

On Wed, Jun 28, 2017 at 10:16 AM, zhongyuhai <199526628@qq.com> wrote:

> hello
>
>
> when I run command as follow
>
>
> kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true
>
>
>  and I got the exception msg as follow :
>
>
> 2017-06-28 10:08:17,432 INFO  [main StorageCleanupJob:145]: options: '
> delete=true'
> 2017-06-28 10:08:17,432 INFO  [main StorageCleanupJob:146]: delete option
> value: 'true'
> 2017-06-28 10:08:17,432 INFO  [main StorageCleanupJob:147]: force option
> value: 'null'
> 2017-06-28 10:08:17,759 WARN  [main HiveConf:3683]: HiveConf of name
> hive.multigroupby.singlemar does not exist
> 2017-06-28 10:08:17,872 INFO  [main HiveMetaStore:564]: 0: Opening raw
> store
> with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> 2017-06-28 10:08:17,956 INFO  [main ObjectStore:325]: ObjectStore,
> initialize called
> 2017-06-28 10:08:18,208 INFO  [main Persistence:77]: Property
> datanucleus.fixedDatastore unknown - will be ignored
> 2017-06-28 10:08:18,209 INFO  [main Persistence:77]: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 2017-06-28 10:08:18,209 INFO  [main Persistence:77]: Property
> datanucleus.cache.level2 unknown - will be ignored
> 2017-06-28 10:08:19,617 WARN  [main HiveConf:3683]: HiveConf of name
> hive.multigroupby.singlemar does not exist
> 2017-06-28 10:08:19,619 INFO  [main ObjectStore:429]: Setting MetaStore
> object pin classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,
> Partition,Database,Type,FieldSchema,Order"
> 2017-06-28 10:08:21,953 INFO  [main MetaStoreDirectSql:145]: Using direct
> SQL, underlying DB is MYSQL
> 2017-06-28 10:08:21,957 INFO  [main ObjectStore:308]: Initialized
> ObjectStore
> 2017-06-28 10:08:22,455 INFO  [main HiveMetaStore:651]: Added admin role in
> metastore
> 2017-06-28 10:08:22,459 INFO  [main HiveMetaStore:660]: Added public role
> in
> metastore
> 2017-06-28 10:08:22,532 INFO  [main HiveMetaStore:700]: No user is added in
> admin role, since config is empty
> 2017-06-28 10:08:22,808 INFO  [main HiveMetaStore:734]: 0: get_all_tables:
> db=default
> 2017-06-28 10:08:22,811 INFO  [main audit:295]: ugi=root
> ip=unknown-ip-addr      cmd=get_all_tables: db=default
> 2017-06-28 10:08:22,947 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 0363e422-1be2-4652-89e5-24a727ce98ba, ignore it
> 2017-06-28 10:08:22,954 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 05990f76-b2e7-463c-bd5f-9b6906db4a63, ignore it
> 2017-06-28 10:08:22,958 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 198969b3-dd27-4bd6-b5eb-476d963cc59c, ignore it
> 2017-06-28 10:08:22,964 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 1f97122e-1e38-4007-a8a2-705b8af1521a, ignore it
> 2017-06-28 10:08:22,969 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 39f59489-aaea-43a6-be2e-436447d67dae, ignore it
> 2017-06-28 10:08:22,974 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 42de30ea-82a6-455c-9cbc-2c6df04812cd, ignore it
> 2017-06-28 10:08:22,978 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 6849dbc5-6a48-41e3-b258-b248c77e0a52, ignore it
> 2017-06-28 10:08:22,983 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 6c9aa465-ae82-402f-ba7b-cbbac2adca3f, ignore it
> 2017-06-28 10:08:22,989 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 82994592-ff8b-4863-aa42-dc0bab66d47d, ignore it
> 2017-06-28 10:08:22,994 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 84cc3f86-29fc-438a-be45-5ec46792c568, ignore it
> 2017-06-28 10:08:22,998 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 90219e38-3b2a-4ec8-8d3a-7c86930d5caa, ignore it
> 2017-06-28 10:08:23,003 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID 9dc10d6b-d1b4-4470-adf2-dedbe1f0f90e, ignore it
> 2017-06-28 10:08:23,008 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID b4d99f1d-bc13-45eb-8433-84de238279a1, ignore it
> 2017-06-28 10:08:23,013 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID c0f24c74-44f9-4171-b56e-8056d19e3ae1, ignore it
> 2017-06-28 10:08:23,017 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID ca7a8143-e676-4c9f-8ffe-fdadcc84724a, ignore it
> 2017-06-28 10:08:23,024 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID dd8f4ea5-5c3c-4165-88e1-f06a796ca095, ignore it
> 2017-06-28 10:08:23,029 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID e57934e1-1398-457f-9070-6bc5010f461a, ignore it
> 2017-06-28 10:08:23,032 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID e9a845c5-5aba-4c83-a589-99984b1b07ee, ignore it
> 2017-06-28 10:08:23,037 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID eadcc797-b9ef-439c-891c-9ac912cc7ce6, ignore it
> 2017-06-28 10:08:23,043 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID ef9e5b30-d8fa-498d-ad38-48494a15fb22, ignore it
> 2017-06-28 10:08:23,048 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID f1c5089a-b6e1-4118-b58e-48164833de91, ignore it
> 2017-06-28 10:08:23,052 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID f469d883-d066-4f48-ad4a-098dede4a74b, ignore it
> 2017-06-28 10:08:23,056 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID f4ddb713-be4e-4c30-a24a-645d87847f01, ignore it
> 2017-06-28 10:08:23,060 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID f6579632-3b05-4546-a826-96e761c4e678, ignore it
> 2017-06-28 10:08:23,065 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID f9ae3fd4-9f74-4c59-9b47-d604388e1970, ignore it
> 2017-06-28 10:08:23,069 WARN  [main StorageCleanupJob:285]: Failed to find
> segment ID from job ID fbe1e38d-c4f0-47ad-a93a-7489f5638cdf, ignore it
> 2017-06-28 10:08:23,069 INFO  [main StorageCleanupJob:289]: Working jobIDs:
> [0363e422-1be2-4652-89e5-24a727ce98ba, 05990f76-b2e7-463c-bd5f-
> 9b6906db4a63,
> 198969b3-dd27-4bd6-b5eb-476d963cc59c, 1f97122e-1e38-4007-a8a2-
> 705b8af1521a,
> 39f59489-aaea-43a6-be2e-436447d67dae, 42de30ea-82a6-455c-9cbc-
> 2c6df04812cd,
> 6849dbc5-6a48-41e3-b258-b248c77e0a52, 6c9aa465-ae82-402f-ba7b-
> cbbac2adca3f,
> 82994592-ff8b-4863-aa42-dc0bab66d47d, 84cc3f86-29fc-438a-be45-
> 5ec46792c568,
> 90219e38-3b2a-4ec8-8d3a-7c86930d5caa, 9dc10d6b-d1b4-4470-adf2-
> dedbe1f0f90e,
> b4d99f1d-bc13-45eb-8433-84de238279a1, c0f24c74-44f9-4171-b56e-
> 8056d19e3ae1,
> ca7a8143-e676-4c9f-8ffe-fdadcc84724a, dd8f4ea5-5c3c-4165-88e1-
> f06a796ca095,
> e57934e1-1398-457f-9070-6bc5010f461a, e9a845c5-5aba-4c83-a589-
> 99984b1b07ee,
> eadcc797-b9ef-439c-891c-9ac912cc7ce6, ef9e5b30-d8fa-498d-ad38-
> 48494a15fb22,
> f1c5089a-b6e1-4118-b58e-48164833de91, f469d883-d066-4f48-ad4a-
> 098dede4a74b,
> f4ddb713-be4e-4c30-a24a-645d87847f01, f6579632-3b05-4546-a826-
> 96e761c4e678,
> f9ae3fd4-9f74-4c59-9b47-d604388e1970, fbe1e38d-c4f0-47ad-a93a-
> 7489f5638cdf]
> 2017-06-28 10:08:23,096 INFO  [main StorageCleanupJob:292]: Checking table
> kylin_intermediate_biam_busi_anl_day_kind_p_29fd1441_f7c4_
> 4625_af15_58a39cbdd08a
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.tool.StorageCleanupJob
>         at
> org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:42)
>         at
> org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:394)
> Caused by: java.lang.NullPointerException
>         at
> org.apache.kylin.common.util.ClassUtil.forRenamedClass(ClassUtil.java:84)
>         at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.
> java:76)
>         at
> org.apache.kylin.job.execution.ExecutableManager.
> parseTo(ExecutableManager.java:439)
>         at
> org.apache.kylin.job.execution.ExecutableManager.
> getJob(ExecutableManager.java:129)
>         at
> org.apache.kylin.tool.StorageCleanupJob.getSegmentIdFromJobId(
> StorageCleanupJob.java:375)
>         at
> org.apache.kylin.tool.StorageCleanupJob.isTableInUse(
> StorageCleanupJob.java:382)
>         at
> org.apache.kylin.tool.StorageCleanupJob.cleanUnusedIntermediateHiveTab
> le(StorageCleanupJob.java:311)
>         at
> org.apache.kylin.tool.StorageCleanupJob.execute(
> StorageCleanupJob.java:153)
>         at
> org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
>         ... 1 more
> 2017-06-28 10:08:23,110 INFO  [Thread-0
> ConnectionManager$HConnectionImplementation:2139]: Closing master
> protocol:
> MasterService
> 2017-06-28 10:08:23,142 INFO  [Thread-0
> ConnectionManager$HConnectionImplementation:1710]: Closing zookeeper
> sessionid=0x685caabe66b20097
> 2017-06-28 10:08:23,146 INFO  [Thread-0 ZooKeeper:684]: Session:
> 0x685caabe66b20097 closed
> 2017-06-28 10:08:23,146 INFO  [main-EventThread ClientCnxn:512]:
> EventThread
> shut down
>
>
>
> My software version as follow:
>
>
> hadoop-2.6.0-cdh5.4.0
> hbase-1.2.3-bin
> apache-hive-2.1.0-bin
> kylin.version:2.0.0
> commit:375fd807c281d8c5deff0620747c806be2019782;
> os.name:Linux
> os.arch:amd64
> os.version:3.10.0-229.el7.x86_64
> java.version:1.8.0_102
> java.vendor:Oracle Corporation
>
>
>     Thank you !
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/errors-when-run-StorageCleanupJob-tp8321.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message