kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaofeng SHI (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KYLIN-3368) "/kylin/kylin_metadata/metadata/" has many gargage for spark cubing
Date Fri, 04 May 2018 09:46:00 GMT

     [ https://issues.apache.org/jira/browse/KYLIN-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shaofeng SHI updated KYLIN-3368:
--------------------------------
    Description: 
If using Spark as cube engine, Kylin will dump metadata to HDFS in "/kylin/kylin_metadata/metadata/";
As time goes on, many files were left there.

 

They should be deleted when the job is finished.

 

hadoop fs -ls /kylin/kylin_metadata/metadata
Found 583 items
drwxr-xr-x - root hdfs 0 2018-02-11 16:20 /kylin/kylin_metadata/metadata/007d8b6c-5db1-478f-a373-7fd57231494b
drwxr-xr-x - root hdfs 0 2018-02-21 08:24 /kylin/kylin_metadata/metadata/00f93335-0d85-4c19-953c-5e4660f4d58e
drwxr-xr-x - root hdfs 0 2018-03-21 10:17 /kylin/kylin_metadata/metadata/011d3c7b-e707-42ca-b355-a0037820159b
drwxr-xr-x - root hdfs 0 2018-04-07 00:41 /kylin/kylin_metadata/metadata/0263789f-969a-485f-aa6c-df4394a45df8
drwxr-xr-x - root hdfs 0 2018-02-23 00:24 /kylin/kylin_metadata/metadata/02977825-92cd-475d-8a3f-0f75255a9ba9
drwxr-xr-x - root hdfs 0 2018-02-15 08:24 /kylin/kylin_metadata/metadata/02e058fd-ac6a-451f-99b4-f0b03ad6e3a5
drwxr-xr-x - root hdfs 0 2018-03-15 10:21 /kylin/kylin_metadata/metadata/0337c6c9-aebb-4ae0-8485-9905283fc818
drwxr-xr-x - root hdfs 0 2018-02-23 12:22 /kylin/kylin_metadata/metadata/03d7f34f-10dc-4329-9dde-f646cc040ba9
drwxr-xr-x - root hdfs 0 2018-04-10 12:20 /kylin/kylin_metadata/metadata/03fa7654-7b14-4bda-aed8-64ac9d08081a
drwxr-xr-x - root hdfs 0 2018-03-30 00:23 /kylin/kylin_metadata/metadata/0406e5bb-112f-4c54-bdf8-7b1e34652522
drwxr-xr-x - root hdfs 0 2018-03-24 00:33 /kylin/kylin_metadata/metadata/0426907d-541f-4f23-9db7-9a965f5b6739
drwxr-xr-x - root hdfs 0 2018-02-07 15:03 /kylin/kylin_metadata/metadata/04771302-2090-4793-956d-bf5ba9360d9b
drwxr-xr-x - root hdfs 0 2018-03-30 00:20 /kylin/kylin_metadata/metadata/0482b928-6b09-48d1-909f-68c66f71a827
drwxr-xr-x - root hdfs 0 2018-03-04 16:36 /kylin/kylin_metadata/metadata/04943bbd-8425-4304-b9c8-cc13a55e8ac1
drwxr-xr-x - root hdfs 0 2018-03-05 08:35 /kylin/kylin_metadata/metadata/04ee2c3d-bd07-4e7a-90db-3d87d088ff7d
drwxr-xr-x - root hdfs 0 2018-03-02 16:24 /kylin/kylin_metadata/metadata/05808a9d-2e55-4dc3-8c28-5b662068dda8
drwxr-xr-x - root hdfs 0 2018-03-04 08:35 /kylin/kylin_metadata/metadata/05c63d7c-4ea5-48fc-897d-6db9d78ffaf7
drwxr-xr-x - root hdfs 0 2018-03-01 01:02 /kylin/kylin_metadata/metadata/05ea3568-e85f-452a-b44d-4657145a724a

  was:
If using Spark as cube engine, Kylin will dump metadata to HDFS in "/kylin/kylin_metadata/metadata/";
As time goes on, many files were left there.

 

They should be deleted when the job is finished.


> "/kylin/kylin_metadata/metadata/" has many gargage for spark cubing
> -------------------------------------------------------------------
>
>                 Key: KYLIN-3368
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3368
>             Project: Kylin
>          Issue Type: Bug
>          Components: Spark Engine
>    Affects Versions: v2.2.0, v2.3.0
>            Reporter: Shaofeng SHI
>            Priority: Major
>
> If using Spark as cube engine, Kylin will dump metadata to HDFS in "/kylin/kylin_metadata/metadata/";
As time goes on, many files were left there.
>  
> They should be deleted when the job is finished.
>  
> hadoop fs -ls /kylin/kylin_metadata/metadata
> Found 583 items
> drwxr-xr-x - root hdfs 0 2018-02-11 16:20 /kylin/kylin_metadata/metadata/007d8b6c-5db1-478f-a373-7fd57231494b
> drwxr-xr-x - root hdfs 0 2018-02-21 08:24 /kylin/kylin_metadata/metadata/00f93335-0d85-4c19-953c-5e4660f4d58e
> drwxr-xr-x - root hdfs 0 2018-03-21 10:17 /kylin/kylin_metadata/metadata/011d3c7b-e707-42ca-b355-a0037820159b
> drwxr-xr-x - root hdfs 0 2018-04-07 00:41 /kylin/kylin_metadata/metadata/0263789f-969a-485f-aa6c-df4394a45df8
> drwxr-xr-x - root hdfs 0 2018-02-23 00:24 /kylin/kylin_metadata/metadata/02977825-92cd-475d-8a3f-0f75255a9ba9
> drwxr-xr-x - root hdfs 0 2018-02-15 08:24 /kylin/kylin_metadata/metadata/02e058fd-ac6a-451f-99b4-f0b03ad6e3a5
> drwxr-xr-x - root hdfs 0 2018-03-15 10:21 /kylin/kylin_metadata/metadata/0337c6c9-aebb-4ae0-8485-9905283fc818
> drwxr-xr-x - root hdfs 0 2018-02-23 12:22 /kylin/kylin_metadata/metadata/03d7f34f-10dc-4329-9dde-f646cc040ba9
> drwxr-xr-x - root hdfs 0 2018-04-10 12:20 /kylin/kylin_metadata/metadata/03fa7654-7b14-4bda-aed8-64ac9d08081a
> drwxr-xr-x - root hdfs 0 2018-03-30 00:23 /kylin/kylin_metadata/metadata/0406e5bb-112f-4c54-bdf8-7b1e34652522
> drwxr-xr-x - root hdfs 0 2018-03-24 00:33 /kylin/kylin_metadata/metadata/0426907d-541f-4f23-9db7-9a965f5b6739
> drwxr-xr-x - root hdfs 0 2018-02-07 15:03 /kylin/kylin_metadata/metadata/04771302-2090-4793-956d-bf5ba9360d9b
> drwxr-xr-x - root hdfs 0 2018-03-30 00:20 /kylin/kylin_metadata/metadata/0482b928-6b09-48d1-909f-68c66f71a827
> drwxr-xr-x - root hdfs 0 2018-03-04 16:36 /kylin/kylin_metadata/metadata/04943bbd-8425-4304-b9c8-cc13a55e8ac1
> drwxr-xr-x - root hdfs 0 2018-03-05 08:35 /kylin/kylin_metadata/metadata/04ee2c3d-bd07-4e7a-90db-3d87d088ff7d
> drwxr-xr-x - root hdfs 0 2018-03-02 16:24 /kylin/kylin_metadata/metadata/05808a9d-2e55-4dc3-8c28-5b662068dda8
> drwxr-xr-x - root hdfs 0 2018-03-04 08:35 /kylin/kylin_metadata/metadata/05c63d7c-4ea5-48fc-897d-6db9d78ffaf7
> drwxr-xr-x - root hdfs 0 2018-03-01 01:02 /kylin/kylin_metadata/metadata/05ea3568-e85f-452a-b44d-4657145a724a



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message