kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Temple Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3430) Global Dictionary Cleanup
Date Thu, 05 Jul 2018 03:20:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533238#comment-16533238
] 

Temple Zhou commented on KYLIN-3430:
------------------------------------

Hi Shaofeng,

I have no objection to the 1) & 2), but the 3).

I do not check about whether a dictionary has reference or not, as I found that even the GD
was being used, there were still redundant GD.

{noformat}
hdfs@t1:~$ hdfs dfs -du -h /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_GLOBAL_DICT/USER_ID/
148  444  /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_GLOBAL_DICT/USER_ID/version_1530759906521
157  471  /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_GLOBAL_DICT/USER_ID/version_1530759913804
166  498  /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_GLOBAL_DICT/USER_ID/version_1530759915097
{noformat}

Besides, I analyzed the contents of GD and found that the newer GD seems to contain everything
in the older GD, so I think only the newest GD should be reserved, even the GD is being used
by active cube. I don't know if I express it clearly.


> Global Dictionary Cleanup
> -------------------------
>
>                 Key: KYLIN-3430
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3430
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Tools, Build and Test
>    Affects Versions: v2.1.0, v2.2.0, v2.3.0, v2.3.1, v2.4.0
>            Reporter: Temple Zhou
>            Assignee: Temple Zhou
>            Priority: Major
>         Attachments: KYLIN-3430.master.001.patch
>
>
> I had run "{{./bin/metastore.sh clean --delete true" to cleanup my Kylin metadata, but,
after that, the Global Dictionary still exists in my HDFS and the size of directory "/kylin_metadata/resources/GlobalDict/dict"
hasn't shrunk.}}
>  
> {{BTW: I'm very sure that there are redundant Global Dictionaries.}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message