carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "suo tong (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CARBONDATA-429) Remove unnecessary file name check in dictionary cache
Date Wed, 14 Dec 2016 07:01:06 GMT

     [ https://issues.apache.org/jira/browse/CARBONDATA-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

suo tong updated CARBONDATA-429:
--------------------------------
    Description: 
1. In dictionary cache, there are currently necessary file name check for each column, which
increase the number of HDFS interactions.
2. And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we call
cache.get to check if the local is valid or not.  The local dictionary cache may be invalid
after another job finished load data. 

  was:
In dictionary cache, there are currently necessary file name check for each column, which
increase the number of HDFS interactions.
And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we call cache.get
to check if the local is valid or not.  The local dictionary cache may be invalid after another
job finished load data. 


> Remove unnecessary file name check in dictionary cache
> ------------------------------------------------------
>
>                 Key: CARBONDATA-429
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-429
>             Project: CarbonData
>          Issue Type: Sub-task
>          Components: core
>    Affects Versions: 0.1.1-incubating
>            Reporter: Jacky Li
>            Assignee: Jacky Li
>             Fix For: 1.0.0-incubating
>
>          Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> 1. In dictionary cache, there are currently necessary file name check for each column,
which increase the number of HDFS interactions.
> 2. And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we
call cache.get to check if the local is valid or not.  The local dictionary cache may be invalid
after another job finished load data. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message