Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 59475200BDC for ; Wed, 14 Dec 2016 08:01:10 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 57D6E160B34; Wed, 14 Dec 2016 07:01:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A0995160B13 for ; Wed, 14 Dec 2016 08:01:09 +0100 (CET) Received: (qmail 86669 invoked by uid 500); 14 Dec 2016 07:01:08 -0000 Mailing-List: contact issues-help@carbondata.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.incubator.apache.org Delivered-To: mailing list issues@carbondata.incubator.apache.org Received: (qmail 86660 invoked by uid 99); 14 Dec 2016 07:01:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Dec 2016 07:01:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D028DD0095 for ; Wed, 14 Dec 2016 07:01:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -7.019 X-Spam-Level: X-Spam-Status: No, score=-7.019 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id eZPLOAkFce47 for ; Wed, 14 Dec 2016 07:01:07 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id E05B55F1A0 for ; Wed, 14 Dec 2016 07:01:06 +0000 (UTC) Received: (qmail 86627 invoked by uid 99); 14 Dec 2016 07:01:06 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Dec 2016 07:01:06 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 4CF242C0087 for ; Wed, 14 Dec 2016 07:01:06 +0000 (UTC) Date: Wed, 14 Dec 2016 07:01:06 +0000 (UTC) From: "suo tong (JIRA)" To: issues@carbondata.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CARBONDATA-429) Remove unnecessary file name check in dictionary cache MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 14 Dec 2016 07:01:10 -0000 [ https://issues.apache.org/jira/browse/CARBONDATA-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] suo tong updated CARBONDATA-429: -------------------------------- Description: 1. In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions. 2. And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we call cache.get to check if the local is valid or not. The local dictionary cache may be invalid after another job finished load data. was: In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions. And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we call cache.get to check if the local is valid or not. The local dictionary cache may be invalid after another job finished load data. > Remove unnecessary file name check in dictionary cache > ------------------------------------------------------ > > Key: CARBONDATA-429 > URL: https://issues.apache.org/jira/browse/CARBONDATA-429 > Project: CarbonData > Issue Type: Sub-task > Components: core > Affects Versions: 0.1.1-incubating > Reporter: Jacky Li > Assignee: Jacky Li > Fix For: 1.0.0-incubating > > Time Spent: 3h 20m > Remaining Estimate: 0h > > 1. In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions. > 2. And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we call cache.get to check if the local is valid or not. The local dictionary cache may be invalid after another job finished load data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)