Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2DFC017F9A for ; Fri, 28 Aug 2015 00:51:50 +0000 (UTC) Received: (qmail 58409 invoked by uid 500); 28 Aug 2015 00:51:49 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 58252 invoked by uid 500); 28 Aug 2015 00:51:49 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 58065 invoked by uid 99); 28 Aug 2015 00:51:49 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2015 00:51:49 +0000 Date: Fri, 28 Aug 2015 00:51:49 +0000 (UTC) From: "Sergey Shelukhin (JIRA)" To: dev@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Sergey Shelukhin created HIVE-11675: --------------------------------------- Summary: make use of file footer PPD API in ETL strategy or separate strategy Key: HIVE-11675 URL: https://issues.apache.org/jira/browse/HIVE-11675 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Need to take a look at the best flow. It won't be much different if we do filtering metastore call for each partition. So perhaps we'd need the custom sync point/batching after all. Or we can make it opportunistic and not fetch any footers unless it can be pushed down to metastore or fetched from local cache, that way the only slow threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)