Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 3 Mar 2015 22:56:04 +0000 (UTC)
From: "Rohini Palaniswamy (JIRA)" <jira@apache.org>
To: dev@hive.apache.org
Message-ID: <JIRA.12779215.1425423318000.69246.1425423364505@Atlassian.JIRA>
In-Reply-To: <JIRA.12779215.1425423318000@Atlassian.JIRA>
References: <JIRA.12779215.1425423318000@Atlassian.JIRA>
 <JIRA.12779215.1425423318637@arcas>
Subject: [jira] [Created] (HIVE-9845) HCatSplit repeats information making
 input split data size huge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Rohini Palaniswamy created HIVE-9845:
----------------------------------------

             Summary: HCatSplit repeats information making input split data size huge
                 Key: HIVE-9845
                 URL: https://issues.apache.org/jira/browse/HIVE-9845
             Project: Hive
          Issue Type: Bug
          Components: HCatalog
            Reporter: Rohini Palaniswamy


Pig on Tez jobs with larger tables hit PIG-4443. Running on HDFS data which has even triple the number of splits(100K+ splits and tasks) does not hit that issue.

{code}
HCatBaseInputFormat.java:
 //Call getSplit on the InputFormat, create an
      //HCatSplit for each underlying split
      //NumSplits is 0 for our purposes
      org.apache.hadoop.mapred.InputSplit[] baseSplits = 
        inputFormat.getSplits(jobConf, 0);

      for(org.apache.hadoop.mapred.InputSplit split : baseSplits) {
        splits.add(new HCatSplit(
            partitionInfo,
            split,allCols));
      }
{code}

Each hcatSplit duplicates partition schema and table schema.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)