hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <>
Subject [jira] Commented: (HIVE-74) Hive can use CombineFileInputFormat for when the input are many small files
Date Wed, 18 Feb 2009 19:12:02 GMT


Joydeep Sen Sarma commented on HIVE-74:

Is it possible to do this in a way that Hive continues to compile against 0.17/18/19?. I think
this is almost a hard requirement.

One possibility is to have a new version of HiveInputSplit that only compiles against 0.20
- and have this conditionally in the code only for 0.20 and onwards. (for example in
- there's a conditional tag (//[exclude_0_19]) that does some conditional code inclusion).
I am not sure how this was implemented.

But even this is less than ideal. How will we deploy this with 17 (with combinefilesplit and
related patches) (unless we are not using the open source version directly)

> Hive can use CombineFileInputFormat for when the input are many small files
> ---------------------------------------------------------------------------
>                 Key: HIVE-74
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.2.0
>         Attachments: hiveCombineSplit.patch, hiveCombineSplit.patch
> There are cases when the input to a Hive job are thousands of small files. In this case,
there is a mapper for each file. Most of the overhead for spawning all these mappers can be
avoided if Hive used CombineFileInputFormat introduced via HADOOP-4565

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message