hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <>
Subject [jira] Commented: (HIVE-1488) CombineHiveInputFormat for hadoop-19 is broken
Date Mon, 26 Jul 2010 21:15:17 GMT


Joydeep Sen Sarma commented on HIVE-1488:

all for getting rid of stuff.

however  my understanding of the background is a little different. multifileinputformat can
combine stuff inside a single dir - but does not do so based on locality. that was the biggest
difference between CFIF and MFIF. Also - hive doesn't combine stuff across partitions (at
least that has been my observation - would be happy to be corrected). so not sure that difference

but given that no one uses it and the stuff is so obviously broke - i don't understand what
the point of spending time on dead code is. so +1 for deprecating/removing this. (hadoop-19
was also not a particularly popular release - neither FB or Yahoo used it).

> CombineHiveInputFormat for hadoop-19 is broken
> ----------------------------------------------
>                 Key: HIVE-1488
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Joydeep Sen Sarma
>            Assignee: Ning Zhang
> I don't if anyone is using it. After making some recent testing related changes in HIVE-1408,
combine[12].q are no longer working when testing against 19. I have seen them fail earlier
as well and not investigated. Looking at the code, it seems pretty hokey:
> getInputPathsShim():
>       Path[] newPaths = new Path[paths.length];
>       // remove file:                                                               
>       for (int pos = 0; pos < paths.length; pos++) {
>         newPaths[pos] = new Path(paths[pos].toString().substring(5));
>       }
> since we are no longer using 'file:' namespace for test warehouse, this is broke. But
this would be broken against any hdfs instance it would seem(?). Also not clear what we are
trying to do here.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message