hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gunther Hagleitner (JIRA)" <>
Subject [jira] [Commented] (HIVE-4139) MiniDFS shim does not work for hadoop 2
Date Sat, 16 Mar 2013 02:30:12 GMT


Gunther Hagleitner commented on HIVE-4139:

- Was there a reason to add this flag -XX:+CMSClassUnloadingEnabled

When running the unit tests against the hadoop 2 minidfs/minimr stuff, you run out of PermGen
space. This setting (as well as the one to increase the PermGen space for junit fixes the

- Shim for DFS: There's a change in MiniDFS that changes the return type to a sub type of
one of the public methods (getFilesystem). API compatible, but not binary compatible. By creating
the shim, each version will be compiled against the right version of hadoop.

- Shim for MiniMRCluster: There was some really ugly code in QTestUtil, that used exceptions
to distinguish between hadoop 1/2 and initialized the MiniMRcluster accordingly. By creating
shims for these you can just init that stuff differently for each hadoop version (the apis
are the same, but the behavior and configs have changed)

- Yes, these dependencies are needed. The build system won't pick them up otherwise.
> MiniDFS shim does not work for hadoop 2
> ---------------------------------------
>                 Key: HIVE-4139
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Gunther Hagleitner
>            Assignee: Gunther Hagleitner
>         Attachments: HIVE-4139.1.patch, HIVE-4139.2.patch, HIVE-4139.3.patch
> There's an incompatibility between hadoop 1 & 2 wrt to the MiniDfsCluster class.
That causes the hadoop 2 line Minimr tests to fail with a "MethodNotFound" exception.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message