pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
Date Thu, 27 Sep 2012 19:21:08 GMT

    [ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465003#comment-13465003
] 

Dmitriy V. Ryaboy commented on PIG-2885:
----------------------------------------

HBASE-5680 is marked as resolved -- does that unblock us, or are we depending on them to make
a release?
                
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
>                 Key: PIG-2885
>                 URL: https://issues.apache.org/jira/browse/PIG-2885
>             Project: Pig
>          Issue Type: Bug
>         Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>            Priority: Minor
>              Labels: hbase
>         Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping
the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a
jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and
protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> -        HTable table = new HTable(TESTTABLE_2);
> +        HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since
it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
>          // use the estimation
>          Configuration conf = cluster.getConfiguration();
>          HBaseTestingUtility util = new HBaseTestingUtility(conf);
> -        util.startMiniZKCluster();
> +        int clientPort = util.startMiniZKCluster().getClientPort();
>          util.startMiniHBaseCluster(1, 1); 
>      
>          String query = "a = load '/passwd';" + 
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>      
>          pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
>          pc.getConf().setProperty("pig.exec.reducers.max", "10");
> +        pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
>          ConfigurationValidator.validatePigProperties(pc.getProperties());
>          conf = ConfigurationUtil.toConfiguration(pc.getProperties());
>          JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage
fails in hadoop 0.23, and I haven't investigated that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message