hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
Date Fri, 05 Apr 2013 21:39:16 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624087#comment-13624087
] 

Devaraj Das commented on HBASE-8258:
------------------------------------

{quote}
When it comes out we deal with it.
The isolation of the particular relevant yarn configs demonstrate that the vmem stuff is red
herring and unrelated as of hadoop 2.0.2.
{quote}

Are you saying that we fix the problems in HBase assuming hadoop-2.0.2-alpha as the underlying
hadoop version? So far, the assumption I have been working with is that we wanted to get HBase
working with 2.0.4-alpha (and this whole thread started with trying to make HBase work with
hadoo-2.0.3-alpha, ref HBASE-7904).

bq. copyConfigurationValues seems overly broad and doesn't expose the root cause.

Hmm.. One can dump all the yarn configurations (yarn.* config keys) before and after the copyConfigurationValues
call, and figure out what changed. That might be a worthwhile experiment. So let's say that
we identify two or three configs. We should then change copyConfigurationValues to be more
restrictive in copying?

bq. You also skipped the last two questions – without getting to the root, this still doesn't
explain why it works for other hbase-mr tests and not here, and how to make it so that any
other mr tests doesn't have to have a special copyConfiguraitonValues equivalent call

Sorry about missing these earlier...

On the first one, as per the last hadoopQA run without the fixes with hadoop-2.0.4-SNAPSHOT
(https://issues.apache.org/jira/browse/HBASE-8258?focusedCommentId=13621696&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621696),
almost all of the mapreduce tests failed. I haven't looked at all the tests that passed but
one quick check on one of the passing test in the mapreduce package like TestTableSplit.java
doesn't submit jobs, and maybe that's why they passed.

On the other question, if the approach of calling copyConfigurationValues in the common part
of the test code makes the tests work (as per the last 1 or 2 hadoopqa runs), I think we should
make a note of this in some doc. We can get to this detail once we all agree on the proposed
patch.

bq. My fear is that there are fixes we need in upstream so lets figure them out sooner rather
than later (figuring them out after 2.0.4-alpha means hbase 0.96 can't release till 2.0.5-alpha/beta
whenever that is).

Stack, are you suggesting that we should depend on 2.0.4-alpha sooner rather than later, maybe
even check with 2.0.4-SNAPSHOT on a periodic basis until 2.0.4-alpha is released?
                
> Make mapreduce tests pass on hadoop2
> ------------------------------------
>
>                 Key: HBASE-8258
>                 URL: https://issues.apache.org/jira/browse/HBASE-8258
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.95.1
>
>         Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt,
8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
>
>
> HBASE-7904 was a first attempt at making this work but it got lost in the weeds.
> This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce
on hadoop1)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message