hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-8258) Make mapreduce tests pass on hadoop2
Date Fri, 05 Apr 2013 23:33:19 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624173#comment-13624173
] 

Jonathan Hsieh edited comment on HBASE-8258 at 4/5/13 11:33 PM:
----------------------------------------------------------------

Hm.. On my second try, it looks like hadoop1 has got some failures (essentially same as hadoopqa
bot), but the hadoop2 build (on an internal jenkins) is good. 

On my first run against hadoop2 I had problems with vmem overuse:
{code}
2013-04-05 14:57:45,983 WARN  [AsyncDispatcher event handler] resourcemanager.RMAuditLogger(255):
USER=jenkins	OPERATION=Application Finished - Failed	TARGET=RMAppManager	RESULT=FAILURE	DESCRIPTION=App
failed with state: FAILED	PERMISSIONS=Application application_1365199040071_0001 failed 1
times due to AM Container for appattempt_1365199040071_0001_000001 exited with  exitCode:
143 due to: Container [pid=24860,containerID=container_1365199040071_0001_01_000001] is running
beyond virtual memory limits. Current usage: 269.3mb of 2.0gb physical memory used; 6.3gb
of 4.2gb virtual memory used. Killing container.
{code}

I've uncommented the pmem/vmem lines in the patch to make hadoop2 happy.
{code}
    // Tests were failing because this process used 6GB of virtual memory and was getting
killed.
    // we up the VM usable so that processes don't get killed.
    conf.setInt("yarn.nodemanager.resource.memory-mb", 8 * 1024);
    conf.setFloat("yarn.nodemanager.vmem-pmem-ratio", 8.0f);
{code}

Digging into the hadoop 1 issues now.



                
      was (Author: jmhsieh):
    Hm.. On my second try, it looks like hadoop1 has got some failures (essentiall same as
hadoopqa bot), but the hadoop2 build (on an internal jenkins) is good. 

On my first run against hadoop2 I had problems with vmem overuse:
{code}
2013-04-05 14:57:45,983 WARN  [AsyncDispatcher event handler] resourcemanager.RMAuditLogger(255):
USER=jenkins	OPERATION=Application Finished - Failed	TARGET=RMAppManager	RESULT=FAILURE	DESCRIPTION=App
failed with state: FAILED	PERMISSIONS=Application application_1365199040071_0001 failed 1
times due to AM Container for appattempt_1365199040071_0001_000001 exited with  exitCode:
143 due to: Container [pid=24860,containerID=container_1365199040071_0001_01_000001] is running
beyond virtual memory limits. Current usage: 269.3mb of 2.0gb physical memory used; 6.3gb
of 4.2gb virtual memory used. Killing container.
{code}

I've uncommented the pmem/vmem lines in the patch to make hadoop2 happy.
{code}
    // Tests were failing because this process used 6GB of virtual memory and was getting
killed.
    // we up the VM usable so that processes don't get killed.
    conf.setInt("yarn.nodemanager.resource.memory-mb", 8 * 1024);
    conf.setFloat("yarn.nodemanager.vmem-pmem-ratio", 8.0f);
{code}

Digging into the hadoop 1 issues now.



                  
> Make mapreduce tests pass on hadoop2
> ------------------------------------
>
>                 Key: HBASE-8258
>                 URL: https://issues.apache.org/jira/browse/HBASE-8258
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.95.1
>
>         Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt,
8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
>
>
> HBASE-7904 was a first attempt at making this work but it got lost in the weeds.
> This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce
on hadoop1)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message