hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
Date Fri, 05 Apr 2013 00:00:17 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623091#comment-13623091
] 

Ted Yu commented on HBASE-8258:
-------------------------------

[~jmhsieh]:
bq. I'd like to get this fixed without having to upgrade hadoop versions to a snapshot version.
Did you mean that we should wait for 2.0.4-alpha to come out ? I think for 0.95, we should
use 2.0.4-alpha which is to be released soon.

If we use 2.0.3-alpha, we would get (due to capacity-scheduler.xml missing in artifact):
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13584497&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13584497

In case you haven't read comments from Siddharth, you can find them starting with this one
:

https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13611080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611080

That was how the HBaseConfiguration.merge() call in HBaseTestingUtility.java came about.

Changes in TestImportExport.java were to align with remaining tests in that file. The comment
below would provide some more context:
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13617700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13617700

bq. vague statements "necessary configuration values" without explanation
I think you were referring to the following:
{code}
    // copy or add the necessary configuration values from the map reduce config to the hbase
config
{code}
Such comments have been in TestImportExport. This command can show us the history:
{code}
$ svn blame hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
| grep 'necessary configurati'
{code}
For SecureBulkLoadEndpoint.java, here is the background:
Secure bulk loading needs to establish staging directory on hdfs. So it calls:
{code}
      fs = FileSystem.get(conf);
{code}
The conf above is an instance of CompoundConfiguration. NameNodeProxies would set up RPC engine
by modifying conf.
This results in:
{code}
2013-03-25 22:36:19,493 ERROR [IPC Server handler 2 on 40900] access.SecureBulkLoadEndpoint$1(240):
Failed to complete bulk load
java.lang.UnsupportedOperationException: Immutable Configuration
  at org.apache.hadoop.hbase.CompoundConfiguration.setClass(CompoundConfiguration.java:474)
  at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:193)
  at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:249)
  at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:168)
  at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:421)
  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:388)
  at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:125)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2277)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2311)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2293)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:317)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
  at org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:224)
  at org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:218)
{code}
In 8258-v1-hadoop-2.0.txt, I tried to limit the scope of changes while satisfying requirement
from hadoop 2.0
That is, clone a mutable Configuration (an HBaseConfiguration to be more exact) from the CompoundConfiguration
and pass it to FileSystem.get().

I am open to discussion about proposed changes. I can upload the patch onto review board so
that opinion on each change can be expressed better.

I am open to other proposals as well.
                
> Make mapreduce tests pass on hadoop2
> ------------------------------------
>
>                 Key: HBASE-8258
>                 URL: https://issues.apache.org/jira/browse/HBASE-8258
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.95.1
>
>         Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt
>
>
> HBASE-7904 was a first attempt at making this work but it got lost in the weeds.
> This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce
on hadoop1)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message