hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhihong Yu (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
Date Thu, 23 Feb 2012 20:30:48 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215024#comment-13215024
] 

Zhihong Yu commented on HBASE-5317:
-----------------------------------

{code}
LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ find . -name stderr
./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/stderr
./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1330028372796_0001/container_1330028372796_0001_01_000001/stderr
./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1330028372796_0001/container_1330028372796_0001_01_000006/stderr
./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1330028372796_0001/container_1330028372796_0001_01_000004/stderr
./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1330028372796_0001/container_1330028372796_0001_01_000005/stderr
./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_2/application_1330028372796_0001/container_1330028372796_0001_01_000002/stderr
./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_3/application_1330028372796_0001/container_1330028372796_0001_01_000003/stderr
LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/stderr
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter
in all the log4j.properties files.
LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_3/application_1330028372796_0001/container_1330028372796_0001_01_000003/stderr
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter
in all the log4j.properties files.
{code}
>From org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/syslog:
{code}
2012-02-23 12:22:34,208 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler:
Assiging lm-sjn-00713032.corp.ebay.com:61263 with 1 to fetcher#2
2012-02-23 12:22:34,208 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler:
assigned 1 of 1 to lm-sjn-00713032.corp.ebay.com:61263 to fetcher#2
2012-02-23 12:22:34,469 WARN [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.Fetcher:
Failed to connect to lm-sjn-00713032.corp.ebay.com:61263 with 1 map outputs
java.io.IOException: Server returned HTTP response code: 503 for URL: http://lm-sjn-00713032.corp.ebay.com:61263/mapOutput?job=job_1330028372796_0001&reduce=0&map=attempt_1330028372796_0001_m_000000_1
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
  at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:220)
  at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152)
2012-02-23 12:22:34,471 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler:
Reporting fetch failure for attempt_1330028372796_0001_m_000000_1 to jobtracker.
2012-02-23 12:22:34,471 FATAL [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler:
Shuffle failed with too many fetch failures and insufficient progress!
2012-02-23 12:22:34,472 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler:
lm-sjn-00713032.corp.ebay.com:61263 freed by fetcher#2 in 264s
2012-02-23 12:22:34,472 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:zhihyu (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
error in shuffle in fetcher#2
2012-02-23 12:22:34,472 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in
fetcher#2
  at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123)
  at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:396)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
  at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.checkReducerHealth(ShuffleScheduler.java:253)
  at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.copyFailed(ShuffleScheduler.java:187)
  at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:247)
  at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152)

2012-02-23 12:22:34,476 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the
task
{code}
Looks like 503 caused fetch failure.
                
> Fix TestHFileOutputFormat to work against hadoop 0.23
> -----------------------------------------------------
>
>                 Key: HBASE-5317
>                 URL: https://issues.apache.org/jira/browse/HBASE-5317
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.92.0, 0.94.0
>            Reporter: Gregory Chanan
>            Assignee: Gregory Chanan
>         Attachments: HBASE-5317-v0.patch, HBASE-5317-v1.patch, HBASE-5317-v3.patch, HBASE-5317-v4.patch,
HBASE-5317-v5.patch, HBASE-5317-v6.patch, TEST-org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.xml
>
>
> Running
> mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
> yields this on 0.92:
> Failed tests:   testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat):
HFile for column family info-A not found
> Tests in error: 
>   test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_000000_0
(Is a directory)
>   testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable
>   testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat):
TestTable
> It looks like on trunk, this also results in an error:
>   testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat):
TestTable
> I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't
fixed the other 3 yet.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message