hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Duxbury (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-14) [hbase] Hung MR job writing hbase
Date Mon, 04 Feb 2008 20:13:07 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565501#action_12565501
] 

Bryan Duxbury commented on HBASE-14:
------------------------------------

What's the status of this issue?

> [hbase] Hung MR job writing hbase
> ---------------------------------
>
>                 Key: HBASE-14
>                 URL: https://issues.apache.org/jira/browse/HBASE-14
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Minor
>         Attachments: TriplesTest.java
>
>
> Running a simple, fairly large job on the cluster, a few of the reduce steps failed;
all that did are effectively hung.  Looking at failures, I see this strange output:
> {code}
> 2007-11-13 02:09:09,606 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers
because: java.io.IOException: Region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
closed
> 	at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1142)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1239)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:19,778 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers
because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:30,039 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers
because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:40,275 DEBUG org.apache.hadoop.hbase.HTable: reloading table servers
because: org.apache.hadoop.hbase.NotServingRegionException: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:55,379 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException:
triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> 	at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> 	at org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> 	at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:734)
> 	at org.apache.hadoop.hbase.HTable.commit(HTable.java:706)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:89)
> 	at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:63)
> 	at org.apache.hadoop.mapred.ReduceTask$2.collect(ReduceTask.java:308)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:66)
> 	at TriplesTest$TestReducer.reduce(TriplesTest.java:51)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:326)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {code}
> Why is the reducer client being pig-headed trying to go against a region that has been
closed?  Why not recalibrate?
> Here is master log from around same time on the problem region:
> {code}
> ...
> 2007-11-13 02:10:17,867 INFO org.apache.hadoop.hbase.HMaster: region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
split. New regions are: triples,http://purl.org/dc/terms/bibliographicCitation,1194919707661,
triples,http://purl.org/dc/terms/references,1194919707661
> ...
> 2007-11-13 02:11:47,313 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanner:
-3075797136896920808 regioninfo: {regionname: triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610,
startKey: <http://purl.org/dc/terms/bibliographicCitation>, offline: true, split: true,
tableDesc: {name: triples, families: {triples:={name: triples, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none}}}}, server: 208.76.44.140:60010,
startCode: -647590197693256616
> ...
> 2007-11-13 02:11:47,317 INFO org.apache.hadoop.hbase.HMaster: Deleting region triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
because daughter splits no longer hold references
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message