hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars George (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HBASE-3452) Report errors better when async flushCommits() fail
Date Tue, 18 Jan 2011 21:33:44 GMT

     [ https://issues.apache.org/jira/browse/HBASE-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars George resolved HBASE-3452.
--------------------------------

    Resolution: Duplicate

Dupe of HBASE-2330

> Report errors better when async flushCommits() fail
> ---------------------------------------------------
>
>                 Key: HBASE-3452
>                 URL: https://issues.apache.org/jira/browse/HBASE-3452
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.89.20100924
>            Reporter: Lars George
>            Priority: Minor
>             Fix For: 0.90.1
>
>
> We had the issue where a MapReduce would fail with the following error:
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Still had 3913 puts left after
retrying 10 times.
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfPuts(HConnectionManager.java:1526)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:664)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:549)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
> at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:104)
> at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:65)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:512)
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at com.lp.sessionized.mapred.HbaseIndexerReducer.reduce(HbaseIndexerReducer.java:91)
> at com.lp.sessionized.mapred.HbaseIndexerReducer.reduce(HbaseIndexerReducer.java:1)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:570)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
> at org.apache.hadoop.mapred.Child.main(Child.java:211)
> {code}
> which sent us on a wild goose hunt to figure why we could read from a table but not write
back into it. We finally checked the server logs and got this:
> {code}
> 2011-01-18 13:47:56,479 WARN org.apache.hadoop.hbase.regionserver.HRegion: No such column
family in batch put
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family dim does
not exist in region sessions6,,1295358051638.272a4ba588438119f9f866f491a4428c. in table {NAME
=
> > 'foo', FAMILIES => [{NAME => 'dims', BLOOMFILTER => 'NONE', REPLICATION_SCOPE
=> '0'
> , COMPRESSION => 'LZO', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMOR
> Y => 'false', BLOCKCACHE => 'true'}, ...
> at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:2931)
> at org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:1683)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1356)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1814)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:24
> 79)
> {code}
> So we had a typo in the colfam and that was reported server side. That error never made
it into the task logs, but should have to make this much easier to track down. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message