hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clint Morgan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2191) Commit fails occasionally with NotServingRegionException in transactional hbase inserts
Date Tue, 04 May 2010 18:43:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863938#action_12863938
] 

Clint Morgan commented on HBASE-2191:
-------------------------------------

This is by design. We don't want pending transactions to hold up Region splitting, so if a
region splits mid-transactions, the transaction will have to be retried.

If you are simply doing a bulk upload of data without any other reads/writes happening, then
I wonder if you really need to use transactions.

In general your code should handle the CommitUnsucessfulException when trying to commit. The
CME could happen when the region splits as above, but also when previously committed transaction
had writes which interferer the trx being committed. In such a case the trx can not be committed
and will throw CME. The solution is to run your transaction again from the start which would
cause the next read to be valid...

If you really need individual transactions to work across splits, can you comment on your
workload and requirements a bit more. Making it work would probably involve a deal of work.

> Commit fails occasionally with NotServingRegionException in transactional hbase inserts
> ---------------------------------------------------------------------------------------
>
>                 Key: HBASE-2191
>                 URL: https://issues.apache.org/jira/browse/HBASE-2191
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: contrib
>    Affects Versions: 0.20.3
>         Environment: OpenSolaris, java 1.6.0_10, pseudo-distributed setup; Hadoop 0.20.1
and HBase 0.20.3, single java client, Hadoop dfs.replication=1, HBase configured to use transactional
server.
>            Reporter: Jesse Hallio
>
> I'm testing a simple bulk upload to HBase. Single table, single family. The aim is to
upload 200M+ lines into HBase. I've  tried to commit the upload in various batches, from 1
line at time to 25000 lines at time. Each time, at seemingly random point, the upload fails
in commit with a org.apache.hadoop.hbase.NotServingRegionException
> Failure stack trace from the client:
> ---
> 10/02/07 16:38:40 DEBUG transactional.TransactionManager: Begining transaction 5311384003652808027
> 10/02/07 16:38:40 DEBUG transactional.TransactionState: Adding new hregion [db,,1265553236583]
to transaction [5311384003652808027]
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: atempting to commit trasaction:
id: 5311384003652808027, particpants: 1
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Region [db,,1265553236583]
votes to commit transaction 5311384003652808027
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Commiting [5311384003652808027]
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Committed transaction [5311384003652808027]
in [379]ms
> 10/02/07 16:38:46 DEBUG transactional.TransactionManager: Begining transaction -5274908504613584338
> 10/02/07 16:38:46 DEBUG transactional.TransactionState: Adding new hregion [db,,1265553236583]
to transaction [-5274908504613584338]
> 10/02/07 16:38:47 DEBUG client.HConnectionManager$TableServers: Removed db,,1265553236583
for tableName=db from cache because of \x00\x00\x01\x24\xAC\xC3\x98\x1C
> 10/02/07 16:38:47 DEBUG client.HConnectionManager$TableServers: Cached location for db,,1265553236583
is 1.1.1.1:43917
> 10/02/07 16:38:48 DEBUG client.HConnectionManager$TableServers: Removed db,,1265553236583
for tableName=db from cache because of \x00\x00\x01\x24\xAC\xC3\x98\x1C
> 10/02/07 16:38:48 DEBUG client.HConnectionManager$TableServers: Cached location for db,,1265553236583
is 1.1.1.1:43917
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: Removed db,,1265553236583
for tableName=db from cache because of \x00\x00\x01\x24\xAC\xC3\x98\x1C
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: locateRegionInMeta attempt
0 of 10 failed; retrying after sleep of 1000 because: No server address listed in .META. for
region db,\x00\x00\x01\x24\xAC\xA5\xEAB,1265553527794
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: Removed .META.,,1 for
tableName=.META. from cache because of db,\x00\x00\x01\x24\xAC\xC3\x98\x1C,99999999999999
> 10/02/07 16:38:49 DEBUG client.HConnectionManager$TableServers: Cached location for .META.,,1
is 1.1.1.1:43917
> 10/02/07 16:38:50 DEBUG client.HConnectionManager$TableServers: Cached location for db,\x00\x00\x01\x24\xAC\xA5\xEAB,1265553527794
is 1.1.1.1:43917
> 10/02/07 16:38:50 DEBUG transactional.TransactionState: Adding new hregion [db,\x00\x00\x01\x24\xAC\xA5\xEAB,1265553527794]
to transaction [-5274908504613584338]
> 10/02/07 16:38:51 DEBUG transactional.TransactionManager: atempting to commit trasaction:
id: -5274908504613584338, particpants: 2
> 10/02/07 16:38:51 DEBUG transactional.TransactionManager: Commit of transaction [-5274908504613584338]
was unsucsessful
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException:
db,,1265553236583
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2266)
>         at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.getTransactionalRegion(TransactionalRegionServer.java:131)
>         at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.commitRequest(TransactionalRegionServer.java:179)
>         at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>         at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:723)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
>         at $Proxy0.commitRequest(Unknown Source)
>         at org.apache.hadoop.hbase.client.transactional.TransactionManager.prepareCommit(TransactionManager.java:94)
>         at org.apache.hadoop.hbase.client.transactional.TransactionManager.tryCommit(TransactionManager.java:151)
>         at dbloader.dbLoader.run2(dbLoader.java:181)
>         at dbloader.dbLoader.main(dbLoader.java:24)
> Exception in thread "main" org.apache.hadoop.hbase.client.transactional.CommitUnsuccessfulException:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException:
db,,1265553236583
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2266)
>         at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.getTransactionalRegion(TransactionalRegionServer.java:131)
>         at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.commitRequest(TransactionalRegionServer.java:179)
>         at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>         at org.apache.hadoop.hbase.client.transactional.TransactionManager.prepareCommit(TransactionManager.java:135)
>         at org.apache.hadoop.hbase.client.transactional.TransactionManager.tryCommit(TransactionManager.java:151)
>         at dbloader.dbLoader.run2(dbLoader.java:181)
>         at dbloader.dbLoader.main(dbLoader.java:24)
> Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException:
db,,1265553236583
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2266)
>         at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.getTransactionalRegion(TransactionalRegionServer.java:131)
>         at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.commitRequest(TransactionalRegionServer.java:179)
>         at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>         at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:723)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
>         at $Proxy0.commitRequest(Unknown Source)
>         at org.apache.hadoop.hbase.client.transactional.TransactionManager.prepareCommit(TransactionManager.java:94)
>         ... 3 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message