hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksandr Shulman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8619) [HBase Client]: Improve client to retry if master is down when requesting getHTableDescriptor
Date Sun, 11 Aug 2013 04:01:48 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13736179#comment-13736179
] 

Aleksandr Shulman commented on HBASE-8619:
------------------------------------------

I'm seeing this occur in my testing. It's a race condition so it's hard to repro, unfortunately.
The latest stack trace where I saw it looked like this:
{code}  Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
20:30:00  	at $Proxy7.getHTableDescriptors(Unknown Source)
20:30:00  	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1863)
20:30:00  	at org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:440)
20:30:00  	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureCompression(HFileOutputFormat.java:458)
20:30:00  	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:375)
20:30:00  	at org.apache.hadoop.hbase.mapreduce.ImportTsv.createSubmittableJob(ImportTsv.java:280)
20:30:00  	at org.apache.hadoop.hbase.mapreduce.ImportTsv.main(ImportTsv.java:424)
20:30:00  Caused by: java.io.IOException: Call to hbase-rolling-6.ent.cloudera.com/10.20.186.99:22001
failed on local exception: java.io.IOException: Connection reset by peer
20:30:00  	at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1033)
20:30:00  	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1002)
20:30:00  	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
20:30:00  	... 7 more
20:30:00  Caused by: java.io.IOException: Connection reset by peer
20:30:00  	at sun.nio.ch.FileDispatcher.read0(Native Method)
20:30:00  	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
20:30:00  	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
20:30:00  	at sun.nio.ch.IOUtil.read(IOUtil.java:171)
20:30:00  	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
20:30:00  	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:56)
20:30:00  	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
20:30:00  	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
20:30:00  	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
20:30:00  	at java.io.FilterInputStream.read(FilterInputStream.java:116)
20:30:00  	at java.io.FilterInputStream.read(FilterInputStream.java:116)
20:30:00  	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:373)
20:30:00  	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
20:30:00  	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
20:30:00  	at java.io.DataInputStream.readInt(DataInputStream.java:370)
20:30:00  	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
20:30:00  	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580)
{code}
                
> [HBase Client]: Improve client to retry if master is down when requesting getHTableDescriptor
> ---------------------------------------------------------------------------------------------
>
>                 Key: HBASE-8619
>                 URL: https://issues.apache.org/jira/browse/HBASE-8619
>             Project: HBase
>          Issue Type: Bug
>          Components: Client
>    Affects Versions: 0.94.8
>            Reporter: Aleksandr Shulman
>            Assignee: Elliott Clark
>            Priority: Minor
>             Fix For: 0.94.12
>
>
> In our rolling upgrade testing, running ImportTsv failed in the job submission phase
when the master was down. This was when the master was failing over to the backup master.
In this case, a retry would have been helpful and made sure that the job would get submitted.
> A good solution would be to refresh the master information before placing the call to
getHTableDescriptor.
> Command:
> {code} sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=HBASE_ROW_KEY,c1,c2,c3
-Dimporttsv.bulk.output=/user/hbase/storeFiles2_2/import2_table1369439156 import2_table1369439156
/user/hbase/tsv2{code}
> Here is the stack trace:
> {code} 13/05/24 16:55:49 INFO compress.CodecPool: Got brand-new compressor [.deflate]
> 16:45:44  Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> 16:45:44  	at $Proxy7.getHTableDescriptors(Unknown Source)
> 16:45:44  	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1861)
> 16:45:44  	at org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:440)
> 16:45:44  	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureCompression(HFileOutputFormat.java:458)
> 16:45:44  	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:375)
> 16:45:44  	at org.apache.hadoop.hbase.mapreduce.ImportTsv.createSubmittableJob(ImportTsv.java:280)
> 16:45:44  	at org.apache.hadoop.hbase.mapreduce.ImportTsv.main(ImportTsv.java:424)
> 16:45:44  Caused by: java.io.IOException: Call to hbase-rolling-6.ent.cloudera.com/10.20.186.99:22001
failed on local exception: java.io.EOFException
> 16:45:44  	at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1030)
> 16:45:44  	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:999)
> 16:45:44  	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
> 16:45:44  	... 7 more
> 16:45:44  Caused by: java.io.EOFException
> 16:45:44  	at java.io.DataInputStream.readInt(DataInputStream.java:375)
> 16:45:44  	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
> 16:45:44  	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580){code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message