hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mubarak Seyed (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5614) createTableAsync gets blocked and it is not an asynchronous operation
Date Wed, 21 Mar 2012 22:42:41 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235115#comment-13235115
] 

Mubarak Seyed commented on HBASE-5614:
--------------------------------------

{{HBaseAdmin.createAsyncTable}} call from client to Master:

{code}
12/03/21 18:00:26 WARN zookeeper.ZKTable: Moving table table_A state to enabled but was already
enabled
12/03/21 18:00:26 INFO wal.HLog: HLog configuration: blocksize=128 MB, rollsize=121.6 MB,
enabled=true, optionallogflushinternal=1000ms
12/03/21 18:00:26 INFO wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
12/03/21 18:00:26 INFO wal.HLog: New hlog /hbase/root/table_A/6d3fe8aa7eae1c13b4ce5b54a13af1e0/.logs/hlog.1332352826075
12/03/21 18:00:26 INFO wal.HLog: Using getNumCurrentReplicas--HDFS-826
12/03/21 18:00:26 INFO regionserver.HRegion: Onlined table_A,,1332352825867.6d3fe8aa7eae1c13b4ce5b54a13af1e0.;
next sequenceid=1
12/03/21 18:00:26 INFO catalog.MetaEditor: Added 1 regions to META
12/03/21 18:00:26 INFO regionserver.HRegion: Closed table_A,,1332352825867.6d3fe8aa7eae1c13b4ce5b54a13af1e0.
12/03/21 18:00:26 INFO regionserver.HRegion: Onlined table_A,0015d867,1332352825873.ecf7c1e64eae49c396577e3156bc6555.;
next sequenceid=1
12/03/21 18:00:26 INFO regionserver.HRegion: Closed table_A,0015d867,1332352825873.ecf7c1e64eae49c396577e3156bc6555.
{code}

After 5 minutes {{hbase.rpc.timeout}} value, master client channel gets ClosedChannelException
{code}
12/03/21 18:05:28 WARN ipc.HBaseServer: IPC Server Responder, call createTable({NAME =>
'table_A', FAMILIES => [{NAME => 'CF1s', BLOOMFILTER => 'ROW', REPLICATION_SCOPE
=> '0', VERSIONS => '1', COMPRESSION => 'LZO', TTL => '298700000', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}, [[B@25a0504a) from 1.1.1.1:34981:
output error
12/03/21 18:05:28 WARN ipc.HBaseServer: IPC Server handler 73 on 60000 caught: java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
        at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1341)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
{code}

{{AssignmentManager}} gets {{null}} value as server location
{code}
12/03/21 18:35:13 INFO master.AssignmentManager: Server null returned java.lang.NullPointerException:
Passed server is null for 6d3fe8aa7eae1c13b4ce5b54a13af1e0
12/03/21 18:35:13 INFO master.AssignmentManager: Server null returned java.lang.NullPointerException:
Passed server is null for a47815dfcdf24f49e40f50b6fdd65dd9
12/03/21 18:35:13 INFO master.AssignmentManager: Server null returned java.lang.NullPointerException:
Passed server is null for ecf7c1e64eae49c396577e3156bc6555
{code}

>From master log, bulk assignment happens (100 regions at a time), i can increase the value
to 200 (or higher) but {{createTableAsync}} shouldn't get blocked.
                
> createTableAsync gets blocked and it is not an asynchronous operation
> ---------------------------------------------------------------------
>
>                 Key: HBASE-5614
>                 URL: https://issues.apache.org/jira/browse/HBASE-5614
>             Project: HBase
>          Issue Type: Bug
>          Components: master, regionserver
>    Affects Versions: 0.90.4
>            Reporter: Mubarak Seyed
>              Labels: noob
>
> I have created table A using {{HBaseAdmin.createTableAsync()}} with 2000 pre-split regions,
the expectation was client RPC call does not get blocked and i had only used 5 minutes as
RPC socket timeout value.
> It appears from logs/code that clients gets blocked (honoring the {{hbase.rpc.timeout}}
value), times out after 5 minutes, master nio channel gets closed (related to client which
invoked the createAsyncTable operation), put region entries in inconsistent state in META,
meaning {{info:server}} is marked as null and AssignmentManager gets null server value (which
causes NPE) and causes to table creation fail, i could not succeed until changing the {{hbase.rpc.timeout}}
value to 15 minutes and created tables one at a time (each table has more than 2K pre-split
regions in it)
> Please refer the logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message