hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "PeterL in (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-827) Failed to open scanner in shell after series of action executed
Date Wed, 13 Aug 2008 11:34:48 GMT
Failed to open scanner in shell after series of action executed

                 Key: HBASE-827
                 URL: https://issues.apache.org/jira/browse/HBASE-827
             Project: Hadoop HBase
          Issue Type: Bug
    Affects Versions: 0.3.0
         Environment: Java runtime version : 1.6.0_06
Hbase version : occured in both 0.2.0 RC2 & 0.3.0 trunk (svn 685494)
Hbase configuration: run in localhost without hadoop
            Reporter: PeterL in

Background introduction :
I was intent to implement a large queue-liked structure based on HBase.
Different from normal queue structure, in my structure, a pop action means 'randomly' pick
one elements(which is key/value pair from a row) in queue, and decrease the queueSize of that

For performance consideration, I implemented pop action by following steps :
1.get the queue size
2.random a number between 0~queueSize
3.take step2 as a qualifier, get the value of that element, and put the last element's value
to the original qualifier.
4.decrease the queue size. (I didn't delete the last elements since storage space is not my
primary consideration)

When the queueSize and number of pop action become large, hbase failed to open a scanner.
(Even the data of database is correct)

My program can be split into few steps:
1.create a large queue.
2.pop until the queue empty
3.keep popping the queue 
4.delete the queue ( delete a row with deleteAll )
5.create a small queue with same name (i.e: same row)

After create a small queue, I checked every key/value in the hbase and found it is correct.
But when I tried to open a scanner, it got Unknown Exception as follows:

In shell :
NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server for region pleaseCrash,,1218621377152, row '', but failed after
3 attempts.
java.net.SocketTimeoutException: timed out waiting for rpc response
java.net.SocketTimeoutException: timed out waiting for rpc response
java.net.SocketTimeoutException: timed out waiting for rpc response

        from org/apache/hadoop/hbase/client/HConnectionManager.java:873:in `getRegionServerWithRetries'
        from org/apache/hadoop/hbase/client/HTable.java:1415:in `nextScanner'
        from org/apache/hadoop/hbase/client/HTable.java:1360:in `<init>'
        from org/apache/hadoop/hbase/client/HTable.java:985:in `getScanner'
        from org/apache/hadoop/hbase/client/HTable.java:779:in `getScanner'
        from org/apache/hadoop/hbase/client/HTable.java:706:in `getScanner'
        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
        from java/lang/reflect/Method.java:597:in `invoke'
        from org/jruby/javasupport/JavaMethod.java:250:in `invokeWithExceptionHandling'
        from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
        from org/jruby/javasupport/JavaClass.java:416:in `execute'
        from org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in `call'
        from org/jruby/internal/runtime/methods/DynamicMethod.java:78:in `call'
        from org/jruby/runtime/CallSite.java:155:in `cacheAndCall'
... 128 levels...
        from ruby.home.itspeter.hbase_minus_0_dot_3_dot_0_minus_dev.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
        from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
        from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
        from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
        from org/jruby/runtime/CallSite.java:298:in `call'
        from ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:350:in
        from ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
        from ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
        from org/jruby/Ruby.java:512:in `runScript'
        from org/jruby/Ruby.java:432:in `runNormally'
        from org/jruby/Ruby.java:312:in `runFromMain'
        from org/jruby/Main.java:144:in `run'
        from org/jruby/Main.java:89:in `run'
        from org/jruby/Main.java:80:in `main'
        from /home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:271:in `scan'
        from (hbase):3:in `binding'

Side effects :
	After scan got time out, I discovered that I also failed to disable the table (But not every
time), somehow more strange is that table will be dropped successfully even disable failed
(but not every time).

System Resources:
	Hbase will rob the cpu...
	In my platform(4-core), cpu usage will raise from 10% -> 50% ..100% -> 200% -> 300%

Procedure to reproduce :
Run the attached java file. 
Try to type "scan 'pleaseCrash'" in hbase shell after program finished.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message