hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rong-En Fan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-827) Failed to open scanner in shell after series of action executed
Date Sun, 17 Aug 2008 05:21:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623175#action_12623175
] 

Rong-En Fan commented on HBASE-827:
-----------------------------------

I grab the MyClient.java and latest trunk (r686572) to try it out. So, the example program
do the following things:

1. insert 40000 columns under one col family
2. insert a number into the same col family to record the total # of cols in this family
3. repeat 100,000 times, do read one column value, modify it, insert back
4. delete everything
5. repeat 1 but with much smaller columns to insert

Then I try to get a scanner from java client:

table.getScanner(new String[]{"insPool:"});

The table in question has only few columns but lots of deleted records (about 140,000). The
scanner needs some time to finish it's job. The line above requires 240 seconds to finish
(that's 4 mins).

However, in hirb.rb, we have

@configuration.setInt("hbase.client.retries.number", 3)
@configuration.setInt("ipc.client.connect.max.retries", 3)

These are smaller than hbase-default.xml, and this is why scan shell fails.

> Failed to open scanner in shell after series of action executed
> ---------------------------------------------------------------
>
>                 Key: HBASE-827
>                 URL: https://issues.apache.org/jira/browse/HBASE-827
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.3.0
>         Environment: Java runtime version : 1.6.0_06
> Hbase version : occured in both 0.2.0 RC2 & 0.3.0 trunk (svn 685494)
> Hbase configuration: run in localhost without hadoop
>            Reporter: PeterL in
>         Attachments: hbase.log, jstack.log, MyClient.java
>
>
> Background introduction :
> I was intent to implement a large queue-liked structure based on HBase.
> Different from normal queue structure, in my structure, a pop action means 'randomly'
pick one elements(which is key/value pair from a row) in queue, and decrease the queueSize
of that queue.
> For performance consideration, I implemented pop action by following steps :
> 1.get the queue size
> 2.random a number between 0~queueSize
> 3.take step2 as a qualifier, get the value of that element, and put the last element's
value to the original qualifier.
> 4.decrease the queue size. (I didn't delete the last elements since storage space is
not my primary consideration)
> When the queueSize and number of pop action become large, hbase failed to open a scanner.
(Even the data of database is correct)
> My program can be split into few steps:
> 1.create a large queue.
> 2.pop until the queue empty
> 3.keep popping the queue 
> 4.delete the queue ( delete a row with deleteAll )
> 5.create a small queue with same name (i.e: same row)
> After create a small queue, I checked every key/value in the hbase and found it is correct.
But when I tried to open a scanner, it got Unknown Exception as follows:
> In shell :
> NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server 127.0.0.1:54141 for region pleaseCrash,,1218621377152, row '', but failed
after 3 attempts.
> Exceptions:
> java.net.SocketTimeoutException: timed out waiting for rpc response
> java.net.SocketTimeoutException: timed out waiting for rpc response
> java.net.SocketTimeoutException: timed out waiting for rpc response
>         from org/apache/hadoop/hbase/client/HConnectionManager.java:873:in `getRegionServerWithRetries'
>         from org/apache/hadoop/hbase/client/HTable.java:1415:in `nextScanner'
>         from org/apache/hadoop/hbase/client/HTable.java:1360:in `<init>'
>         from org/apache/hadoop/hbase/client/HTable.java:985:in `getScanner'
>         from org/apache/hadoop/hbase/client/HTable.java:779:in `getScanner'
>         from org/apache/hadoop/hbase/client/HTable.java:706:in `getScanner'
>         from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
>         from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
>         from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>         from java/lang/reflect/Method.java:597:in `invoke'
>         from org/jruby/javasupport/JavaMethod.java:250:in `invokeWithExceptionHandling'
>         from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
>         from org/jruby/javasupport/JavaClass.java:416:in `execute'
>         from org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in `call'
>         from org/jruby/internal/runtime/methods/DynamicMethod.java:78:in `call'
>         from org/jruby/runtime/CallSite.java:155:in `cacheAndCall'
> ... 128 levels...
>         from ruby.home.itspeter.hbase_minus_0_dot_3_dot_0_minus_dev.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
`call'
>         from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
>         from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
>         from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
>         from org/jruby/runtime/CallSite.java:298:in `call'
>         from ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:350:in
`__file__'
>         from ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
`__file__'
>         from ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
`load'
>         from org/jruby/Ruby.java:512:in `runScript'
>         from org/jruby/Ruby.java:432:in `runNormally'
>         from org/jruby/Ruby.java:312:in `runFromMain'
>         from org/jruby/Main.java:144:in `run'
>         from org/jruby/Main.java:89:in `run'
>         from org/jruby/Main.java:80:in `main'
>         from /home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:271:in `scan'
>         from (hbase):3:in `binding'
> Side effects :
> 	After scan got time out, I discovered that I also failed to disable the table (But not
every time), somehow more strange is that table will be dropped successfully even disable
failed (but not every time).
> System Resources:
> 	Hbase will rob the cpu...
> 	In my platform(4-core), cpu usage will raise from 10% -> 50% ..100% -> 200% ->
300% ..
> Procedure to reproduce :
> Run the attached java file. 
> Try to type "scan 'pleaseCrash'" in hbase shell after program finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message