hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kohki Nishio <tarop...@icloud.com>
Subject Re: Scan returns incomplete results
Date Sat, 15 Apr 2017 00:03:25 GMT
I set the server's log level to trace, but I don't see anything suspicious, and my code is
really simple

Here's my code snippet
https://pastebin.com/qGKyfAWS

But again, this happens only when lots of put is happening on the same server.
-Kohki


On Apr 14, 2017, at 01:47 PM, Ted Yu <yuzhihong@gmail.com> wrote:

Can you check server log pertaining to scanner ids 692411060 and 692411074 ?

Pastebin relevant snippet, please.

Thanks

On Fri, Apr 14, 2017 at 1:01 PM, Kohki Nishio <taroplus@icloud.com> wrote:

Yes it is from hbase-client 1.3.0, strange thing is that this only happens
under load.
Thanks
-Kohki

On Apr 14, 2017, at 12:30 PM, Ted Yu <yuzhihong@gmail.com> wrote:

Was the trace obtained with 1.3 client ?

Thanks


On Apr 14, 2017, at 11:43 AM, Kohki Nishio <taroplus@icloud.com> wrote:


Here I have trace output, the exact consecutive queries got two different
results, not sure where to look about my region servers..



18:30:00,073 o.a.h.h.c.ClientScanner: Scan table=mytable,
startRow=AAAAAZJRJgq702gz9Qq-bXvyvi5Vfwg-\x00\x00\x00\x00X\xF1\x06\xDC

18:30:00,075 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms

18:30:00,075 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
scanner as id=692411060 associated with replica=0

18:30:00,076 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms

18:30:00,076 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
scanner as id=692411060 associated with replica=0

18:30:00,077 o.a.h.h.c.ClientScanner: number results from RPC: 50,partial
!= null: false,number of partials so far: 0

18:30:00,078 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms

18:30:00,078 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
scanner as id=692411060 associated with replica=0

18:30:00,078 o.a.h.h.c.ClientScanner: number results from RPC: 6,partial
!= null: false,number of partials so far: 0

18:30:00,078 o.a.h.h.c.ScannerCallableWithReplicas: Closing scanner
id=692411060

18:30:00,079 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms

18:30:00,079 o.a.h.h.c.ClientScanner: Finished {ENCODED =>
c6e51623c143984e36c21340b182d2c7, NAME => 'mytable,AAAAAZ-
L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D,1492193926115.
c6e51623c143984e36c21340b182d2c7.', STARTKEY => 'AAAAAZ-
L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D', ENDKEY => '
AAAAAZzZUufbQvQaVdhpt2wUk1CkRcNz\x00\x00\x00\x00X\xE9\xE71'}



18:30:00,458 o.a.h.h.c.ClientScanner: Scan table=mytable,
startRow=AAAAAZJRJgq702gz9Qq-bXvyvi5Vfwg-\x00\x00\x00\x00X\xF1\x06\xDC

18:30:00,459 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms

18:30:00,459 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
scanner as id=692411074 associated with replica=0

18:30:00,461 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms

18:30:00,461 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
scanner as id=692411074 associated with replica=0

18:30:00,461 o.a.h.h.c.ClientScanner: number results from RPC: 44,partial
!= null: false,number of partials so far: 0

18:30:00,461 o.a.h.h.c.ScannerCallableWithReplicas: Closing scanner
id=692411074

18:30:00,461 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 0ms

18:30:00,461 o.a.h.h.c.ClientScanner: Finished {ENCODED =>
c6e51623c143984e36c21340b182d2c7, NAME => 'mytable,AAAAAZ-
L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D,1492193926115.
c6e51623c143984e36c21340b182d2c7.', STARTKEY => 'AAAAAZ-
L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D', ENDKEY => '
AAAAAZzZUufbQvQaVdhpt2wUk1CkRcNz\x00\x00\x00\x00X\xE9\xE71'}




On Apr 13, 2017, at 08:09 PM, Kohki Nishio <taroplus@icloud.com> wrote:



Actually I tried the one with HBASE-15378, but it didn’t fix the problem ..



On Apr 12, 2017, at 8:14 PM, Kohki Nishio <taroplus@icloud.com> wrote:


As far as I can see, the change is in ClientScanner class, which is a part
of Hbase-client, does this work inside Server (region server?) ?


Thanks

-Kohki



On Apr 12, 2017, at 6:03 PM, Ted Yu <yuzhihong@gmail.com> wrote:


Does the release (deployed on servers) contain HBASE-15378 ?


Cheers


On Wed, Apr 12, 2017 at 5:34 PM, Kohki Nishio <taroplus@icloud.com> wrote:


Hello HBase users !


I’m seeing very strange behavior with HBase (1.1.2) and I’d like to ask

some help here. I’m scanning rows but sometimes it returns incomplete

results. Let’s say I’m expecting 60 rows to come back, most of times I get

all of them. However sometimes (1 in 50), I get only 1 or 2 rows.


According to my test, this happens only when there are other activities

(lots of PUT / GET are happening at the same time), I couldn’t reproduce

with a simple standalone scan test case. First we saw this problem with

AsyncHBase client then I tried v1.3 HBase client, both showed the exact

same behavior.


Based on the trace log message, it seems this is not a problem at client

side since two different client implementations showed the same behavior.

However HBase is an established storage solution, it is so strange to see

this problem at this frequency, that leads me to believe that I might be

doing something very strange. But I’m not sure what to do next.


The key symptom is that it returns the correct result for most of times,

but sometimes it returns very small set of results. Any thoughts would be

appreciated (RegionServer is at 1.1.2.x.x)


Thanks

-Kohki




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
    • Unnamed multipart/related (inline, None, 0 bytes)
View raw message