hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Campbell <deegs...@yahoo.com>
Subject Re: REcovering from SocketTimeout during scan in 90.3
Date Fri, 16 Sep 2011 19:01:23 GMT
I'm reducing keys by regions and then building a Scan with
a. start/stop = minkey/max+1
b. 100 cache rows
c. cache blocks false
d. configuring scan like this
  public static void setRowFilters(Scan scan, Iterator<byte[]> keys) {
    FilterList filterlist = new FilterList(Operator.MUST_PASS_ONE);
    for (; keys.hasNext(); ) {
      filterlist.addFilter(new RowFilter(CompareOp.EQUAL, new BinaryComparator(keys.next())));
    }
    scan.setFilter(filterlist);
  }

where the iterator contains the keys I want

Are you saying I should also re-instantiate the htable?

Also, looking at Regionserver stack trace shows the following blocked threads

"IPC Server handler 19 on 60020" daemon prio=10 tid=0x0000000053838000 nid=0x3224 waiting
for monitor entry [0x0000000045799000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2322)
        - waiting to lock <0x0000000782a96ad8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1823)
        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

"IPC Server handler 18 on 60020" daemon prio=10 tid=0x0000000053835800 nid=0x3223 waiting
for monitor entry [0x0000000045698000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2322)
        - waiting to lock <0x000000071d024b60> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1823)
        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

"IPC Server handler 17 on 60020" daemon prio=10 tid=0x0000000053833800 nid=0x3222 waiting
for monitor entry [0x0000000045597000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.io.compress.CodecPool.payback(CodecPool.java:75)
        - locked <0x00000006fbe527f8> (a java.util.HashMap)
        at org.apache.hadoop.io.compress.CodecPool.returnDecompressor(CodecPool.java:152)
        at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.returnDecompressor(Compression.java:245)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1098)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
        - locked <0x000000074aa2a950> (a [B)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1276)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:115)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:255)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:394)




________________________________
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Sent: Friday, September 16, 2011 10:47 AM
Subject: Re: REcovering from SocketTimeout during scan in 90.3

Yeah this should be at the HTable level...

Your solution sounds right.

How did you get yourself in this situation btw? Using a large scanner
caching value plus filters?

J-D

On Fri, Sep 16, 2011 at 10:18 AM, Douglas Campbell <deegs_ca@yahoo.com> wrote:
> What's the best way to recover from this?
>
> Set the start row to last sucessfully processed row + 1 and re-init scanner?
>
> java.io.IOException: java.net.SocketTimeoutException: Call to $host failed on socket
timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be
ready for read. ch
> : java.nio.channels.SocketChannel[connected local=/216.109.127.129:53241
> remote=hostname/ip:60020]
>
> at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1232)
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message