hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu Shaohui (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
Date Thu, 15 Jan 2015 08:05:36 GMT

     [ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Liu Shaohui updated HBASE-12270:
--------------------------------
    Attachment: HBASE-12270-0.98-v2.diff

[~apurtell]
Sorry for late rely.
In patch v2, set the write queue of bucket cache big enough for caching all the block in the
queue. Please kindly help to test in your env , for I can produce the failure in my dev machine.
Thanks.



> A bug in the bucket cache, with cache blocks on write enabled
> -------------------------------------------------------------
>
>                 Key: HBASE-12270
>                 URL: https://issues.apache.org/jira/browse/HBASE-12270
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.11, 0.98.6.1
>         Environment: I can reproduce it on a simple 2 node cluster, one running the master
and another running a RS. I was testing on ec2.
> I used the following configurations for the cluster. 
> hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88
-XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log
> hbase-site:
> hbase.bucketcache.ioengine=offheap
> hbase.bucketcache.size=4196
> hbase.rs.cacheblocksonwrite=true
> hfile.block.index.cacheonwrite=true
> hfile.block.bloom.cacheonwrite=true
>            Reporter: Khaled Elmeleegy
>            Assignee: Liu Shaohui
>            Priority: Critical
>             Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
>
>         Attachments: HBASE-12270-0.98-v1.diff, HBASE-12270-0.98-v2.diff, HBASE-12270-v1.diff,
HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java
>
>
> In my experiments, I have writers streaming their output to HBase. The reader powers
a web page and does this scatter/gather, where it reads 1000 keys written last and passes
them the the front end. With this workload, I get the exception below at the region server.
Again, I am using HBAse (0.98.6.1). Any help is appreciated.
> 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer:
Unexpected throwable object 
> java.lang.IllegalArgumentException
>   at java.nio.Buffer.position(Buffer.java:236)
>  at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434)
>       at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849)
>       at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760)
>  at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248)
>    at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
>       at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317)
>  at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:176)
>   at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780)
>   at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3758)
>   at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950)
>   at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913)
>       at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157)
>       at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
>    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
>      at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
>    at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
>  at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message