hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth P. Shreenivas" <Srikanth_Shreeni...@mindtree.com>
Subject HBase Read and Write Issues in Mutlithreaded Environments
Date Wed, 29 Jun 2011 04:51:23 GMT
Hi,

We are using HBase 0.20.3 (hbase-0.20-0.20.3-1.cloudera.noarch.rpm) cluster in distributed
mode with Hadoop 0.20.2 (hadoop-0.20-0.20.2+320-1.noarch).
We are using pretty much default configuration, and only thing we have customized is that
we have allocated 4GB RAM in /etc/hbase-0.20/conf/hbase-env.sh

In our setup, we have a web application that reads a record from HBase and writes a record
as part of each web request.   The application is hosted in Apache Tomcat 7 and is a stateless
web application providing a REST-like web service API.

We are observing that our reads and writes times out once in a  while.  This happens more
for writes.
We see below exception in our application logs:


Exception Type 1 - During Get:
---------------------------------------
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
10.1.68.36:60020 for region employeedata,be8784ac8b57c45625a03d52be981b88097c2fdc,1308657957879,
row 'd51b74eb05e07f96cee0ec556f5d8d161e3281f3', but failed after 10 attempts.
Exceptions:
java.io.IOException: Call to /10.1.68.36:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException
java.nio.channels.ClosedByInterruptException

        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:417)
     <snip>

Exception  Type 2 - During Put:
---------------------------------------------
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region
server 10.1.68.34:60020 for region audittable,,1309183872019, row '2a012017120f80a801b28f5f66a83dc2a8882d1b',
but failed after 10 attempts.
Exceptions:
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException

        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3.doCall(HConnectionManager.java:1239)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1161)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1247)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:609)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:474)
     <snip>

Any inputs on why this is happening, or how to rectify it will be of immense help.

Thanks,
Srikanth



Srikanth P Shreenivas|Principal Consultant | MindTree Ltd.|Global Village, RVCE Post, Mysore
Road, Bangalore-560 059, INDIA|Voice +91 80 26264000 / Fax +91 80 2626 4100| Mob: 9880141059|email:
srikanth_shreenivas@mindtree.com<mailto:asha_venkatesh@mindtree.com> |www.mindtree.com<http://www.mindtree.com/>
|


________________________________

http://www.mindtree.com/email/disclaimer.html

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message