Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: "Sutton, Ben" <Ben.Sutton@virginmedia.co.uk>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: RE: Slow response on HBase REST api using globbing option
Thread-Topic: Slow response on HBase REST api using globbing option
Thread-Index: AdEuA7s4XkoqZpSZTSen7Cg32OSFuwAC3eEQ
Date: Thu, 3 Dec 2015 21:12:05 +0000
Message-ID: 
 <C4C4C09FB3306A4B919E56C7B35766D8A40B373D@KN2-MBX-P0004.systems.private>
References: 
 <C4C4C09FB3306A4B919E56C7B35766D8A40B3670@KN2-MBX-P0004.systems.private>
In-Reply-To: 
 <C4C4C09FB3306A4B919E56C7B35766D8A40B3670@KN2-MBX-P0004.systems.private>
Accept-Language: en-GB, en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_C4C4C09FB3306A4B919E56C7B35766D8A40B373DKN2MBXP0004syst_"
MIME-Version: 1.0

--_000_C4C4C09FB3306A4B919E56C7B35766D8A40B373DKN2MBXP0004syst_
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

We are currently experiencing an issue with HBase 1.1.1 through the REST AP=
I whereby we are getting a very slow (> 2minutes) /non-existent response wh=
en hitting the Rest API with a globbing option i.e. http://<HBase_Rest>:<HB=
ase_Rest_Port/table/key*<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/k=
ey*>


We are able to return each individual row for the key with no problem i.e. =
http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://%3cHBase_Rest%3e:%3=
cHBase_Rest_Port/table/key01> or http://<HBase_Rest>:<HBase_Rest_Port/table=
/key02<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02> etc.

We have also found if we load in a third of the data to hbase we have no tr=
ouble with the above.

Facing this issue with 1.7billion row hbase table - we are able to retrieve=
 data for initial rows keys through to 1/10 of the region with a slowly deg=
rading performance after this point until no result.

This is with 19 regionservers with each having at least 5 regions.

Seeing the below error:
Failed after attempts=3D36, exceptions:
Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: callTi=
meout=3D60000, callDuration=3D60303: row '$ROW' on table '$TABLE' at region=
=3D$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36., hostname=3D=
$LOCAL_HOST,60020,1448892673650, seqNum=3D444047093
(through reference chain: org.apache.hadoop.hbase.rest.CellSetModelStream["=
Row"])
2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213] mortbay.log: /=
$TABLE/KEY.*
java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhausted=
Exception: Failed after attempts=3D36, exceptions:
Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: callTi=
meout=3D60000, callDuration=3D60303: row $ROW on table $TABLE at region=3D$=
TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36., hos=
tname=3D$LOCAL_HOST,60020,1448892673650, seqNum=3D444047093

        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(A=
bstractClientScanner.java:97)
        at org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(Table=
ScanResource.java:82)
        at com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.ha=
sNext(Lister.java:285)
        at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serial=
izeListBody(ArrayElementProperty.java:137)
        at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBo=
dy(ArrayERProperty.java:159)
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(Clas=
sBeanInfoImpl.java:344)
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XML=
Serializer.java:597)
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(Clas=
sBeanInfoImpl.java:328)
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSeriali=
zer.java:498)
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.=
java:320)
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImp=
l.java:249)
        at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMa=
rshallerImpl.java:95)
        at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.wr=
iteTo(AbstractRootElementProvider.java:179)
        at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.wr=
iteTo(AbstractRootElementProvider.java:157)
        at com.sun.jersey.spi.container.ContainerResponse.write(ContainerRe=
sponse.java:306)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handl=
eRequest(WebApplicationImpl.java:1437)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handle=
Request(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handle=
Request(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebCom=
ponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(Se=
rvletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(Se=
rvletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.jav=
a:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Se=
rvletHandler.java:1221)
        at org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilt=
er.java:76)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Se=
rvletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.j=
ava:399)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.j=
ava:182)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandle=
r.java:216)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.j=
ava:767)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.j=
ava:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.ja=
va:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(H=
ttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoi=
nt.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadP=
ool.java:582)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed=
 after attempts=3D36, exceptions:

Anyone have any ideas?

Thanks,

Ben


--------------------------------------------------------------------
Save Paper - Do you really need to print this e-mail?

Visit www.virginmedia.com for more information, and more fun.

This email and any attachments are or may be confidential and legally privi=
leged
and are sent solely for the attention of the addressee(s). If you have rece=
ived this
email in error, please delete it from your system: its use, disclosure or c=
opying is
unauthorised. Statements and opinions expressed in this email may not repre=
sent
those of Virgin Media. Any representations or commitments in this email are
subject to contract. =


Registered office: Media House, Bartley Wood Business Park, Hook, Hampshire=
, RG27 9UP
Registered in England and Wales with number 2591237

--_000_C4C4C09FB3306A4B919E56C7B35766D8A40B373DKN2MBXP0004syst_--