hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sutton, Ben" <Ben.Sut...@virginmedia.co.uk>
Subject RE: Slow response on HBase REST api using globbing option
Date Thu, 03 Dec 2015 21:12:05 GMT
We are currently experiencing an issue with HBase 1.1.1 through the REST API whereby we are
getting a very slow (> 2minutes) /non-existent response when hitting the Rest API with
a globbing option i.e. http://<HBase_Rest>:<HBase_Rest_Port/table/key*<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>



We are able to return each individual row for the key with no problem i.e. http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01>
or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02>
etc.

We have also found if we load in a third of the data to hbase we have no trouble with the
above.

Facing this issue with 1.7billion row hbase table - we are able to retrieve data for initial
rows keys through to 1/10 of the region with a slowly degrading performance after this point
until no result.

This is with 19 regionservers with each having at least 5 regions.

Seeing the below error:
Failed after attempts=36, exceptions:
Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303:
row '$ROW' on table '$TABLE' at region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36.,
hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
(through reference chain: org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213] mortbay.log: /$TABLE/KEY.*
java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=36, exceptions:
Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303:
row $ROW on table $TABLE at region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36.,
hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093

        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
        at com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
        at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
        at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
        at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
        at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
        at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
        at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36,
exceptions:

Anyone have any ideas?

Thanks,

Ben


--------------------------------------------------------------------
Save Paper - Do you really need to print this e-mail?

Visit www.virginmedia.com for more information, and more fun.

This email and any attachments are or may be confidential and legally privileged
and are sent solely for the attention of the addressee(s). If you have received this
email in error, please delete it from your system: its use, disclosure or copying is
unauthorised. Statements and opinions expressed in this email may not represent
those of Virgin Media. Any representations or commitments in this email are
subject to contract. 

Registered office: Media House, Bartley Wood Business Park, Hook, Hampshire, RG27 9UP
Registered in England and Wales with number 2591237

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message