tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyunsik Choi <hyun...@apache.org>
Subject Re: Tajo with TPC-H benchmark
Date Fri, 28 Nov 2014 05:45:00 GMT
I just heard some information from Jinho. He experienced such a
problem. If you set some config to enable Tajo catalog to access Hive
Meta store, it uses thrift server of Hive. According to his advice, it
causes such a problem frequently. This is a known problem in hive, but
there is a workaround.

The workaround manner is to copy 'hive-site.xml' of your hive to
${TAJO_HOME}/conf. Then, Tajo catalog directly accesses the persistent
RDBMS being used by Hive instead of connecting Hive meta store via
Thrift.

Probably, It will solve your problem.

Best regards,
Hyunsik

On Fri, Nov 28, 2014 at 1:10 PM, Thanh Mai <maihaithanh@gmail.com> wrote:
> Hi Hyunsik,
>
> I have already modified TPC-H queries to be able to run with Tajo. If the
> queries are issued sequentially, there is no problem. However, if they are
> issued concurrently, some queries will fail sometimes (not always).
>
> I cannot share the log file. Instead, I can share the error messages below.
> It seems that the problem is not directly related to Tajo, but to Thrift or
> ProtoBuf. I have also increased the value of
> "hive.metastore.client.socket.timeout" from 20 to 200 and
> "hive.server.read.socket.timeout" from 10 to 100 in hive-site.xml, but the
> problem remains.
>
> Thank you for your help!
>
> Sincerely,
>
> Mai Hai Thanh
>
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> 2014-11-28 11:09:38,535 ERROR hive.log: Got exception:
> org.apache.thrift.transport.TTransportException
> java.net.SocketTimeoutException: Read timed out
> org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
>         at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>         at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>         at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:354)
>         at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
>         at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>         at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_databases(ThriftHiveMetastore.java:625)
>         at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_databases(ThriftHiveMetastore.java:613)
>         at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:837)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>         at com.sun.proxy.$Proxy12.getAllDatabases(Unknown Source)
>         at
> org.apache.tajo.catalog.store.HCatalogStore.existDatabase(HCatalogStore.java:369)
>         at
> org.apache.tajo.catalog.CatalogServer$CatalogProtocolHandler.existDatabase(CatalogServer.java:414)
>         at
> org.apache.tajo.catalog.AbstractCatalogClient$9.call(AbstractCatalogClient.java:215)
>         at
> org.apache.tajo.catalog.AbstractCatalogClient$9.call(AbstractCatalogClient.java:212)
>         at
> org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:95)
>         at
> org.apache.tajo.catalog.AbstractCatalogClient.existDatabase(AbstractCatalogClient.java:212)
>         at
> org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.existDatabase(TajoMasterClientService.java:624)
>         at
> org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:549)
>         at
> org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:103)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>         at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>         at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>         at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>         at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>         at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:152)
>         at java.net.SocketInputStream.read(SocketInputStream.java:122)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>         at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>         ... 47 more
>
> 2014-11-28 11:09:38,538 ERROR hive.log: Converting exception to
> MetaException
> 2014-11-28 11:09:38,550 ERROR org.apache.tajo.catalog.CatalogServer:
> org.apache.tajo.catalog.exception.CatalogException:
> MetaException(message:Got exception:
> org.apache.thrift.transport.TTransportException
> java.net.SocketTimeoutException: Read timed out)
> 2014-11-28 11:09:38,550 ERROR
> org.apache.tajo.catalog.AbstractCatalogClient:
> com.google.protobuf.ServiceException:
> org.apache.tajo.catalog.exception.CatalogException:
> MetaException(message:Got exception:
> org.apache.thrift.transport.TTransportException
> java.net.SocketTimeoutException: Read timed out)
> com.google.protobuf.ServiceException: com.google.protobuf.ServiceException:
> org.apache.tajo.catalog.exception.CatalogException:
> MetaException(message:Got exception:
> org.apache.thrift.transport.TTransportException
> java.net.SocketTimeoutException: Read timed out)
>         at
> org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:105)
>         at
> org.apache.tajo.catalog.AbstractCatalogClient.existDatabase(AbstractCatalogClient.java:212)
>         at
> org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.existDatabase(TajoMasterClientService.java:624)
>         at
> org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:549)
>         at
> org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:103)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>         at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>         at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>         at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>         at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>         at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: com.google.protobuf.ServiceException:
> org.apache.tajo.catalog.exception.CatalogException:
> MetaException(message:Got exception:
> org.apache.thrift.transport.TTransportException
> java.net.SocketTimeoutException: Read timed out)
>         at
> org.apache.tajo.catalog.CatalogServer$CatalogProtocolHandler.existDatabase(CatalogServer.java:421)
>         at
> org.apache.tajo.catalog.AbstractCatalogClient$9.call(AbstractCatalogClient.java:215)
>         at
> org.apache.tajo.catalog.AbstractCatalogClient$9.call(AbstractCatalogClient.java:212)
>         at
> org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:95)
>         ... 30 more
> Caused by: org.apache.tajo.catalog.exception.CatalogException:
> MetaException(message:Got exception:
> org.apache.thrift.transport.TTransportException
> java.net.SocketTimeoutException: Read timed out)
>         at
> org.apache.tajo.catalog.store.HCatalogStore.existDatabase(HCatalogStore.java:372)
>         at
> org.apache.tajo.catalog.CatalogServer$CatalogProtocolHandler.existDatabase(CatalogServer.java:414)
>         ... 33 more
> Caused by: MetaException(message:Got exception:
> org.apache.thrift.transport.TTransportException
> java.net.SocketTimeoutException: Read timed out)
>         at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1102)
>         at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:839)
>         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>         at com.sun.proxy.$Proxy12.getAllDatabases(Unknown Source)
>         at
> org.apache.tajo.catalog.store.HCatalogStore.existDatabase(HCatalogStore.java:369)
>         ... 34 more
>
>
> ERROR: Exception was thrown. Caused by
> org.apache.tajo.cli.InvalidClientSessionException: ERROR: database
> "default" does not exist
> Exception in thread "main" com.google.protobuf.ServiceException:
> org.apache.tajo.cli.InvalidClientSessionException: ERROR: database
> "default" does not exist
>         at
> org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:105)
>         at
> org.apache.tajo.client.TajoClient.getCurrentDatabase(TajoClient.java:262)
>         at org.apache.tajo.cli.TajoCli.<init>(TajoCli.java:243)
>         at org.apache.tajo.cli.TajoCli.main(TajoCli.java:690)
> Caused by: org.apache.tajo.cli.InvalidClientSessionException: ERROR:
> database "default" does not exist
>         at
> org.apache.tajo.client.TajoClient.checkSessionAndGet(TajoClient.java:249)
>         at org.apache.tajo.client.TajoClient.access$000(TajoClient.java:75)
>         at org.apache.tajo.client.TajoClient$1.call(TajoClient.java:265)
>         at org.apache.tajo.client.TajoClient$1.call(TajoClient.java:262)
>         at
> org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:95)
>         ... 3 more
>
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> On Fri, Nov 28, 2014 at 10:36 AM, Hyunsik Choi <hyunsik@apache.org> wrote:
>
>> Hi Mai,
>>
>> Good to see you again!
>>
>> So far, we haven't performed QphH-based benchmark.
>>
>> Some of original TPC-H benchmark queries may not work in Tajo because
>> scalar subquery, in-subquery, exist-subquery are not supported yet.
>> So, you need to change slightly them into multiple queries.
>>
>> I haven't experienced such a  situation you explained, could you share
>> the error log that you met while you were executing multiple queries?
>>
>> Best regards,
>> Hyunsik
>>
>>
>>
>> On Tue, Nov 25, 2014 at 12:12 PM, Thanh Mai <maihaithanh@gmail.com> wrote:
>> > Hi everyone,
>> >
>> > I want to find the performance results of Tajo with the TPC-H benchmark
>> in
>> > both power test and throughput test (and thus, a sample QphH value for
>> > reference). However, from the slides in Tajo website, I found only the
>> > response times of some queries.
>> >
>> > Have any of you tested Tajo in the TPC-H benchmark's throughput test or
>> in
>> > the multi-user situation ? If yes, could you share your experience ?
>> >
>> > When I test Tajo with the TPC-H benchmark's throughput test, some queries
>> > failed occasionally. I use Hive catalog instead of Tajo catalog. It seems
>> > that Tajo has a problem when getting information from Hive catalog
>> > concurrently for multiple queries running in parallel. Have any of you
>> > experienced a similar situation ? Could you share your experience and
>> > insights ?
>> >
>> > Thank you very much!
>> >
>> > Sincerely,
>> >
>> > Mai Hai Thanh
>>

Mime
View raw message