Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D007319856 for ; Wed, 27 Apr 2016 19:11:41 +0000 (UTC) Received: (qmail 18348 invoked by uid 500); 27 Apr 2016 19:11:40 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 18275 invoked by uid 500); 27 Apr 2016 19:11:40 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 18265 invoked by uid 99); 27 Apr 2016 19:11:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2016 19:11:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 952AC1A0354 for ; Wed, 27 Apr 2016 19:11:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id r5XyyL4HR-Av for ; Wed, 27 Apr 2016 19:11:35 +0000 (UTC) Received: from mail-qk0-f169.google.com (mail-qk0-f169.google.com [209.85.220.169]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 50A0E5F24E for ; Wed, 27 Apr 2016 19:11:34 +0000 (UTC) Received: by mail-qk0-f169.google.com with SMTP id r184so22933415qkc.1 for ; Wed, 27 Apr 2016 12:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=lYYTm6rpMqmEG2E2MOZr7DxHtHynNCf6b1I+P09mHP8=; b=FlL5zvu2FqmSGaqkTnP8PgxROiji+jdENs3s3QSpgZ6MxN5IInjZrwvB7SROqNN14K VIZfIjShCl8IAUR00jMceaWtQ2jzhWgsK3LYdWyVSLbVE/vWO9yOgCzgcs85rJn+5PnL tVAj5n5Zy27l7YwWv8cp4ompnX8lEue/M/7SnrTew32XptHAT93m4btyn+jbji94AsQ+ xbZSGxCkSoqL8IgydXfeGXFW5HY7inep3VkFwinmzVNqy2uUhf2Bap2NVnS5Qu80g16B tFFAMmKvWX6/ncCOZiheYqVJo8qv7SbnUqw6EgLdMvtBnSD8p9rwNGhzQTrAgBxE69HS 9H1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=lYYTm6rpMqmEG2E2MOZr7DxHtHynNCf6b1I+P09mHP8=; b=KFfFuNWNtRhox2mKmAM1+tmjtrf5arH7w5OH6uwlo0EqH2ymR2/AnCtBx2z+tYdmJa UJvndck4+vwwTT9VUlw4XFqmd/jy3UMmPk1AtQuryU915qXtihtVkXasQ1sNm7L5gdZ5 Zc76iqxsYyS8mFAmyddszXNwzTCgSy1Du9wnfreBBBkuJzm1/34wi2d6R078hF3RkEr7 +VTJjs8jUgUiPu/C6grE4I1cim8ol9l8f0ApkAhSy99qZwSVPW/BOOGE9KCAwahDMjpO v0Djot88xSa0h/72HSMMC8gS1kzKyyZFdzuSbo+N3rJt5UmSnh8nQQqpOU43AQgIJEXN i2Gg== X-Gm-Message-State: AOPr4FUulaGlmBtS02AoCPj0E+AyltVQjb2OgJpey2lnfAv1DxbiWlpilgnZkK4E3RD2VJ8xCqsIB/q1CFcoqw== MIME-Version: 1.0 X-Received: by 10.55.24.200 with SMTP id 69mr10836608qky.123.1461784286700; Wed, 27 Apr 2016 12:11:26 -0700 (PDT) Received: by 10.55.70.69 with HTTP; Wed, 27 Apr 2016 12:11:26 -0700 (PDT) In-Reply-To: References: Date: Wed, 27 Apr 2016 20:11:26 +0100 Message-ID: Subject: Re: Hive TTransportException - Create Table From: Mich Talebzadeh To: user Content-Type: multipart/alternative; boundary=001a11441e8e25afd905317c2dd5 --001a11441e8e25afd905317c2dd5 Content-Type: text/plain; charset=UTF-8 Hi, When you state: "The Queries are dependent on certain table which are created on Runtime(refreshed every hour), .." What is refreshed the whole of DDL for the same table or table is truncated? If these tables are created and dropped within the same session why not use temporary tables (private to that session). A DDL in any database is a heavy operation if you can truncate or overwrite the existing tables it would be prudent. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 27 April 2016 at 19:51, Vineet Mishra wrote: > Hi All, > > I am running Hive Server2 on a big Hadoop cluster space and things looks > good, I have a custom code which takes care of running some queries on top > of Hive and processing the resultant accordingly. > > The Queries are dependent on certain table which are created on > Runtime(refreshed every hour), since yesterday I have included some more > create statements in the code which have the dependency on the next set of > queries, to be more specific the new set of create table statements are > quite costly are spawn around 50+ MR in comparison to existing use case > where only a few 10 MR would be spawned. > > Now since when I have included these heavy DML queries (create, drop) I am > landing into Transport Exception and my HiveServer goes down(Stack Trace > mentioned below). > > java.sql.SQLException: org.apache.thrift.transport.TTransportException > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:238) > at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:390) > at > org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:117) > at > com.sd.dwh.sc.tungsten.misc.CommonFunction.executeUpdate(CommonFunction.java:816) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:95) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432) > at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.recv_ExecuteStatement(TCLIService.java:229) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216) > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232) > ... 8 more > java.sql.SQLException: Error while cleaning up the server resources > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:569) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:141) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432) > at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:183) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:170) > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:567) > ... 5 more > java.sql.SQLException: org.apache.thrift.transport.TTransportException > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:238) > at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:390) > at > org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:117) > at > com.sd.dwh.sc.tungsten.misc.CommonFunction.executeUpdate(CommonFunction.java:816) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:95) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432) > at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.recv_ExecuteStatement(TCLIService.java:229) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216) > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232) > ... 8 more > java.sql.SQLException: Error while cleaning up the server resources > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:569) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:141) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) > at > org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:480) > at > org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37) > at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:177) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:169) > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:567) > ... 5 more > Caused by: java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) > at java.net.SocketOutputStream.write(SocketOutputStream.java:159) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) > ... 11 more > java.sql.SQLException: org.apache.thrift.transport.TTransportException > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:238) > at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:390) > at > org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:117) > at > com.sd.dwh.sc.tungsten.misc.CommonFunction.executeUpdate(CommonFunction.java:816) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:95) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432) > at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.recv_ExecuteStatement(TCLIService.java:229) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216) > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232) > ... 8 more > java.sql.SQLException: Error while cleaning up the server resources > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:569) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:141) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) > at > org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:480) > at > org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37) > at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:177) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:169) > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:567) > ... 5 more > Caused by: java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) > at java.net.SocketOutputStream.write(SocketOutputStream.java:159) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) > ... 11 more > java.sql.SQLException: org.apache.thrift.transport.TTransportException > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:238) > at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:376) > at > org.apache.hive.jdbc.HivePreparedStatement.executeQuery(HivePreparedStatement.java:107) > at > com.sd.dwh.sc.tungsten.misc.CommonFunction.runQuery(CommonFunction.java:771) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:113) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:432) > at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.recv_ExecuteStatement(TCLIService.java:229) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216) > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232) > ... 8 more > java.sql.SQLException: Error while cleaning up the server resources > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:569) > at > com.sd.dwh.sc.tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:141) > at com.sd.dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) > at > org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:480) > at > org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37) > at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:177) > at > org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:169) > at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:567) > ... 5 more > Caused by: java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) > at java.net.SocketOutputStream.write(SocketOutputStream.java:159) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at > org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) > ... 11 more > > > Any suggestion would be appreciated! > > Thanks! > --001a11441e8e25afd905317c2dd5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

When you state:

"The Queries are dependent on certain table which are= created on Runtime(refreshed every hour), .."

What is refreshed the whole of DDL for the same table or table is truncat= ed?

If these tables are created and dropped within= the same session why not use temporary tables (private to that session).

A DDL in any database is a heavy operation if you c= an truncate or overwrite the existing tables it would be prudent.

HTH


On 27 April 2016 at 19:51, Vineet Mishra <clearmidoubt@gmail.com> wrote:
Hi All,

I am running Hive Serve= r2 on a big Hadoop cluster space and things looks good, I have a custom cod= e which takes care of running some queries on top of Hive and processing th= e resultant accordingly.

The Queries are dependent= on certain table which are created on Runtime(refreshed every hour), since= yesterday I have included some more create statements in the code which ha= ve the dependency on the next set of queries, to be more specific the new s= et of create table statements are quite costly are spawn around 50+ MR in c= omparison to existing use case where only a few 10 MR would be spawned.

Now since when I have included these heavy DML querie= s (create, drop) I am landing into Transport Exception and my HiveServer go= es down(Stack Trace mentioned below).=C2=A0

<= font size=3D"1">java.sql.SQLException: org.apache.thrift.transport.TTranspo= rtException
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement= .java:238)
at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStat= ement.java:390)
at org.apache.hive.jdbc.HivePreparedStatement.executeUp= date(HivePreparedStatement.java:117)
at com.sd.dwh.sc.tungsten.misc.Com= monFunction.executeUpdate(CommonFunction.java:816)
at com.sd.dwh.sc.tun= gsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:95)
= at com.sd.dwh= .sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62)
at java.util= .concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
= at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
= at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.j= ava:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.j= ava:84)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslT= ransport.java:355)
at org.apache.thrift.transport.TSaslTransport.readFr= ame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTrans= port.read(TSaslTransport.java:414)
at org.apache.thrift.transport.TSasl= ClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift= .transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.= protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apach= e.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)<= /div>
at = org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol= .java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClie= nt.java:69)
at org.apache.hive.service.cli.thrift.TCLIService$Client.re= cv_ExecuteStatement(TCLIService.java:229)
at org.apache.hive.service.cl= i.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216)
at o= rg.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232)
... 8= more
java.sql.SQLException: Error while = cleaning up the server resources
at org.apache.hive.jdbc.HiveConnection= .close(HiveConnection.java:569)
at com.sd.dwh.sc.tungsten.custom.HiveRu= nnable.mergeJDBC(HiveRunnable.java:141)
<= span style=3D"white-space:pre-wrap"> at com.sd.dwh.sc.tungsten.custo= m.HiveRunnable.run(HiveRunnable.java:62)
= at java.util.concurrent.Thread= PoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concu= rrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at jav= a.lang.Thread.run(Thread.java:745)
Caused= by: org.apache.thrift.transport.TTransportException
at org.apache.thri= ft.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at or= g.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org= .apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:355)=
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport= .java:432)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTran= sport.java:414)
at org.apache.thrift.transport.TSaslClientTransport.rea= d(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTranspo= rt.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryPro= tocol.readAll(TBinaryProtocol.java:378)
<= span style=3D"white-space:pre-wrap"> at org.apache.thrift.protocol.T= BinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.p= rotocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at o= rg.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)<= /div>
at = org.apache.hive.service.cli.thrift.TCLIService$Client.recv_CloseSession(TCL= IService.java:183)
at org.apache.hive.service.cli.thrift.TCLIService$Cl= ient.CloseSession(TCLIService.java:170)
<= span style=3D"white-space:pre-wrap"> at org.apache.hive.jdbc.HiveCon= nection.close(HiveConnection.java:567)
... 5 more
java.sql.SQLException: org.apache.thrift.transport.TTransportE= xception
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.ja= va:238)
at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStateme= nt.java:390)
at org.apache.hive.jdbc.HivePreparedStatement.executeUpdat= e(HivePreparedStatement.java:117)
at com.sd.dwh.sc.tungsten.misc.Common= Function.executeUpdate(CommonFunction.java:816)
at com.sd.dwh.sc.tungst= en.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:95)
at com.sd.dwh.sc= .tungsten.custom.HiveRunnable.run(HiveRunnable.java:62)
at java.util.co= ncurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<= /div>
at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException<= /div>
at = org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java= :132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java= :84)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTran= sport.java:355)
at org.apache.thrift.transport.TSaslTransport.readFrame= (TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTransport= .read(TSaslTransport.java:414)
at org.apache.thrift.transport.TSaslClie= ntTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.tr= ansport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.pr= otocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
<= font size=3D"1"> at org.apache.= thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at or= g.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.j= ava:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient= .java:69)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv= _ExecuteStatement(TCLIService.java:229)
<= span style=3D"white-space:pre-wrap"> at org.apache.hive.service.cli.= thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216)
at org= .apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232)
... 8 m= ore
java.sql.SQLException: Error while cl= eaning up the server resources
at org.apache.hive.jdbc.HiveConnection.c= lose(HiveConnection.java:569)
at com.sd.dwh.sc.tungsten.custom.HiveRunn= able.mergeJDBC(HiveRunnable.java:141)
at com.sd.dwh.sc.tungsten.custom.= HiveRunnable.run(HiveRunnable.java:62)
at java.util.concurrent.ThreadPo= olExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurr= ent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
=
at java.= lang.Thread.run(Thread.java:745)
Caused b= y: org.apache.thrift.transport.TTransportException: java.net.SocketExceptio= n: Broken pipe
at org.apache.thrift.transport.TIOStreamTransport.flush(= TIOStreamTransport.java:161)
at org.apache.thrift.transport.TSaslTrans= port.flush(TSaslTransport.java:480)
at org.apache.thrift.transport.TSas= lClientTransport.flush(TSaslClientTransport.java:37)
at org.apache.thri= ft.TServiceClient.sendBase(TServiceClient.java:65)
at org.apache.hive.s= ervice.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:177= )
<= /span>at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession= (TCLIService.java:169)
at org.apache.hive.jdbc.HiveConnection.close(Hiv= eConnection.java:567)
... 5 more
Caus= ed by: java.net.SocketException: Broken pipe
at java.net.SocketOutputSt= ream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketW= rite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(= SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushBuff= er(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(= BufferedOutputStream.java:140)
at org.apache.thrift.transport.TIOStream= Transport.flush(TIOStreamTransport.java:159)
... 11 more
java.sql.SQLException: org.apache.thrift.transport.TTra= nsportException
at org.apache.hive.jdbc.HiveStatement.execute(HiveState= ment.java:238)
at org.apache.hive.jdbc.HiveStatement.executeUpdate(Hive= Statement.java:390)
at org.apache.hive.jdbc.HivePreparedStatement.execu= teUpdate(HivePreparedStatement.java:117)
= at com.sd.dwh.sc.tungsten.misc= .CommonFunction.executeUpdate(CommonFunction.java:816)
at com.sd.dwh.sc= .tungsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:95)
<= div> at com.sd= .dwh.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62)
=
at java.= util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<= /font>
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:615)
at java.lang.Thread.run(Thread.java:745)
<= font size=3D"1">Caused by: org.apache.thrift.transport.TTransportException<= /font>
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTranspo= rt.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTranspo= rt.java:84)
at org.apache.thrift.transport.TSaslTransport.readLength(TS= aslTransport.java:355)
at org.apache.thrift.transport.TSaslTransport.re= adFrame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTr= ansport.read(TSaslTransport.java:414)
at org.apache.thrift.transport.TS= aslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thr= ift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thri= ft.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
<= div> at org.ap= ache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
= at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProto= col.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceC= lient.java:69)
at org.apache.hive.service.cli.thrift.TCLIService$Client= .recv_ExecuteStatement(TCLIService.java:229)
at org.apache.hive.service= .cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216)
a= t org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232)=
..= . 8 more
java.sql.SQLException: Error whi= le cleaning up the server resources
at org.apache.hive.jdbc.HiveConnect= ion.close(HiveConnection.java:569)
at com.sd.dwh.sc.tungsten.custom.Hiv= eRunnable.mergeJDBC(HiveRunnable.java:141)
at com.sd.dwh.sc.tungsten.cu= stom.HiveRunnable.run(HiveRunnable.java:62)
at java.util.concurrent.Thr= eadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.co= ncurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<= /div>
at = java.lang.Thread.run(Thread.java:745)
Cau= sed by: org.apache.thrift.transport.TTransportException: java.net.SocketExc= eption: Broken pipe
at org.apache.thrift.transport.TIOStreamTransport.f= lush(TIOStreamTransport.java:161)
at org.apache.thrift.transport.TSaslT= ransport.flush(TSaslTransport.java:480)
<= span style=3D"white-space:pre-wrap"> at org.apache.thrift.transport.= TSaslClientTransport.flush(TSaslClientTransport.java:37)
<= font size=3D"1"> at org.apache.= thrift.TServiceClient.sendBase(TServiceClient.java:65)
at org.apache.hi= ve.service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java= :177)
at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSes= sion(TCLIService.java:169)
at org.apache.hive.jdbc.HiveConnection.close= (HiveConnection.java:567)
... 5 more
= Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutp= utStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.soc= ketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.wr= ite(SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushB= uffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flu= sh(BufferedOutputStream.java:140)
at org.apache.thrift.transport.TIOStr= eamTransport.flush(TIOStreamTransport.java:159)
... 11 more
java.sql.SQLException: org.apache.thrift.transport.T= TransportException
at org.apache.hive.jdbc.HiveStatement.execute(HiveSt= atement.java:238)
at org.apache.hive.jdbc.HiveStatement.executeQuery(Hi= veStatement.java:376)
at org.apache.hive.jdbc.HivePreparedStatement.exe= cuteQuery(HivePreparedStatement.java:107)
at com.sd.dwh.sc.tungsten.mis= c.CommonFunction.runQuery(CommonFunction.java:771)
at com.sd.dwh.sc.tun= gsten.custom.HiveRunnable.mergeJDBC(HiveRunnable.java:113)
at com.sd.dw= h.sc.tungsten.custom.HiveRunnable.run(HiveRunnable.java:62)
at java.uti= l.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j= ava:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.= java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.= java:84)
at org.apache.thrift.transport.TSaslTransport.readLength(TSasl= Transport.java:355)
at org.apache.thrift.transport.TSaslTransport.readF= rame(TSaslTransport.java:432)
at org.apache.thrift.transport.TSaslTrans= port.read(TSaslTransport.java:414)
at org.apache.thrift.transport.TSasl= ClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift= .transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.= protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apach= e.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)<= /div>
at = org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol= .java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClie= nt.java:69)
at org.apache.hive.service.cli.thrift.TCLIService$Client.re= cv_ExecuteStatement(TCLIService.java:229)
at org.apache.hive.service.cl= i.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:216)
at o= rg.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:232)
... 8= more
java.sql.SQLException: Error while = cleaning up the server resources
at org.apache.hive.jdbc.HiveConnection= .close(HiveConnection.java:569)
at com.sd.dwh.sc.tungsten.custom.HiveRu= nnable.mergeJDBC(HiveRunnable.java:141)
<= span style=3D"white-space:pre-wrap"> at com.sd.dwh.sc.tungsten.custo= m.HiveRunnable.run(HiveRunnable.java:62)
= at java.util.concurrent.Thread= PoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concu= rrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at jav= a.lang.Thread.run(Thread.java:745)
Caused= by: org.apache.thrift.transport.TTransportException: java.net.SocketExcept= ion: Broken pipe
at org.apache.thrift.transport.TIOStreamTransport.flus= h(TIOStreamTransport.java:161)
at org.apache.thrift.transport.TSaslTran= sport.flush(TSaslTransport.java:480)
at org.apache.thrift.transport.TSa= slClientTransport.flush(TSaslClientTransport.java:37)
at org.apache.thr= ift.TServiceClient.sendBase(TServiceClient.java:65)
at org.apache.hive.= service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:17= 7)
= at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSessio= n(TCLIService.java:169)
at org.apache.hive.jdbc.HiveConnection.close(Hi= veConnection.java:567)
... 5 more
Cau= sed by: java.net.SocketException: Broken pipe
at java.net.SocketOutpu= tStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.sock= etWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.wri= te(SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushBu= ffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flus= h(BufferedOutputStream.java:140)
at org.apache.thrift.transport.TIOStre= amTransport.flush(TIOStreamTransport.java:159)
... 11 more


Any suggestion would be appreciat= ed!

Thanks!

--001a11441e8e25afd905317c2dd5--