Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3B945200C1F for ; Sat, 18 Feb 2017 15:40:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 38772160B66; Sat, 18 Feb 2017 14:40:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5F92C160B63 for ; Sat, 18 Feb 2017 15:40:03 +0100 (CET) Received: (qmail 93559 invoked by uid 500); 18 Feb 2017 14:40:02 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 93549 invoked by uid 99); 18 Feb 2017 14:40:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Feb 2017 14:40:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 79F03180695 for ; Sat, 18 Feb 2017 14:40:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.78 X-Spam-Level: * X-Spam-Status: No, score=1.78 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, HTML_OBFUSCATE_05_10=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id OUbim2liPGpq for ; Sat, 18 Feb 2017 14:40:00 +0000 (UTC) Received: from mail-qt0-f174.google.com (mail-qt0-f174.google.com [209.85.216.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A97045FD3B for ; Sat, 18 Feb 2017 14:39:59 +0000 (UTC) Received: by mail-qt0-f174.google.com with SMTP id b16so11167198qte.0 for ; Sat, 18 Feb 2017 06:39:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=BdbJRZrEhVUAnYkD1aXSIy1pqxAKX9kO5pL8m5u87fo=; b=198Gqnuaaam6iuYzhy8yXj3Ho1Wu507ww+H3G14oM8jmS9UdRMCaBT9HjbvHxg2GdX nQSktaWemQBSLttmH4qajHQypHEoc+e4w54tbaEQ/7cEMXmYGHKbebjSGVYWmwOT3GEc s5FBAkiszJwx13xSSe/w6drqaWQBPeR2hKY+u37boDd2oUmv9J8/6lAe4e8Qm7T8zvgp hlKijp+bOSFcuWWr6rmsah1mPkeOFs4g3NxoTYZLhaNpgbodgOnYbSvpU9sBDH14AX3i W5Bfen/q57q4q7iF4UPqLAu/hDarRU6fhQPiceEQvQHnGBWPyeA0yfUov8nn1OrwtOo6 VoGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=BdbJRZrEhVUAnYkD1aXSIy1pqxAKX9kO5pL8m5u87fo=; b=HHFBGgDgpzoHTwW4mDpnEiiu3DWLBXLKyIKBNdfLt2Vw0ZfdKjLqNYwyOvekqOCdQe qlK767evOlPXPrPk7zd6iFWmBtao5iPl07bgDtDC3LhR7HaPMCdHRiay8u1pvRLhf3F+ j/cYArdQUsLDrmmmkMmN2DYccJUurKfRVhXf+SocK7caEgUzUIq84xTn8OqC2hcD04ij mzYRByqw3bs67QcP9eWAvONV/jvYp99ABWNiINvJ7FeOA/P63hgpHdw8j5C7aFzGNsro tvuChKrb8mH8dui2Dr7VV4jwmn+7MRF+ct/H6/hL9rT8a7aKiPxnzQeZy881FR24JepL jBcw== X-Gm-Message-State: AMke39lfBcCdbJ71aGiLKNuNtcX3dWHwLX8mNwyE1VwUa6EHV3SHKmrBIoM0Rl8Z98UPidupKvAhAJ13/2PNuKxW X-Received: by 10.200.45.3 with SMTP id n3mr1815965qta.281.1487428789959; Sat, 18 Feb 2017 06:39:49 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.104.130 with HTTP; Sat, 18 Feb 2017 06:39:19 -0800 (PST) In-Reply-To: References: From: Ian Cook Date: Sat, 18 Feb 2017 09:39:19 -0500 Message-ID: Subject: Re: Need inputs on configuring hive timeout + hive on spark : Job hasn't been submitted after 61s. Aborting it. To: user@hive.apache.org Content-Type: multipart/alternative; boundary=001a1141c22ca7f4d40548cf00e3 archived-at: Sat, 18 Feb 2017 14:40:04 -0000 --001a1141c22ca7f4d40548cf00e3 Content-Type: text/plain; charset=UTF-8 Naresh, The properties hive.spark.job.monitor.timeout and hive.spark.client.server. connect.timeout in hive-site.xml control Hive on Spark timeouts. Details at https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark Ian Cook Cloudera On Thu, Feb 16, 2017 at 2:24 PM, naresh gundla wrote: > Hello, > > > i am facing this issue "Job hasn't been submitted after 61s. Aborting it." > when i am running multiple hive queries. > > Details: (Hive on Spark) > I am using spark dynamic allocation and external shuffle service (yarn) > > Assume one queries is using all of the resources in the cluster and when > the new querie launched then it throws with this error in hive log > > 2017-02-16 06:12:59,166 INFO [main]: status.SparkJobMonitor > (RemoteSparkJobMonitor.java:startMonitor(67)) -* Job hasn't been > submitted after 61s. Aborting it.* > 2017-02-16 06:12:59,166 ERROR [main]: status.SparkJobMonitor > (SessionState.java:printError(960)) - Status: SENT > 2017-02-16 06:12:59,167 INFO [main]: log.PerfLogger > (PerfLogger.java:PerfLogEnd(148)) - start=1487254318158 end=1487254379167 duration=61009 > from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor> > 2017-02-16 06:12:59,183 ERROR [main]: ql.Driver > (SessionState.java:printError(960)) - FAILED: Execution Error, return > code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask > 2017-02-16 06:12:59,184 INFO [main]: log.PerfLogger > (PerfLogger.java:PerfLogEnd(148)) - start=1487254317999 end=1487254379184 duration=61185 > from=org.apache.hadoop.hive.ql.Driver> > 2017-02-16 06:12:59,184 INFO [main]: log.PerfLogger > (PerfLogger.java:PerfLogBegin(121)) - from=org.apache.hadoop.hive.ql.Driver> > 2017-02-16 06:12:59,184 INFO [main]: log.PerfLogger > (PerfLogger.java:PerfLogEnd(148)) - start=1487254379184 end=1487254379184 duration=0 > from=org.apache.hadoop.hive.ql.Driver> > 2017-02-16 06:12:59,201 INFO [main]: log.PerfLogger > (PerfLogger.java:PerfLogBegin(121)) - from=org.apache.hadoop.hive.ql.Driver> > 2017-02-16 06:12:59,202 INFO [main]: log.PerfLogger > (PerfLogger.java:PerfLogEnd(148)) - start=1487254379201 end=1487254379202 duration=1 > from=org.apache.hadoop.hive.ql.Driver> > > Is there any parameter to config , that the the query should wait until it > get the requried resources and it should not fail. > > > Thanks, > Naresh > --001a1141c22ca7f4d40548cf00e3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Naresh,

The properties=C2=A0hive.spark.job.monitor.timeout=C2=A0= and hive.spark.client.server.connect.timeout in hive-site.xml=C2=A0control= Hive on Spark timeouts.=C2=A0Details at=C2=A0https://cwiki.apache.org/confluence/display/Hive/Configu= ration+Properties#ConfigurationProperties-Spark

Ian Cook
Cloudera

On Thu, Feb 16, 2017 at 2:24 PM, naresh gundla <nareshgundla@gmail.com> wrote:
Hello,


i am facing this issue "Job hasn&#= 39;t been submitted after 61s. Aborting it." when i am running multipl= e hive queries.

Details: (Hive on Spark)
I am using spark dynamic allocation and external shuffle service (= yarn)

Assume one queries is using all of the resources in the cluster a= nd when the new querie launched then it throws with this error in hive log<= /div>

2017-02-16 06:12:59,166 INFO =C2=A0[main]: status.SparkJobMonitor = (RemoteSparkJobMonitor.java:startMonitor(67)) -=C2=A0Job hasn't= been submitted after 61s. Aborting it.
2017-02-16 06:12:59,1= 66 ERROR [main]: status.SparkJobMonitor (SessionState.java:printError(= 960)) - Status: SENT
2017-02-16 06:12:59,167 INFO =C2=A0[main]: l= og.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method= =3DSparkRunJob start=3D1487254318158 end=3D1487254379167 duration=3D61009 f= rom=3Dorg.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor= >
2017-02-16 06:12:59,183 ERROR [main]: ql.Driver (SessionStat= e.java:printError(960)) - FAILED: Execution Error, return code 2 from = org.apache.hadoop.hive.ql.exec.spark.SparkTask
2017-02-16 06= :12:59,184 INFO =C2=A0[main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(14= 8)) - </PERFLOG method=3DDriver.execute start=3D1487254317999 end= =3D1487254379184 duration=3D61185 from=3Dorg.apache.hadoop.hive.ql.Dri= ver>
2017-02-16 06:12:59,184 INFO =C2=A0[main]: log.PerfLogger= (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=3DreleaseLoc= ks from=3Dorg.apache.hadoop.hive.ql.Driver>
2017-02-16 06= :12:59,184 INFO =C2=A0[main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(14= 8)) - </PERFLOG method=3DreleaseLocks start=3D1487254379184 end=3D1= 487254379184 duration=3D0 from=3Dorg.apache.hadoop.hive.ql.Driver><= /div>
2017-02-16 06:12:59,201 INFO =C2=A0[main]: log.PerfLogger (PerfLo= gger.java:PerfLogBegin(121)) - <PERFLOG method=3DreleaseLocks from= =3Dorg.apache.hadoop.hive.ql.Driver>
2017-02-16 06:12:59,= 202 INFO =C2=A0[main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)= ) - </PERFLOG method=3DreleaseLocks start=3D1487254379201 end=3D14872543= 79202 duration=3D1 from=3Dorg.apache.hadoop.hive.ql.Driver>

Is there any parameter to config , that the the query should wait until = it get the requried resources and it should not fail.


Thanks,
Nares= h

--001a1141c22ca7f4d40548cf00e3--