Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 338B7F4CB for ; Wed, 17 Jul 2013 03:04:59 +0000 (UTC) Received: (qmail 38565 invoked by uid 500); 17 Jul 2013 03:04:56 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 38522 invoked by uid 500); 17 Jul 2013 03:04:55 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 38514 invoked by uid 99); 17 Jul 2013 03:04:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jul 2013 03:04:54 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bruderman@radiumone.com designates 209.85.223.177 as permitted sender) Received: from [209.85.223.177] (HELO mail-ie0-f177.google.com) (209.85.223.177) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jul 2013 03:04:50 +0000 Received: by mail-ie0-f177.google.com with SMTP id aq17so3049744iec.36 for ; Tue, 16 Jul 2013 20:04:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=WpHC/JvOn2sfvJGMpIjHWEOD+8FBZXn+FTs8T8YiLMw=; b=Sm+rbI5wOIKbGUXN4F1P1+rpmyPMASdO5nRXI/bQpsGw51pZSNXVgS65ZCHhrK/gWw P4uHKbAhSimlJSL22n26+RQOVJ6AcT7RmEeuZ2yTiBpra0UXWOFmlxuYTdFxLO/vfFlH Bc4/bzDO08qZCctKBhJENHvckbhNzSTaeq3J3Y5NYKIP31SwarulbryP40Cu1GT3CxK2 IfxtY3cD7NtyM8I9cpF1+GDpGEtU6RXk3wCMp5O8sWZJIQxKDS5Z490bZI1qoPKP+z0k HWc4CUAIskG5bXT+I1wtNN1bfVfDISQxOQJ3T3qICYp4dyA/SF3pkgXslRxe19SaNQIc PL9A== MIME-Version: 1.0 X-Received: by 10.43.88.3 with SMTP id ay3mr3761492icc.61.1374030269952; Tue, 16 Jul 2013 20:04:29 -0700 (PDT) Received: by 10.50.51.234 with HTTP; Tue, 16 Jul 2013 20:04:29 -0700 (PDT) In-Reply-To: References: <1373994716.99042.YahooMailNeo@web142301.mail.bf1.yahoo.com> <128c9eca.1772.13fea822acc.Coremail.kentkong_work@163.com> Date: Tue, 16 Jul 2013 20:04:29 -0700 Message-ID: Subject: Re: how to know a hive query failed From: Brad Ruderman To: user@hive.apache.org Content-Type: multipart/alternative; boundary=001a11c21c9626c67204e1ac5aeb X-Gm-Message-State: ALoCoQk1cMFFAOEUc9EZNVAxNYiYObAE1nM+ea5O+fpvtEDZ3aXVTbiAdNvCcaWuQGdpeMkIQrrk X-Virus-Checked: Checked by ClamAV on apache.org --001a11c21c9626c67204e1ac5aeb Content-Type: text/plain; charset=ISO-8859-1 Sry - sent email prematurely. You need to poll and read the stderr and stdout for text messages alerting there is an error. In python this is what I use: def execute_hive_query(query, hive_exception = None): return_code = None cmd = ["hive","-S", "-e", query] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) while return_code is None: time.sleep(10) return_code = proc.poll() out = proc.stdout.read() error = proc.stderr.read() handle_hive_exception(out,error,hive_exception) def handle_hive_exception(stdout,stderr,hive_exception = None): if hive_exception is not None: hive_exception(stdout,stderr) else: if stderr != '': stderr = stderr.lower() if stderr.find('error') > -1 or stderr.find('failed') >-1: if stderr.find('log4j:error could not find value for key log4j.appender.fa') == -1: raise Exception(stderr) return On Tue, Jul 16, 2013 at 8:03 PM, Brad Ruderman wrote: > You need to stream and read the stderr and stdout for text messages > alerting there is an error. In python this is what I use: > > > On Tue, Jul 16, 2013 at 7:42 PM, kentkong_work wrote: > >> ** >> hi, >> I use a shell script to run hive query in background, like this >> hive -e "select uname, age from usertable" >> result.csv & >> >> sometimes, the hive query failed, but I can't find. >> only thing I can do is waiting for long time, 5 or 6 hours, if I >> still can't see the result.csv, I know the query failed. >> is there some way to let hive tell me the query failed? >> >> >> Kent >> > > --001a11c21c9626c67204e1ac5aeb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Sry - sent email prematurely.

You need to poll and read the= stderr and stdout for text messages alerting there is an error. In python = this is what I use:

def execute_hive_query(query, hive_exception =3D = None):
return_= code =3D None
= cmd =3D ["hive","-S", "-e", query]
proc =3D subprocess= .Popen(cmd, stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.PIPE)
<= span class=3D"" style=3D"white-space:pre"> while return_code is None= :
time.sleep(10)
return_code =3D proc.= poll()
out = =3D proc.stdout.read()
error =3D proc.std= err.read()
ha= ndle_hive_exception(out,error,hive_exception)

def = handle_hive_exception(stdout,stderr,hive_exception =3D None):
if hive_exception i= s not None:
h= ive_exception(stdout,stderr)
else:
if stderr !=3D = 9;':
std= err =3D stderr.lower()
if stderr.find('error') > -1 or stderr.find('fail= ed') >-1:
if stderr.find(&= #39;log4j:error could not find value for key log4j.appender.fa') =3D=3D= -1:
raise= Exception(stderr)
return
=


On Tue, Jul 1= 6, 2013 at 8:03 PM, Brad Ruderman <bruderman@radiumone.com> wrote:
You need to stream and read= the stderr and stdout for text messages alerting there is an error. In pyt= hon this is what I use:


=
On Tue, Jul 16, 2013 at 7:42 PM, kentkong_work <= span dir=3D"ltr"><kentkong_work@163.com> wrote:
hi,
=C2=A0=C2=A0=C2=A0 I=C2=A0use a shell script to run hive query in=20 background, like this
=C2=A0=C2=A0=C2=A0 hive -e "select=C2=A0uname, age=C2=A0from user= table"=20 >> result.csv &
=C2=A0
=C2=A0=C2=A0=C2=A0 sometimes, the hive query failed, but I can't= =20 find.
=C2=A0=C2=A0=C2=A0 only thing I can do is waiting for long time, 5 or = 6=20 hours, if I still can't see the result.csv, I know the query failed.
=C2=A0=C2=A0=C2=A0 is there some way to let hive tell me the query=20 failed?
=C2=A0
=C2=A0
Kent


--001a11c21c9626c67204e1ac5aeb--