Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E2129D6C0 for ; Sat, 11 Aug 2012 07:49:53 +0000 (UTC) Received: (qmail 59632 invoked by uid 500); 11 Aug 2012 07:49:52 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 59410 invoked by uid 500); 11 Aug 2012 07:49:49 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 59391 invoked by uid 99); 11 Aug 2012 07:49:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Aug 2012 07:49:49 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jagatsingh@gmail.com designates 209.85.214.176 as permitted sender) Received: from [209.85.214.176] (HELO mail-ob0-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Aug 2012 07:49:41 +0000 Received: by obbtb18 with SMTP id tb18so4366490obb.35 for ; Sat, 11 Aug 2012 00:49:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=pW4Le2xGqhJqRMFVmNsZ22L9WTRXMg71+mAkHtDzWMg=; b=v8jkNKVIy9EUb7NIX9VkgZiRSuXKRQN/ahBBJ8jUfGoC7Ndgni7UosAz3AzMwmERy2 cU0U26r+Pvtyj7YXwfD2JJB2mUQ4DNu0FdCz2Y0KlU0Zase//XRe1xvbUSlIw35t5D8j /zw0Ez9e5C65iDl3shki82iyY4zyfRFdoaeedFDp0dKk1gu4tARRR/8+O1Qh3Rfq6fZ0 bzVZMOPeFQ5wH9V1HNDhQnF5OS215si1cUAQ/asLOvyhKGDeCD9KJi26qMtKOMU2QkPj j6MWaM2DB8tNwbFtwUAL4opXCxxPYYSIjyOVCcqXcAIxDaoOZYGcrttta2G3AJmgvcQa PUAg== MIME-Version: 1.0 Received: by 10.50.94.195 with SMTP id de3mr511888igb.68.1344671360552; Sat, 11 Aug 2012 00:49:20 -0700 (PDT) Received: by 10.64.63.196 with HTTP; Sat, 11 Aug 2012 00:49:20 -0700 (PDT) Received: by 10.64.63.196 with HTTP; Sat, 11 Aug 2012 00:49:20 -0700 (PDT) In-Reply-To: References: Date: Sat, 11 Aug 2012 13:19:20 +0530 Message-ID: Subject: Re: Hive job fails on hive client even though all map-red stages finish but succeeds on hive server From: Jagat Singh To: user@hive.apache.org Content-Type: multipart/alternative; boundary=e89a8f235443c90fd304c6f8b24a --e89a8f235443c90fd304c6f8b24a Content-Type: text/plain; charset=ISO-8859-1 Hi Anurag, How much space is for /user and /tmp directory on client. Did you check that part? , anything which might stop move task from finishing. ----------- Sent from Mobile , short and crisp. On 11-Aug-2012 1:37 PM, "Anurag Tangri" wrote: > Hi, > We are facing this issue where we run a hive job over huge data about ~6 > TB input. > > We run this from hive client and hive metastore server is on another > machine. > > > If we have smaller input, this job succeeds but for above input size, it > fails with error : > > 2012-08-11 01:34:01,722 Stage-1 map = 100%, reduce = 100% > > 2012-08-11 01:35:02,195 Stage-1 map = 100%, reduce = 100% > > 2012-08-11 01:36:02,682 Stage-1 map = 100%, reduce = 100% > > 2012-08-11 01:37:03,215 Stage-1 map = 100%, reduce = 100% > > 2012-08-11 01:38:03,719 Stage-1 map = 100%, reduce = 100% > > 2012-08-11 01:39:04,311 Stage-1 map = 100%, reduce = 100% > > Ended Job = job_201207072204_34432 > > Loading data to table default.atangri_test_1 > > Failed with exception Unable to fetch table atangri_test_1 > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.MoveTask > > > If we have smaller input (~2 TB), this job succeeds but for above input > size, it fails with error : We have set > hive.metastore.client.socket.timeout to big value like 86400 but still it > fails after about 8-9 hours. > > Does anyone face the same issue or any pointers ? > > The job succeeds if it is directly run on hive server. > > Thanks, > Anurag Tangri > --e89a8f235443c90fd304c6f8b24a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Hi Anurag,

How much space is for /user and /tmp directory on client.

Did you check that part? , anything which might stop move task from fini= shing.

-----------
Sent from Mobile , short and crisp.

On 11-Aug-2012 1:37 PM, "Anurag Tangri"= ; <tangri.anurag@gmail.com> wrote:
Hi,
We are facing this issue where we run a hive job over huge data abou= t ~6 TB input.

We run this from hive client and hive metastore serve= r is on another machine.


If we have smaller input, this job succ= eeds but for above input size, it fails with error :

2012-08-11 01:34:01,722 S= tage-1 map =3D 100%,=A0 reduce =3D 100%

2012-08-11 01:35:02,195 S= tage-1 map =3D 100%,=A0 reduce =3D 100%

2012-08-11 01:36:02,682 S= tage-1 map =3D 100%,=A0 reduce =3D 100%

2012-08-11 01:37:03,215 S= tage-1 map =3D 100%,=A0 reduce =3D 100%

2012-08-11 01:38:03,719 S= tage-1 map =3D 100%,=A0 reduce =3D 100%

2012-08-11 01:39:04,311 S= tage-1 map =3D 100%,=A0 reduce =3D 100%

Ended Job =3D job_2012070= 72204_34432

Loading data to table def= ault.atangri_test_1

Failed with exception Una= ble to fetch table atangri_test_1

FAILED: Execution Error, = return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask



If we have smaller input (~2 TB), this job succeeds but for above i= nput size, it fails with error : We have set hive.metastore.client.socket.t= imeout to big value like 86400=A0 but still it fails after about 8-9 hours.=

Does anyone face the same issue or any pointers ?

The job succee= ds if it is directly run on hive server.

Thanks,
Anurag Tangri
--e89a8f235443c90fd304c6f8b24a--