Return-Path: Delivered-To: apmail-hadoop-hive-user-archive@minotaur.apache.org Received: (qmail 71470 invoked from network); 8 Apr 2010 21:23:20 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 8 Apr 2010 21:23:20 -0000 Received: (qmail 26572 invoked by uid 500); 8 Apr 2010 21:23:20 -0000 Delivered-To: apmail-hadoop-hive-user-archive@hadoop.apache.org Received: (qmail 26545 invoked by uid 500); 8 Apr 2010 21:23:20 -0000 Mailing-List: contact hive-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-user@hadoop.apache.org Delivered-To: mailing list hive-user@hadoop.apache.org Received: (qmail 26537 invoked by uid 99); 8 Apr 2010 21:23:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:23:20 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com designates 209.85.160.48 as permitted sender) Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:23:13 +0000 Received: by pwi7 with SMTP id 7so2254173pwi.35 for ; Thu, 08 Apr 2010 14:22:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type; bh=Eb181jhua3lBgZWXjESbXxpUgbYcYTbAh3ceAHrKwk8=; b=OpXba3uNv6fxk/q/Isoj1keWKpDplwfR6AhxwIeVLvPca1AABKQ/SUeFoJb4Ks7wB8 ydHQtZDa8/NqRo76WHq9kZ05WD4AAxdx10uofNfpn9NBGF7fVgNq7McxJAgf2GlDSy/I RPPzfxyCCYN46o6DHYxbEvyN/v+q+cSqS1Ocs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=E/Mmv/urdgkIY9t9bApepakO+wzZbODFEVFTjIoRKy19z+5VqgPXa+xsvklyuo+085 gfYKyKQenRflclHhPf/HFy9YhOcQpX2DmoazKhLubPWo4apja+9ptC+fyx3hJzeF9ndj ZsQ0QfFq92p5Czk89xpwpzl4+ynQgiMc1VBew= MIME-Version: 1.0 Received: by 10.143.11.12 with HTTP; Thu, 8 Apr 2010 14:22:52 -0700 (PDT) In-Reply-To: <3120E6F5005EE7419C125CE166D55E90068F969745@SC-MBXC1.TheFacebook.com> References: <9A91DDD7-658D-48CE-9C42-7C994AAC3939@facebook.com> <3120E6F5005EE7419C125CE166D55E90068F969745@SC-MBXC1.TheFacebook.com> Date: Thu, 8 Apr 2010 17:22:52 -0400 Received: by 10.143.24.14 with SMTP id b14mr501263wfj.346.1270761772871; Thu, 08 Apr 2010 14:22:52 -0700 (PDT) Message-ID: Subject: Re: enough alter tables in the same .q file eventually fail From: Edward Capriolo To: hive-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001636e0b63910f1d40483c04824 --001636e0b63910f1d40483c04824 Content-Type: text/plain; charset=ISO-8859-1 On Thu, Apr 8, 2010 at 5:14 PM, Paul Yang wrote: > Seems to be fixed in 0.6. Here's what I got: > > test.q: > alter table tmp_pyang_t ADD PARTITION (ds='2') LOCATION '/tmp/blah2'; > alter table tmp_pyang_t ADD PARTITION (ds='3') LOCATION '/tmp/blah2'; > alter table tmp_pyang_t ADD PARTITION (ds='4') LOCATION '/tmp/blah2'; > alter table tmp_pyang_t ADD PARTITION (ds='5') LOCATION '/tmp/blah2'; > alter table tmp_pyang_t ADD PARTITION (ds='6') LOCATION '/tmp/blah2'; > alter table tmp_pyang_t ADD PARTITION (ds='7') LOCATION '/tmp/blah2'; > alter table tmp_pyang_t ADD PARTITION (ds='8') LOCATION '/tmp/blah2'; > > > Hive history file=/tmp/pyang/hive_job_log_pyang_201004081410_378771152.txt > OK > Time taken: 4.101 seconds > OK > Time taken: 0.558 seconds > OK > Time taken: 0.453 seconds > OK > Time taken: 0.416 seconds > OK > Time taken: 0.378 seconds > OK > Time taken: 0.457 seconds > OK > Time taken: 0.454 seconds > > > Can you the stack trace in /tmp//hive.log? > > -----Original Message----- > From: Prasad Chakka [mailto:pchakka@facebook.com] > Sent: Thursday, April 08, 2010 1:03 PM > To: hive-user@hadoop.apache.org > Subject: Re: enough alter tables in the same .q file eventually fail > > There was a bug that got fixed where each request was creating a separate > metastore client. That could be it or something similar that hasn't gotten > fixed. > > On Apr 8, 2010, at 11:47 AM, Edward Capriolo wrote: > > > Hive 5.0 mysql as metastore backend. Using external tables with location > for partitions... > > > > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid = > '000843') LOCATION 'hit_date=20100329/mid=000843'; > > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid = > '000844') LOCATION 'hit_date=20100329/mid=000844'; > > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid = > '000849') LOCATION 'hit_date=20100329/mid=000849'; > > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid = > '000850') LOCATION 'hit_date=20100329/mid=000850'; > > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid = > '000851') LOCATION 'hit_date=20100329/mid=000851'; > > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid = > '000852') LOCATION 'hit_date=20100329/mid=000852'; > > > > Eventually this fails after a number of entries. > > > > Time taken: 0.159 seconds > > OK > > Time taken: 0.17 seconds > > OK > > Time taken: 0.241 seconds > > FAILED: Error in metadata: Unable to fetch table XXXXX_action > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask > > > > Restarting the process after removing the already added tables works > until it breaks again. Anyone ever dealt with this? > > > > Doing one hive -e per table always works but takes a lot longer ...3 > seconds a partition rather then ~.5 seconds. > > > > > > It does not happen after 4 or 5 more like 100 or 1000+. I will try to track this down a bit. Edward --001636e0b63910f1d40483c04824 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Thu, Apr 8, 2010 at 5:14 PM, Paul Yan= g <pyang@faceboo= k.com> wrote:
Seems to be fixed in 0.6. Here's what I got:

test.q:
alter table tmp_pyang_t ADD PARTITION (ds=3D'2') LOCATION '/tmp= /blah2';
alter table tmp_pyang_t ADD PARTITION (ds=3D'3') LOCATION '/tmp= /blah2';
alter table tmp_pyang_t ADD PARTITION (ds=3D'4') LOCATION '/tmp= /blah2';
alter table tmp_pyang_t ADD PARTITION (ds=3D'5') LOCATION '/tmp= /blah2';
alter table tmp_pyang_t ADD PARTITION (ds=3D'6') LOCATION '/tmp= /blah2';
alter table tmp_pyang_t ADD PARTITION (ds=3D'7') LOCATION '/tmp= /blah2';
alter table tmp_pyang_t ADD PARTITION (ds=3D'8') LOCATION '/tmp= /blah2';


Hive history file=3D/tmp/pyang/hive_job_log_pyang_201004081410_378771152.tx= t
OK
Time taken: 4.101 seconds
OK
Time taken: 0.558 seconds
OK
Time taken: 0.453 seconds
OK
Time taken: 0.416 seconds
OK
Time taken: 0.378 seconds
OK
Time taken: 0.457 seconds
OK
Time taken: 0.454 seconds


Can you the stack trace in /tmp/<username>/hive.log?

-----Original Message-----
From: Prasad Chakka [mailto:pchakka= @facebook.com]
Sent: Thursday, April 08, 2010 1:03 PM
To: hive-user@hadoop.apache.= org
Subject: Re: enough alter tables in the same .q file eventually fail

There was a bug that got fixed where each request was creating a separate m= etastore client. That could be it or something similar that hasn't gott= en fixed.

On Apr 8, 2010, at 11:47 AM, Edward Capriolo wrote:

> Hive 5.0 mysql as metastore backend. Using external tables with locati= on for partitions...
>
> alter table XXXX_action ADD PARTITION (hit_date =3D '20100329'= , mid =3D '000843') LOCATION 'hit_date=3D20100329/mid=3D000843= ';
> alter table XXXX_action ADD PARTITION (hit_date =3D '20100329'= , mid =3D '000844') LOCATION 'hit_date=3D20100329/mid=3D000844= ';
> alter table XXXX_action ADD PARTITION (hit_date =3D '20100329'= , mid =3D '000849') LOCATION 'hit_date=3D20100329/mid=3D000849= ';
> alter table XXXX_action ADD PARTITION (hit_date =3D '20100329'= , mid =3D '000850') LOCATION 'hit_date=3D20100329/mid=3D000850= ';
> alter table XXXX_action ADD PARTITION (hit_date =3D '20100329'= , mid =3D '000851') LOCATION 'hit_date=3D20100329/mid=3D000851= ';
> alter table XXXX_action ADD PARTITION (hit_date =3D '20100329'= , mid =3D '000852') LOCATION 'hit_date=3D20100329/mid=3D000852= ';
>
> Eventually this fails after a number of entries.
>
> Time taken: 0.159 seconds
> OK
> Time taken: 0.17 seconds
> OK
> Time taken: 0.241 seconds
> FAILED: Error in metadata: Unable to fetch table XXXXX_action
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.= exec.DDLTask
>
> Restarting the process after removing the already added tables works u= ntil it breaks again. Anyone ever dealt with this?
>
> Doing one hive -e per table always works but takes a lot longer ...3 s= econds a partition rather then ~.5 seconds.
>
>


It does not happen after 4 or 5 more lik= e 100 or 1000+. I will try to track this down a bit.

Edward
--001636e0b63910f1d40483c04824--