hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: enough alter tables in the same .q file eventually fail
Date Thu, 08 Apr 2010 21:22:52 GMT
On Thu, Apr 8, 2010 at 5:14 PM, Paul Yang <pyang@facebook.com> wrote:

> Seems to be fixed in 0.6. Here's what I got:
>
> test.q:
> alter table tmp_pyang_t ADD PARTITION (ds='2') LOCATION '/tmp/blah2';
> alter table tmp_pyang_t ADD PARTITION (ds='3') LOCATION '/tmp/blah2';
> alter table tmp_pyang_t ADD PARTITION (ds='4') LOCATION '/tmp/blah2';
> alter table tmp_pyang_t ADD PARTITION (ds='5') LOCATION '/tmp/blah2';
> alter table tmp_pyang_t ADD PARTITION (ds='6') LOCATION '/tmp/blah2';
> alter table tmp_pyang_t ADD PARTITION (ds='7') LOCATION '/tmp/blah2';
> alter table tmp_pyang_t ADD PARTITION (ds='8') LOCATION '/tmp/blah2';
>
>
> Hive history file=/tmp/pyang/hive_job_log_pyang_201004081410_378771152.txt
> OK
> Time taken: 4.101 seconds
> OK
> Time taken: 0.558 seconds
> OK
> Time taken: 0.453 seconds
> OK
> Time taken: 0.416 seconds
> OK
> Time taken: 0.378 seconds
> OK
> Time taken: 0.457 seconds
> OK
> Time taken: 0.454 seconds
>
>
> Can you the stack trace in /tmp/<username>/hive.log?
>
> -----Original Message-----
> From: Prasad Chakka [mailto:pchakka@facebook.com]
> Sent: Thursday, April 08, 2010 1:03 PM
> To: hive-user@hadoop.apache.org
> Subject: Re: enough alter tables in the same .q file eventually fail
>
> There was a bug that got fixed where each request was creating a separate
> metastore client. That could be it or something similar that hasn't gotten
> fixed.
>
> On Apr 8, 2010, at 11:47 AM, Edward Capriolo wrote:
>
> > Hive 5.0 mysql as metastore backend. Using external tables with location
> for partitions...
> >
> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid =
> '000843') LOCATION 'hit_date=20100329/mid=000843';
> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid =
> '000844') LOCATION 'hit_date=20100329/mid=000844';
> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid =
> '000849') LOCATION 'hit_date=20100329/mid=000849';
> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid =
> '000850') LOCATION 'hit_date=20100329/mid=000850';
> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid =
> '000851') LOCATION 'hit_date=20100329/mid=000851';
> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid =
> '000852') LOCATION 'hit_date=20100329/mid=000852';
> >
> > Eventually this fails after a number of entries.
> >
> > Time taken: 0.159 seconds
> > OK
> > Time taken: 0.17 seconds
> > OK
> > Time taken: 0.241 seconds
> > FAILED: Error in metadata: Unable to fetch table XXXXX_action
> > FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> >
> > Restarting the process after removing the already added tables works
> until it breaks again. Anyone ever dealt with this?
> >
> > Doing one hive -e per table always works but takes a lot longer ...3
> seconds a partition rather then ~.5 seconds.
> >
> >
>
>
It does not happen after 4 or 5 more like 100 or 1000+. I will try to track
this down a bit.

Edward

Mime
View raw message