hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Brondsema <dbronds...@geek.net>
Subject Re: dynamic partition query dies with LeaseExpiredException
Date Fri, 01 Oct 2010 19:24:22 GMT
Thanks Ning,

Our TRANSFORM script is pretty quick, the whole query only took 30 seconds
when we added "and project='abc' " to limit the amount of input data.  So it
seems like the slowdown is somewhere in handling the data before (or after?)
it gets to the transform script.  I can try adding some logging in our
script though, so we can see how far it does get through the data.

We'll take a look at 0.20.2 but it might take a while until we can upgrade
our main cluster to it.

On Fri, Oct 1, 2010 at 12:24 PM, Ning Zhang <nzhang@facebook.com> wrote:

> Hi Dave,
>
> There may be 2 issues here:
>  1) according to the HDFS JIRA it seems to be a bug introduced in Hadoop
> 0.18 and was fixed by another hadoop JIRA in 0.20.2 suggested by Hairong,
> you may want to try 0.20.2 if that's possible.
>  2) the fact that the lease expired seems to be caused by the fact that the
> open file has not get any write for a long time. From you query the
> TRANSFORM script is called for each row that will going to be written to
> HDFS file. Does the python script cause long latency (the time between two
> output records from the transform script)?
>
>
> On Oct 1, 2010, at 8:34 AM, Dave Brondsema wrote:
>
> We're trying to insert into a table, using a dynamic partition, but the
> query runs for a while and then dies with a LeaseExpiredException.  The
> hadoop details & some discussion is at
> https://issues.apache.org/jira/browse/HDFS-198  Is there a way to
> configure hive, or our query, to work around this?  If we adjust our query
> to handle less data at once, it can complete in under 10 minutes, but then
> we have to run the query many more times to get all the data processed.
>
> The query is:
> FROM (
>     FROM (
>         SELECT file, os, country, dt, project
>         FROM downloads WHERE dt='2010-10-01'
>         DISTRIBUTE BY project
>         SORT BY project asc, file asc
>     ) a
>     SELECT TRANSFORM(file, os, country, dt, project)
>     USING 'transformwrap reduce.py'
>     AS (file, downloads, os, country, project)
> ) b
> INSERT OVERWRITE TABLE dl_day PARTITION (dt='2010-10-01', project)
> SELECT file, downloads, os, country, FALSE, project
>
> The project partition has roughly 100000 values.
>
> We're using Hive trunk from about a month ago.
>  Hadoop 0.18.3-14.cloudera.CH0_3
>
> --
> Dave Brondsema
> Software Engineer
> Geeknet
>
> www.geek.net
>
>
>


-- 
Dave Brondsema
Software Engineer
Geeknet

www.geek.net

Mime
View raw message