hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Hoffman <ste...@goofy.net>
Subject Re: plugable partitioning
Date Mon, 15 Apr 2013 15:45:49 GMT
Yes, I understand that is how it works today and I'll use a cron job or
something to create these as needed.  I'll also look into the bucketing by
hour aspect.

That said, I'm suggesting that perhaps an alternative implementation (and
associated abstraction/plugability) may be of value.




On Mon, Apr 15, 2013 at 9:53 AM, Nitin Pawar <nitinpawar432@gmail.com>wrote:

> whenever you create a partition in hive, it needs to be registered with the
> metadata store. So short answer would be partition data is looked from
> metadata store instead of  the actual source data.
> having a lot of partitions does slow down hive (around 10000+). Normally
> have not seen anyone using hourly partitions. You may want to look at
> adding daily partition and bucket by hour.
>
> but if you are adding data directly into partition directories then there
> is no alternative other than adding partitions to metadata store manually
> apart from alter partition.
>
> if you are using hcatalog as metadata store then it does provide an api to
> register your partition so you can automate your data loading and
> registering both in a single flow.
>
> Others will correct me if I have made any wrong assumption
>
>
> On Mon, Apr 15, 2013 at 8:15 PM, Steve Hoffman <steveh@goofy.net> wrote:
>
> > Looking for some pointers on where the partitioning is figured out in the
> > source when a query is executed.
> > I'm investigating an alternative partitioning scheme based on date
> patterns
> > (using external tables).
> >
> > The situation is that I have data being written to some HDFS root
> directory
> > with some dated pattern (i.e. YYYY/MM/DD).  Today I have to run an alter
> > table to insert this partition every day.  It gets worse if you have
> hourly
> > partitions.  This seems like it can be described once (root + date
> > partition pattern in the metastore).
> >
> > So looking for some pointers on where in the code this is currently
> > handled.
> >
> > Thanks,
> > Steve
> >
>
>
>
> --
> Nitin Pawar
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message