Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (athena.apache.org: domain of suresh.krishnappa@gmail.com
 designates 74.125.82.51 as permitted sender)
MIME-Version: 1.0
Date: Thu, 7 Mar 2013 20:01:19 +0530
Message-ID: 
 <CALWST80T-9_tQNju3c-TNiZp4MQ9GeggT2GjM276-o6TPkLe1w@mail.gmail.com>
Subject: HIVE issues when using large number of partitions
From: Suresh Krishnappa <suresh.krishnappa@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=f46d041826da609f9404d7568f8f

--f46d041826da609f9404d7568f8f
Content-Type: text/plain; charset=ISO-8859-1

Hi All,
I have a hadoop cluster with data present in large number of directories (
> 10,000)
To run HIVE queries over this data I created an external partitioned table
and pointed each directory as a partition to the external table using
'alter table add partition' command.
Is there a better way to create a HIVE external table over large number of
directories?

Also I am facing the following issues due to the large number of partitions
1) The DDL operations of creating the table and adding partitions to the
table takes a very long time. Takes about an hour to add around 10,000
partitions
2) Getting 'out of memory' java exception while adding partitions > 50000
3) Sometimes getting 'out of memory' java exception for select queries for
partitions > 10000

What is the recommended limit to the number of partitions that we can
create with an HIVE table?
Are there any configuration settings in hive/hadoop to support large number
of partitions?

I am using HIVE 0.10.0. I re-ran the tests by replacing derby with
postgresql as metastore and still faced similar issues.

Would appreciate any inputs on this

Thanks
Suresh

--f46d041826da609f9404d7568f8f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi All,<div>I have a hadoop cluster with data present in large number of di=
rectories ( &gt; 10,000)</div><div>To run HIVE queries over this data I cre=
ated an external partitioned table and pointed each directory as a partitio=
n to the external table using &#39;alter table add partition&#39; command.=
=A0</div>
<div>Is there a better way to create a HIVE external table over large numbe=
r of directories?</div><div><br></div><div>Also I am facing the following i=
ssues due to the large number of partitions</div><div>1) The DDL operations=
 of creating the table and adding partitions to the table takes a very long=
 time. Takes about an hour to add around 10,000 partitions</div>
<div>2) Getting &#39;out of memory&#39; java exception while adding partiti=
ons &gt; 50000</div><div>3) Sometimes getting &#39;out of memory&#39; java =
exception for select queries for partitions &gt; 10000</div><div><br></div>
<div>What is the recommended limit to the number of partitions that we can =
create with an HIVE table?</div><div>Are there any configuration settings i=
n hive/hadoop to support large number of partitions?</div><div><br></div>
<div>I am using HIVE 0.10.0. I re-ran the tests by replacing derby with pos=
tgresql as metastore and still faced similar issues.</div><div><br></div><d=
iv>Would appreciate any inputs on this</div><div><br></div><div>Thanks</div=
>
<div>Suresh</div><div><br></div>

--f46d041826da609f9404d7568f8f--