Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (athena.apache.org: domain of matthewtckr@gmail.com
 designates 209.85.161.176 as permitted sender)
Subject: Re: split into less files
References: 
 <CAAY=0mvJJXokejnZF4W7+U8fgi1_4L=SHxz5fTZo56JotaCwDA@mail.gmail.com>
 <CANbn9uk1kTzCJY-VW9huhRoC0Gu6Oz2diTMCYKLng_6usvczBw@mail.gmail.com>
From: Matt Tucker <matthewtckr@gmail.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-45CAA965-D32C-4307-A759-DE69213033FF
In-Reply-To: 
 <CANbn9uk1kTzCJY-VW9huhRoC0Gu6Oz2diTMCYKLng_6usvczBw@mail.gmail.com>
Message-Id: <DCD8879E-9022-4003-8B22-6826C21A4666@gmail.com>
Date: Tue, 8 Nov 2011 22:05:36 -0500
To: "user@hive.apache.org" <user@hive.apache.org>
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)


--Apple-Mail-45CAA965-D32C-4307-A759-DE69213033FF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

It sounds like you want to look at setting hive.merge.mapredfiles to true in=
 your hive-site.xml.

Just be aware that it will likely add another map step to your jobs to conso=
lidate the files.

Matt Tucker


On Nov 8, 2011, at 6:19 PM, Shouguo Li <the1plummie@gmail.com> wrote:

> i think that has to do with your configured block size, check what's your v=
alue for dfs.block.size in /hdfs-site.xml   =20
> but just curious, why would number of files matter for your use case?
>=20
>=20
> On Fri, Oct 21, 2011 at 1:18 AM, Vikas Srivastava <vikas.srivastava@one97.=
net> wrote:
> Hey All,
>=20
>=20
> i have an issue like i got a table having single partition but in that par=
tition say around 100 200mb files  when i overwrite this into other table it=
s make 100 files of 20 mb(compressed) what i want is that it should make onl=
y 1 or 2 or 10 file of 200mb or 100mb
>=20
>=20
> means after overwrite its should make less no of file as compare to non co=
mpressed.=20
>=20
>=20
>=20
>=20
> --=20
> With Regards
> Vikas Srivastava
>=20
> DWH & Analytics Team
> Mob:+91 9560885900
> One97 | Let's get talking !
>=20
>=20

--Apple-Mail-45CAA965-D32C-4307-A759-DE69213033FF
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=utf-8

<html><head></head><body bgcolor="#FFFFFF"><div>It sounds like you want to look at setting&nbsp;hive.merge.mapredfiles to true in your hive-site.xml.</div><div><br></div><div>Just be aware that it will likely add another map step to your jobs to consolidate the files.</div><div><br></div><div>Matt Tucker<br><br><br></div><div><br>On Nov 8, 2011, at 6:19 PM, Shouguo Li &lt;<a href="mailto:the1plummie@gmail.com">the1plummie@gmail.com</a>&gt; wrote:<br><br></div><div></div><blockquote type="cite"><div>i think that has to do with your configured block size, check what's your value for dfs.block.size in /hdfs-site.xml&nbsp;&nbsp;&nbsp; <br>but just curious, why would number of files matter for your use case?<br><br><br><div class="gmail_quote">
On Fri, Oct 21, 2011 at 1:18 AM, Vikas Srivastava <span dir="ltr">&lt;<a href="mailto:vikas.srivastava@one97.net">vikas.srivastava@one97.net</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<font color="#000099"><font size="2"><font face="arial,helvetica,sans-serif">Hey All,<br><br><br>i have an issue like i got a table having single partition but in that partition say around 100 200mb files&nbsp; when i overwrite this into other table its make 100 files of 20 mb(compressed) what i want is that it should make only 1 or 2 or 10 file of 200mb or 100mb<br>

<br><br>means after overwrite its should make less no of file as compare to non compressed. <br><span class="HOEnZb"><font color="#888888"><br><br><br clear="all"></font></span></font></font></font><span class="HOEnZb"><font color="#888888"><br>
-- <br><font style="font-family:arial,helvetica,sans-serif" face="verdana, sans-serif"><font color="#000099">With Regards<br>
Vikas Srivastava<br><br>DWH &amp; Analytics Team</font></font><div style="font-family:arial,helvetica,sans-serif"><font color="#000099">Mob:<a href="tel:%2B91%209560885900" value="+919560885900" target="_blank">+91 9560885900</a><br>
One97 | Let's get talking !</font></div><br>
</font></span></blockquote></div><br>
</div></blockquote></body></html>
--Apple-Mail-45CAA965-D32C-4307-A759-DE69213033FF--