Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
Sender: niels@basj.es
In-Reply-To: <542034F6.2020903@vesseltracker.com>
References: <542034F6.2020903@vesseltracker.com>
Date: Mon, 22 Sep 2014 17:13:18 +0200
Message-ID: 
 <CADoiZqoMYXMawp=mayL7hi6ZxF2F=-2PkVrqyJ5ZaWOzpyoK3Q@mail.gmail.com>
Subject: Re: Bzip2 files as an input to MR job
From: Niels Basjes <Niels@basjes.nl>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b41773b068e040503a8e41f

--047d7b41773b068e040503a8e41f
Content-Type: text/plain; charset=UTF-8

Hi,

You can use the GZip inside the AVRO files and still have splittable AVRO
files.
This has the to with the fact that there is a block structure inside the
AVRO and these blocks are gzipped.

I suggest you simply try it.

Niels


On Mon, Sep 22, 2014 at 4:40 PM, Georgi Ivanov <ivanov@vesseltracker.com>
wrote:

> Hi guys,
> I would like to compress the files on HDFS to save some storage.
>
> As far as i see bzip2 is the only format which is splitable (and slow).
>
> The actual files are Avro.
>
> So in my driver class i have :
>
> job.setInputFormatClass(AvroKeyInputFormat.class);
>
> I have number of jobs running processing Avro files so i would like to
> keep the code change to a minimum.
>
> Is it possible to comrpess these avro files with bzip2 and keep the code
> of MR jobs the same (or with little change)
> If it is , please give me some hints as so far i don't seem to find any
> good resources on the Internet.
>
>
> Georgi
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

--047d7b41773b068e040503a8e41f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi,</div><div><br></div>You can use the GZip inside t=
he AVRO files and still have splittable AVRO files.<div>This has the to wit=
h the fact that there is a block structure inside the AVRO and these blocks=
 are gzipped.<br></div><div class=3D"gmail_extra"><br></div><div class=3D"g=
mail_extra">I suggest you simply try it.</div><div class=3D"gmail_extra"><b=
r></div><div class=3D"gmail_extra">Niels</div><div class=3D"gmail_extra"><b=
r></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, S=
ep 22, 2014 at 4:40 PM, Georgi Ivanov <span dir=3D"ltr">&lt;<a href=3D"mail=
to:ivanov@vesseltracker.com" target=3D"_blank">ivanov@vesseltracker.com</a>=
&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi guys,<br>
I would like to compress the files on HDFS to save some storage.<br>
<br>
As far as i see bzip2 is the only format which is splitable (and slow).<br>
<br>
The actual files are Avro.<br>
<br>
So in my driver class i have :<br>
<br>
job.setInputFormatClass(<u></u>AvroKeyInputFormat.class);<br>
<br>
I have number of jobs running processing Avro files so i would like to keep=
 the code change to a minimum.<br>
<br>
Is it possible to comrpess these avro files with bzip2 and keep the code of=
 MR jobs the same (or with little change)<br>
If it is , please give me some hints as so far i don&#39;t seem to find any=
 good resources on the Internet.<span class=3D"HOEnZb"><font color=3D"#8888=
88"><br>
<br>
<br>
Georgi<br>
</font></span></blockquote></div><br><br clear=3D"all"><div><br></div>-- <b=
r>Best regards / Met vriendelijke groeten,<br><br>Niels Basjes
</div></div>

--047d7b41773b068e040503a8e41f--