Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jiayu.ji@gmail.com designates
 74.125.82.53 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACUUc6DRDXTEkv6UmHE5BUeZwoE5-od5g1ntdE_26nys0wAs0w@mail.gmail.com>
References: 
 <CADYfWQN_WT+883U1n-NW802MVs8J231=znQ+tByqz1n=LmOyag@mail.gmail.com>
	<CACUUc6DRDXTEkv6UmHE5BUeZwoE5-od5g1ntdE_26nys0wAs0w@mail.gmail.com>
Date: Sun, 15 Dec 2013 18:28:28 -0600
Message-ID: 
 <CADYfWQNWe1PNOvn9d49Y9UmSpcXOBb3q-7c8Hb1+fhCTTs=jjQ@mail.gmail.com>
Subject: Re: How does mapreduce job determine the compress codec
From: Jiayu Ji <jiayu.ji@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e0149422412aa2d04ed9be4a9

--089e0149422412aa2d04ed9be4a9
Content-Type: text/plain; charset=ISO-8859-1

Thanks Tao. I know I can tell it is a lzo file based on the magic number.
What I am curious is which class in hadoop used by the mapreduce job to
determine the file compression algorithm. At the end of the day, I am
trying to figure out whether all the inputs of a mapreduce job have to be
compressed with the same algorithm.


On Fri, Dec 13, 2013 at 11:16 PM, Tao Xiao <xiaotao.cs.nju@gmail.com> wrote:

> I suggest you download the lzo compressed file, no matter weather it has a
> lzo extension as its file name,  and open it in the form of hex bytes with
> tools like UltraEdit, and have a look at its heading contents.
>
>
> 2013/12/14 Jiayu Ji <jiayu.ji@gmail.com>
>
>> Hi
>>
>> I am having this question on how does mapreduce job determine the
>> compress codec on hdfs. From what I read on the definitive guide (page
>> 86)," the CompressionCodecFactory provides a way of mapping a filename
>> extension to a CompressionCodec using its getCodec() method". I did a test
>> with a lzo compressed file without a lzo extension. However, the mapreduce
>> job was still able to get the right codec. Does anyone know why? Thanks in
>> advance.
>>
>> Jiayu
>>
>
>


-- 
Jiayu (James) Ji,

Cell: (312)823-7393

--089e0149422412aa2d04ed9be4a9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks Tao. I know I can tell it is a lzo file based on th=
e magic number. What I am curious is which class in hadoop used by the mapr=
educe job to determine the file compression algorithm. At the end of the da=
y, I am trying to figure out whether all the inputs of a mapreduce job have=
 to be compressed with the same algorithm.=A0</div>
<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Fri, Dec 1=
3, 2013 at 11:16 PM, Tao Xiao <span dir=3D"ltr">&lt;<a href=3D"mailto:xiaot=
ao.cs.nju@gmail.com" target=3D"_blank">xiaotao.cs.nju@gmail.com</a>&gt;</sp=
an> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">I suggest you download the =
lzo compressed file, no matter weather it has a lzo extension as its file n=
ame,=A0 and open it in the form of hex bytes with tools like UltraEdit, and=
 have a look at its heading contents.=A0 <br>

</div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><b=
r><br><div class=3D"gmail_quote">2013/12/14 Jiayu Ji <span dir=3D"ltr">&lt;=
<a href=3D"mailto:jiayu.ji@gmail.com" target=3D"_blank">jiayu.ji@gmail.com<=
/a>&gt;</span><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<div dir=3D"ltr">Hi<div><br></div><div>I am having this question on how doe=
s mapreduce job determine the compress codec on hdfs. From what I read on t=
he definitive guide (page 86),&quot; the CompressionCodecFactory provides a=
 way of mapping a filename extension to a CompressionCodec using its getCod=
ec() method&quot;. I did a test with a lzo compressed file without a lzo ex=
tension. However, the mapreduce job was still able to get the right codec. =
Does anyone know why? Thanks in advance.=A0</div>

<span><font color=3D"#888888">
<div><br></div><div><div dir=3D"ltr"><div>Jiayu</div></div>
</div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div dir=3D"ltr">Jiayu (James) Ji,<div><br></div><div><span style=3D"border=
-collapse:collapse;color:rgb(32,32,32)"><font face=3D"arial, helvetica, san=
s-serif"><span style=3D"border-collapse:separate;color:rgb(0,0,0)">Cell: (3=
12)823-7393</span></font></span></div>
<div><br></div></div>
</div>

--089e0149422412aa2d04ed9be4a9--