Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.223.181 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BAY177-W341655649D568A6E4402F29F5A0@phx.gbl>
References: 
 <CAJJCyea6ahjb=ERN3SR79k6+O+CX-YMq0sboui5sGre+LOXSyw@mail.gmail.com>
 <CAOcnVr0mAugffunmm0+7fZYALyH0_-Ydr+MGGOMpfOBpVC5EVg@mail.gmail.com>
 <BAY177-W341655649D568A6E4402F29F5A0@phx.gbl>
From: Harsh J <harsh@cloudera.com>
Date: Sun, 11 Aug 2013 20:52:19 +0530
Message-ID: 
 <CAOcnVr01vbuoZQGdeH7q68HXJx5DfZTH2B=1-BrFgKUZz19W8A@mail.gmail.com>
Subject: Re: How to compress MapFile programmatically
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

A MapFile.Reader will automatically detect and decompress without
needing to be told anything special. You needn't have to worry about
decompressing files by yourself in Apache Hadoop generally - the
framework handles it for you transparently if you're using the proper APIs.

On Sun, Aug 11, 2013 at 8:49 PM, Abhijit Sarkar
<abhijit.sarcar@gmail.com> wrote:
> Thanks Harsh. However, if I compress the MapFile using the MapFile.Writer
> Constructor option and then put it in a DistributedCache, how do I
> uncompress it in the Map/Reduce? There isn't any API method to do that
> apparently.
>
> Regards,
> Abhijit
>
>> From: harsh@cloudera.com
>> Date: Sun, 11 Aug 2013 12:56:43 +0530
>> Subject: Re: How to compress MapFile programmatically
>> To: user@hadoop.apache.org
>
>>
>> A MapFile isn't a directory. It is a directory _containing_ two files.
>> You cannot "open" a directory for reading.
>>
>> The MapFile API is documented at
>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/io/MapFile.ht=
ml
>> and thats what you're to be using for reading/writing them.
>>
>> Compression is a simple option you need to provide when invoking the
>> writer:
>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/io/MapFile.Wr=
iter.html#MapFile.Writer(org.apache.hadoop.conf.Configuration,%20org.apache=
.hadoop.fs.FileSystem,%20java.lang.String,%20org.apache.hadoop.io.WritableC=
omparator,%20java.lang.Class,%20org.apache.hadoop.io.SequenceFile.Compressi=
onType,%20org.apache.hadoop.io.compress.CompressionCodec,%20org.apache.hado=
op.util.Progressable)
>>
>> On Sun, Aug 11, 2013 at 1:46 AM, Abhijit Sarkar
>> <abhijit.sarcar@gmail.com> wrote:
>> > Hi,
>> > I'm a Hadoop newbie. This is my first question to this mailing list,
>> > hoping
>> > for a good start :)
>> >
>> > MapFile is a directory so when I try to open an InputStream to it, it
>> > fails
>> > with FileNotFoundException. How do I compress MapFile programmatically=
?
>> >
>> > Code snippet:
>> > final FileSystem fs =3D FileSystem.get(conf);
>> > final InputStream inputStream =3D fs.open(new Path(uncompressedStr));
>> >
>> > Exception:
>> > java.io.FileNotFoundException: /some/directory (No such file or
>> > directory)
>> > at java.io.FileInputStream.open(Native Method)
>> > at java.io.FileInputStream.<init>(FileInputStream.java:120)
>> > at
>> >
>> > org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.<init>=
(RawLocalFileSystem.java:71)
>> > at
>> >
>> > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(=
RawLocalFileSystem.java:107)
>> > at
>> > org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:1=
77)
>> > at
>> >
>> > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(=
ChecksumFileSystem.java:126)
>> > at
>> > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:2=
83)
>> > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
>> > at
>> > name.abhijitsarkar.learning.hadoop.io.IOUtils.compress(IOUtils.java:10=
4)
>> >
>> > Regards,
>> > Abhijit
>>
>>
>>
>> --
>> Harsh J


--=20
Harsh J