Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of jason.hadoop@gmail.com
 designates 209.85.146.178 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=FQQ7u2SlSf3obIL9oxqTSoD3VJARbzYyC6kup8S8ChXiqg0ZSjX5P14e6d0OAYdLhX
         6HYoz9VjoNXvv8PhQGWk4i5Qx3+4dYJ40ZYUH5Y5gotQeCXi9V7xm6GhisMT1bN48qMc
         eXRUEqkhMd/jJ4BJGRU3v3LlGllVEdG5jEy+Q=
MIME-Version: 1.0
In-Reply-To: <c9b0d8bd0901260935v59d5660dn11cc96b9c0dfd874@mail.gmail.com>
References: <c9b0d8bd0901231503k28cee90eg472e319e3ca4ce24@mail.gmail.com>
	 <497A4E00.3000102@yahoo-inc.com>
	 <c9b0d8bd0901231529l3985d2d2vb4233dd3e4b06b7c@mail.gmail.com>
	 <497A579D.8070707@yahoo-inc.com>
	 <771c51630901232101q35e51276rfa666a11fe814179@mail.gmail.com>
	 <c9b0d8bd0901241721h1c98a3fv78f011336ea1ef30@mail.gmail.com>
	 <771c51630901241859h24fbfbdbwa147284746657f01@mail.gmail.com>
	 <497DF33E.1060603@apache.org>
	 <c9b0d8bd0901260935v59d5660dn11cc96b9c0dfd874@mail.gmail.com>
Date: Mon, 26 Jan 2009 09:53:35 -0800
Message-ID: <314098690901260953l6fa4c4b8tedf39573acbff6a@mail.gmail.com>
Subject: Re: HDFS - millions of files in one directory?
From: jason hadoop <jason.hadoop@gmail.com>
To: core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636417a85efc52b0461666ae7

--001636417a85efc52b0461666ae7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

We like compression if the data is readily compressible and large as it
saves on IO time.


On Mon, Jan 26, 2009 at 9:35 AM, Mark Kerzner <markkerzner@gmail.com> wrote:

> Doug,
> SequenceFile looks like a perfect candidate to use in my project, but are
> you saying that I better use uncompressed data if I am not interested in
> saving disk space?
>
> Thank you,
> Mark
>
> On Mon, Jan 26, 2009 at 11:30 AM, Doug Cutting <cutting@apache.org> wrote:
>
> > Philip (flip) Kromer wrote:
> >
> >> Heretrix <http://en.wikipedia.org/wiki/Heritrix>,
> >> Nutch<http://en.wikipedia.org/wiki/Nutch>,
> >> others use the ARC file format
> >>  http://www.archive.org/web/researcher/ArcFileFormat.php
> >>  http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml
> >>
> >
> > Nutch does not use ARC format but rather uses Hadoop's SequenceFile to
> > store crawled pages.  The keys of crawl content files are URLs and the
> > values are:
> >
> >
> >
> http://lucene.apache.org/nutch/apidocs/org/apache/nutch/protocol/Content.html
> >
> > I believe that the implementation of this class pre-dates SequenceFile's
> > support for compressed values, so the values are decompressed on demand,
> > which needlessly complicates its implementation and API.  It's basically
> a
> > Writable that stores binary content plus headers, typically an HTTP
> > response.
> >
> > Doug
> >
>

--001636417a85efc52b0461666ae7--