Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <4C2E5FFB.4000501@darose.net>
References: <4C2E5FFB.4000501@darose.net>
Date: Fri, 2 Jul 2010 15:14:02 -0700
Message-ID: <AANLkTikWp83d4TgPj1xC7FUwFfCsqbCDiNlw3yO4M8Iw@mail.gmail.com>
Subject: Re: Text files vs. SequenceFiles
From: Alex Loddengaard <alex@cloudera.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016369204e792b492048a6ee74f

--0016369204e792b492048a6ee74f
Content-Type: text/plain; charset=ISO-8859-1

Hi David,

On Fri, Jul 2, 2010 at 2:54 PM, David Rosenstrauch <darose@darose.net>wrote:
>
> * We should use a SequenceFile (binary) format as it's faster for the
> machine to read than parsing text, and the files are smaller.
>
> * We should use a text file format as it's easier for humans to read,
> easier to change, text files can be compressed quite small, and a) if the
> text format is designed well and b) given the context of a distributed
> system like Hadoop where you can throw more nodes at a problem, the text
> parsing time will wind up being negligible/irrelevant in the overall
> processing time.
>

SequenceFiles can also be compressed, either per record or per block.  This
is advantageous if you want to use gzip, because gzip isn't splittable.  A
SF compressed by blocks is therefor splittable, because each block is
gzipped vs. the entire file being gzipped.

As for readability, "hadoop fs -text" is the same as "hadoop fs -cat" for
SequenceFiles.

Lastly, I promise that eventually you'll run out of space in your cluster
and wish you did better compression.  Plus compression makes jobs faster.

The general recommendation is to use SequenceFiles as early in your ETL as
possible.  Usually people get their data in as text, and after the first MR
pass they work with SequenceFiles from there on out.

Alex

--0016369204e792b492048a6ee74f--