hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: [DISCUSS] Remove append?
Date Wed, 21 Mar 2012 18:27:51 GMT
On Wed, Mar 21, 2012 at 10:47 AM,  <Milind.Bhandarkar@emc.com> wrote:
> Answers inline.
> On 3/21/12 10:32 AM, "Eli Collins" <eli@cloudera.com> wrote:
>>Why not just write new files and use Har files, because Har files are a
> Yes, and har creation is an MR job, which is totally I/O bound, and yet
> takes up slots/containers, reducing cluster utilization.
>>Can you elaborate on the 1st one, how it's especially helpful for
> Say you have daily log files (consider many small job history files).
> Instead of keeping them as separate files, one appends them to a monthly
> files (this in itself is a complete rewrite), but appending monthly files
> to year-to-date files should not require rewrite (because after March, it
> becomes very inefficient.)

Why not just keep the original daily files instead of continually
either rewriting (yuck) or duplicating (yuck) the data by aggregating
them into rollups?  I can think of two reasons:

1. If the daily files are smaller than 1 block (seems unlikely)
2. The small files problem (a typical NN can store 100-200M files, so
a problem for big users)

In which case maybe better to focus on #2 rather than work around it?


> Reducing number of files this way also makes it easy to copy, take
> snapshots etc without having to write special parallel code to do it.
>>I assume the 2nd one refers to not having to Multi*InputFormat. And
>>the 3rd refers to appending to an old file instead of creating a new
> Yes.
>>> In addition, the small-files problem in HDFS forces people to write MR
>>> code, and causes rewrite of large datasets even if a small amount of
>>> is added to it.
>>Do people rewrite large datasets today just to add 1mb? I haven't
>>heard of that from big users (Yahoo!, FB, Twitter, eBay..) or my
>>customer base.  If so I'd would have expected people to put energy
>>into getting append working in 1.x which know was has put energy into
>>(I know some people feel the 20-based design is unworkable, I don't
>>know it well enough to comment there).
> With HDFS, they do not rewrite large datasets just to add a small amount
> of data. Instead they create new files, and use a separate
> metadata-service (or just file numbering conventions) to make the added
> data part of the large dataset. But with other file systems, they just
> ">>".
> Thanks,
> - milind
>>Milind Bhandarkar
>>Greenplum Labs, EMC
>>(Disclaimer: Opinions expressed in this email are those of the author,
>>and do not necessarily represent the views of any organization, past or
>>present, the author might be affiliated with.)

View raw message