hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Milind.Bhandar...@emc.com>
Subject Re: [DISCUSS] Remove append?
Date Wed, 21 Mar 2012 19:30:09 GMT

>1. If the daily files are smaller than 1 block (seems unlikely)

Even at a large hdfs installation, the avg file size was < 1.5 blocks.
Bucketing causes the file sizes to drop.

>2. The small files problem (a typical NN can store 100-200M files, so
>a problem for big users)

Big users probably have enough people to write their own roll-up code to
avoid small-files problem. Its the rest that are used to storage systems
handling billions of files.

- milind

---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)



>
>In which case maybe better to focus on #2 rather than work around it?
>
>Thanks,
>Eli
>
>>
>> Reducing number of files this way also makes it easy to copy, take
>> snapshots etc without having to write special parallel code to do it.
>>
>>>
>>>I assume the 2nd one refers to not having to Multi*InputFormat. And
>>>the 3rd refers to appending to an old file instead of creating a new
>>>one.
>>
>> Yes.
>>
>>>
>>>> In addition, the small-files problem in HDFS forces people to write MR
>>>> code, and causes rewrite of large datasets even if a small amount of
>>>>data
>>>> is added to it.
>>
>>
>>>
>>>Do people rewrite large datasets today just to add 1mb? I haven't
>>>heard of that from big users (Yahoo!, FB, Twitter, eBay..) or my
>>>customer base.  If so I'd would have expected people to put energy
>>>into getting append working in 1.x which know was has put energy into
>>>(I know some people feel the 20-based design is unworkable, I don't
>>>know it well enough to comment there).
>>
>> With HDFS, they do not rewrite large datasets just to add a small amount
>> of data. Instead they create new files, and use a separate
>> metadata-service (or just file numbering conventions) to make the added
>> data part of the large dataset. But with other file systems, they just
>> ">>".
>>
>> Thanks,
>>
>> - milind
>>
>>
>>>---
>>>Milind Bhandarkar
>>>Greenplum Labs, EMC
>>>(Disclaimer: Opinions expressed in this email are those of the author,
>>>and do not necessarily represent the views of any organization, past or
>>>present, the author might be affiliated with.)
>>
>


Mime
View raw message