hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: [DISCUSS] Remove append?
Date Wed, 21 Mar 2012 17:33:52 GMT
On Wed, Mar 21, 2012 at 10:32 AM, Eli Collins <eli@cloudera.com> wrote:
> Thanks for the feedback Milind, questions inline.
>
> On Wed, Mar 21, 2012 at 10:17 AM,  <Milind.Bhandarkar@emc.com> wrote:
>> As someone who has worked with hdfs-compatible distributed file systems
>> that support append, I can vouch for its extensive usage.
>>
>> I have seen how simple it becomes to create tar archives, and later append
>> files to them, without writing special inefficient code to do so.
>>
>
> Why not just write new files and use Har files, because Har files are a pita?
>
>> I have seen it used in archiving cold data, reducing MR task launch
>> overhead without having to use a different input format, so that the same
>> code can be used for both hot and cold data.
>>
>
> Can you elaborate on the 1st one, how it's especially helpful for archival?
>
> I assume the 2nd one refers to not having to Multi*InputFormat. And
> the 3rd refers to appending to an old file instead of creating a new
> one.
>
>> In addition, the small-files problem in HDFS forces people to write MR
>> code, and causes rewrite of large datasets even if a small amount of data
>> is added to it.
>
> Do people rewrite large datasets today just to add 1mb? I haven't
> heard of that from big users (Yahoo!, FB, Twitter, eBay..) or my
> customer base.  If so I'd would have expected people to put energy
> into getting append working in 1.x which know was has put energy into

Arg, that should read "no one has put energy into".  </drinks coffee>

Mime
View raw message