hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daren Hasenkamp" <dhasenk...@berkeley.edu>
Subject record-aware file splitting
Date Tue, 01 Jun 2010 20:56:44 GMT

I am interested in implementing record-aware file splitting for hadoop. I
am looking for someone who knows the hadoop internals well and is willing
to discuss some details of how to accomplish this.

By "record-aware file splitting", I mean that I want to be able to put
files into hadoop with a custom InputFormat implementation, and hadoop
will split the files into blocks such that no record is split between

I believe that record-aware file splitting could offer considerable
speedup when dealing with large records--say, 10s or 100s of megabytes per
record--since it eliminates the need to stream part of a record from one
datanode to another when said record is split between block boundaries.

(The motivation here is that large records occur commonly when dealing
with scientific datasets. Imagine, for example, a set of climate
simulation data, where each "record" consists of climate data over the
entire globe at a given time step. This is a huge amount of data per
record. Essentially, I want to modify Hadoop to work faster with large
scientific datasets.)

If you are interested in discussing this with me, I would love to talk
more with you.

Daren Hasenkamp
Computer Science/Applied Mathematics, UC Berkeley
Student Assistant, Lawrence Berkeley National Lab

View raw message