hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Creating splits/tasks at the client
Date Fri, 29 Sep 2006 16:28:00 GMT
Benjamin Reed wrote:
> Split will write the hosts first, so in the JobTracker, when you get the
> byte array representing the Split, any fields from the sub class will
> follow the Split serialized bytes. The JobTracker can skip the Type in
> the bytes representing the serialized Split and then deserialize just a
> Split (ignoring the rest).

That could work, but it would be hard to pass these direclty over either 
RPC or via a SequenceFile, no?  We'd have to write a custom container 
for the array of Splits.  The container could then be passed as a whole 
over RPC or placed in a SequenceFile.  That could become a little awkward.


View raw message