hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naresh Rapolu <nareshreddy.rap...@gmail.com>
Subject Re: Need help understanding the source
Date Mon, 06 Jul 2009 18:14:09 GMT

Hello Todd,

My aim is to make the reduce move ahead with reduction as and when it gets
the data required, instead of waiting for all the maps to complete.  If it
knows how many records it needs and compares it with number of records it
has got until now,  it can move on once they become equal without waiting
for all the maps to finish. 

So if i can know the number of records received from each file the
MapOutputCopier has copied, then i can do this comparison. But now , the 
lengths received by the copier  ( present in these variables ...

long decompressedLength =
Long.parseLong(connection.getHeaderField(RAW_MAP_OUTPUT_LENGTH));
long compressedLength =
Long.parseLong(connection.getHeaderField(MAP_OUTPUT_LENGTH));  )

are not even multiples of the record size . I want to know how to get the
number of records from these lengths.

Iam doing all these as an experiment in my graduate research.. and if
everything works.. i can come up with a contrib file for the same.

Thanks,
Naresh Rapolu.

-- 
View this message in context: http://www.nabble.com/Re%3A-Need-help-understanding-the-source-tp24360327p24360327.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.


Mime
View raw message