hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Some new requests about mapreduce
Date Mon, 06 Nov 2006 17:04:07 GMT
Feng Jiang wrote:
> I think some features are very useful for us:
> 
> 1. Multi-key types supported in input. for example: SEQ file A is <Ka, Va>
> pair, and SEQ file B is <Kb, Vb> pair. I can simply add both of these files
> as input file, and the map funtion could be map(Object, Ojbect). By this
> way, i don't have to wrap Ka and Kb into ObjectWritable, and the program
> will be more readable.

This is addressed by http://issues.apache.org/jira/browse/HADOOP-372.

> 2. Value comparator supported. There is key comparator supported in current
> hadoop, and by this way, i can specify the order the key in reduce phase.
> But sometimes, i also need specify the order the value sequence in reduce
> phase. For example, values in reduce phase consist of Shop and Goods, and i
> want to the Shop object always be the 1st object in the values because the
> output needs shop infor. Currently i have to store the Goods Info in a
> buffer until the Shop object has been found.

This is addressed by http://issues.apache.org/jira/browse/HADOOP-485.

> 3. More effective "ObjectWritable". Look at the ObjectWritable's
> implementation, the class type information is always written into sequence
> file. But in many cases, both of key and value are pretty small, the class
> type information is even much larger than key& value themselves.

ObjectWritable is not used so much for bulk data, but rather for small 
items, like RPC parameters, so the size overhead is usually not an 
issue.  Where are you finding this overhead onerous?

> 4. Compression supported. Sequence file contains a lot of similar data, if
> it could be compressed before it is really written into disk, a lot of time
> will be saved. For example, if the value type is ObjectWritable, there must
> be a lot of class declaration information could be compressed. In my
> experience, 20% bandwidth and disk space will be saved.

SequenceFile already supports compression, with extensible codecs:

http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/io/SequenceFile.html

So far, this uses Java's built-in codecs.  In the next release we also 
hope to include native support for zlib and lzo, greatly improving 
compression performance.

http://issues.apache.org/jira/browse/HADOOP-538

Lzo doesn't compress quite as well as zlib, but it's much faster.  In 
particular, zlib is generally slower than disk & net, while lzo is 
faster.  So zlib tends to save space but not time, while lzo should save 
both.

Doug

Mime
View raw message