hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "manishbhoge@rocketmail.com" <manishbh...@rocketmail.com>
Subject Re: Saving counters in Mapfile
Date Wed, 24 Jul 2013 02:45:55 GMT
If you intend to use those counters in further functions then I think Hadoop will take care
this by itself, you can explore combiner for the same. In Sequence file mapfile and index
files has its own functionality and as per my understanding index file expect offset to move
over map file and logically you can't use it for counter.

Sent via Rocket from my HTC 

----- Reply message -----
From: "Elazar Leibovich" <elazarl@gmail.com>
To: <user@hadoop.apache.org>
Subject: Saving counters in Mapfile
Date: Wed, Jul 24, 2013 3:27 AM


A common use case one want an ordered structure for, is for saving counters.

Naturally, I wanted to save the counters in a Mapfile:

    for (long ix = 0; ix < MAXVALUE; ix++) {
        mapfile.append(new Text("counter key of val " + ix), new

This however looks a bit inefficient. We'll store two files, and an index
file. The index file will contain an offset (long) to the sequence file,
which would contain a single long.

I'd rather have only the index file, that would store the counter value
instead of offsets.

Is there a way to do that with Mapfile? Perhaps there's a better way to
save searchable counters in HDFS file?

View raw message