hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <mse...@segel.com>
Subject Re: Saving counters in Mapfile
Date Wed, 24 Jul 2013 02:26:32 GMT

You want to save the counters as in counts per job run or something? (Remember HDFS == WORM)

Then you could do a sequence file and then use something like HBase to manage the index. 
(Every time you add a set of counters, you have a new file and a new index.) 
Heck you could use HBase for the whole thing but it would be overkill if this was all that
you were doing. 

On Jul 23, 2013, at 4:57 PM, Elazar Leibovich <elazarl@gmail.com> wrote:

> Hi,
> A common use case one want an ordered structure for, is for saving counters.
> Naturally, I wanted to save the counters in a Mapfile:
>     for (long ix = 0; ix < MAXVALUE; ix++) {
>         mapfile.append(new Text("counter key of val " + ix), new LongWritable(ix));
>     }
> This however looks a bit inefficient. We'll store two files, and an index file. The index
file will contain an offset (long) to the sequence file, which would contain a single long.
> I'd rather have only the index file, that would store the counter value instead of offsets.
> Is there a way to do that with Mapfile? Perhaps there's a better way to save searchable
counters in HDFS file?
> Thanks,

View raw message