hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi" <runp...@yahoo-inc.com>
Subject RE: HashMap which can spill to disk for Hadoop?
Date Thu, 20 Dec 2007 04:20:03 GMT

It would be nice if you can contribute a file backed hashmap, or a file
backed implementation of the unique count aggregator.

Short of that, if you just need to count the unique values for each
event id, you can do so by using the aggregate classes with each
event-id/event-value pair as a key and simply counting the number of
occurrences of each composite key.

Runping


> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Wednesday, December 19, 2007 11:59 AM
> To: hadoop-user@lucene.apache.org
> Subject: HashMap which can spill to disk for Hadoop?
> 
> Hi All:
> 
>   The aggregation classes in Hadoop use a HashMap to hold unique
values in
> memory when computing unique counts, etc.  I ran into a situation on
32-
> node grid (4G memory/node) where a single node runs out of memory
within
> the reduce phase trying to manage a very large HashMap.  This was
> disappointing because the dataset is only 44M rows (4G) of data.  This
is
> a scenario where I am counting unique values associated with various
> events, where the total number of events is very small and the number
of
> unique values is very high.  Since the event IDs serve as keys as the
> number of distinct event IDs is small, there is a consequently small
> number of reducers running, where each reducer is expected to manage a
> very large HashMap of unique values.
> 
>   It looks like I need to build my own unique aggregator, so I am
looking
> for an implementation of HashMap which can spill to disk as needed.
I've
> considered using BDB as a backing store, and I've looking into Derby's
> BackingStoreHashtable as well.
> 
>   For the present time I can restructure my data in an attempt to get
more
> reducers to run, but I can see in the very near future where even that
> will run out of memory.
> 
>   Any thoughts,comments, or flames?
> 
>   Thanks,
>   C G
> 
> 
> 
> ---------------------------------
> Looking for last minute shopping deals?  Find them fast with Yahoo!
> Search.

Mime
View raw message