hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aseem Anand <aseem.ii...@gmail.com>
Subject Re: PriorityQueueWritable
Date Mon, 15 Oct 2012 18:07:00 GMT
Hi Chris,
I had a few PriorityQueue's at the mappers which I wished to send to some
reducers. After this each reducer(receiving PriorityQueues from each
mapper) would perform some operations on these by removing the top and
hence accessing the elements in sorted order(which is very essential to my
application). Even I thought of pushing them in an ArrayWritable but was
wondering if there would be an existing implementation of PriorityQueue.
Would it be advisable to insert elements into ArrayWritable in sorted order
and reconstruction of merged PriorityQueues at the other end now ?


On Mon, Oct 15, 2012 at 11:07 PM, Chris Nauroth <cnauroth@hortonworks.com>wrote:

> Hello Aseem,
> I'm aware of nothing in Hadoop or related projects that provides a
> PriorityQueueWritable.  You could achieve this by taking some existing
> priority queue class and subclassing it or wrapping it to implement the
> Writable.write and Writable.readFields methods.
> If you could give us some additional context around what you want to
> solve, then we might be able to offer some other suggestions.  For example,
> depending on the problem, maybe you could sort values and wrap them in
> ArrayWritable (which already exists), which would save you the trouble of
> coding your own custom Writable.
> Thank you,
> --Chris
> On Mon, Oct 15, 2012 at 9:56 AM, Aseem Anand <aseem.iiith@gmail.com>wrote:
>> Hi,
>> Is anyone familiar with a PriorityQueueWritable to be used to pass data
>> from mapper to reducers ?
>> Regards,
>> Aseem

View raw message