hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: Can Hadoop replace the use of MQ b/w processes?
Date Mon, 20 Aug 2012 00:43:01 GMT
Last checkin on Azkaban was 11 months ago:
https://github.com/azkaban/azkaban/commit/b105570625bcb2002de1acf4012c8d0e4388470a

But, the last checkin for Hamake was June 2010. And it's still a cool
little Hadoop/Pig scheduler.
http://hamake.googlecode.com/

On Sun, Aug 19, 2012 at 2:49 PM, Michael Segel
<michael_segel@hotmail.com> wrote:
> There has been some work to replace the use of queues with HBase.
> This would be used to feed processes off the queue to help balance out the load on the
cluster.
>
> In one specific use case, this was effective because the time spent processing each mapper.map()
iteration is a couple of orders of magnitude as the time it takes to pull the data from the
'queue' and to each node for processing.
>
> Again, YMMV, it is an interesting hack though....
>
> On Aug 19, 2012, at 11:46 AM, Robert Nicholson <robert.nicholson@gmail.com> wrote:
>
>> We have an application or a series of applications that listen to incoming feeds
they then distribute this data in XML form to a number of queues.  Another set of processes
listen to these queues and process the messages. Order of processing is important in so far
as related messages need to be processed in sequence hence today all related messages go to
the same queue and are processed by the same queue consumer.
>>
>> The idea would be replace the use of MQ with some kind of reliable distributed dispatch.
Does Hadoop provide that?
>>
>>
>>
>>
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message