hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject Re: Can Hadoop replace the use of MQ b/w processes?
Date Mon, 20 Aug 2012 01:31:19 GMT
Glad to hear about Hamake. FWIW, I've had good success with Azkaban in the
past for very complex, lengthy Hadoop/Pig/Streaming pipelines. It even has
a DAG GUI.

On Sun, Aug 19, 2012 at 5:43 PM, Lance Norskog <goksron@gmail.com> wrote:

> Last checkin on Azkaban was 11 months ago:
>
> https://github.com/azkaban/azkaban/commit/b105570625bcb2002de1acf4012c8d0e4388470a
>
> But, the last checkin for Hamake was June 2010. And it's still a cool
> little Hadoop/Pig scheduler.
> http://hamake.googlecode.com/
>
> On Sun, Aug 19, 2012 at 2:49 PM, Michael Segel
> <michael_segel@hotmail.com> wrote:
> > There has been some work to replace the use of queues with HBase.
> > This would be used to feed processes off the queue to help balance out
> the load on the cluster.
> >
> > In one specific use case, this was effective because the time spent
> processing each mapper.map() iteration is a couple of orders of magnitude
> as the time it takes to pull the data from the 'queue' and to each node for
> processing.
> >
> > Again, YMMV, it is an interesting hack though....
> >
> > On Aug 19, 2012, at 11:46 AM, Robert Nicholson <
> robert.nicholson@gmail.com> wrote:
> >
> >> We have an application or a series of applications that listen to
> incoming feeds they then distribute this data in XML form to a number of
> queues.  Another set of processes listen to these queues and process the
> messages. Order of processing is important in so far as related messages
> need to be processed in sequence hence today all related messages go to the
> same queue and are processed by the same queue consumer.
> >>
> >> The idea would be replace the use of MQ with some kind of reliable
> distributed dispatch. Does Hadoop provide that?
> >>
> >>
> >>
> >>
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com

Mime
View raw message