Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com
 designates 65.55.111.97 as permitted sender)
Message-ID: <BLU0-SMTP153E073412EEAD33B7D53558FBA0@phx.gbl>
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Mac OS X Mail 6.0 \(1485\))
Subject: Re: Can Hadoop replace the use of MQ b/w processes?
From: Michael Segel <michael_segel@hotmail.com>
In-Reply-To: <48ECEBDF-1210-4E0C-B87D-1DC6493A6547@gmail.com>
Date: Sun, 19 Aug 2012 16:49:52 -0500
Content-Transfer-Encoding: quoted-printable
References: <48ECEBDF-1210-4E0C-B87D-1DC6493A6547@gmail.com>
To: user@hadoop.apache.org

There has been some work to replace the use of queues with HBase.=20
This would be used to feed processes off the queue to help balance out =
the load on the cluster.=20

In one specific use case, this was effective because the time spent =
processing each mapper.map() iteration is a couple of orders of =
magnitude as the time it takes to pull the data from the 'queue' and to =
each node for processing.=20

Again, YMMV, it is an interesting hack though....

On Aug 19, 2012, at 11:46 AM, Robert Nicholson =
<robert.nicholson@gmail.com> wrote:

> We have an application or a series of applications that listen to =
incoming feeds they then distribute this data in XML form to a number of =
queues.  Another set of processes listen to these queues and process the =
messages. Order of processing is important in so far as related messages =
need to be processed in sequence hence today all related messages go to =
the same queue and are processed by the same queue consumer.
>=20
> The idea would be replace the use of MQ with some kind of reliable =
distributed dispatch. Does Hadoop provide that?
>=20
>=20
>=20
>=20