Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of eric818@gmail.com designates
 209.85.214.175 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Multiple agents and multiple clusters
From: Eric Yang <eric818@gmail.com>
In-Reply-To: <ABBC256D-C664-4EB3-871F-4F397736F637@gmail.com>
Date: Fri, 15 Jul 2011 09:22:30 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <782B9D02-9525-49C3-82E4-2667A621DC27@gmail.com>
References: 
 <CAODJVXhPZzj-qTQkTW=3EKCzp_kFN55MUEq+bhkkQRg3Wjjhnw@mail.gmail.com>
 <CAJcOcz5yv8jkXz-Rk+8UGB7d0Wt8Yo++d1Ka-hh+aU3eu=RiXg@mail.gmail.com>
 <ABBC256D-C664-4EB3-871F-4F397736F637@gmail.com>
To: chukwa-user@incubator.apache.org

The data is written locally on the source node.  Both agents will =
progress independently, and sending data at their own pace to the =
collectors.  They shouldn't affect each other.  The =
UTF8FileTailingAdaptor should track the timestamp of the log files and =
send it over in time sequence order, hence one failed does not affect =
others.

regards,
Eric

On Jul 14, 2011, at 8:45 PM, highpointe wrote:

> Ahhhh. Good insight. I didn't think about that.
>=20
> Yes, I am aggregating the same data for two clusters.  =20
>=20
> So now my challenge is:  If one cluster is unreachable...  How do I =
that cluster's sender fail without affecting the other sender or the =
connector.=20
>=20
> Excellent insight Ari!  Thank you!
>=20
> Sent from my iPhone
>=20
> On Jul 14, 2011, at 11:54 AM, Ariel Rabkin <asrabkin@gmail.com> wrote:
>=20
>> Just to be clear -- you want the same data to end up in each cluster?
>>=20
>> You'll have to do some hacking. I think it won't be so very =
monstrous.
>> In the Agent, there's a notion of pluggable Connectors and Senders. I
>> think you want a new Connector with two senders.
>>=20
>> One caveat -- you'll need to think through what happens if just one
>> cluster is down, and perhaps other partial failure scenarios too.
>>=20
>> --Ari
>>=20
>> On Thu, Jul 14, 2011 at 10:44 AM, high pointe =
<highpointe3i@gmail.com> wrote:
>>> I have a situation where I need to take the one data stream, send it =
through
>>> two agents and have the end point residing in separate clusters.
>>> Rough explanation....
>>> Streaming Data Box ---> Chukwa Box
>>> Agent one on Chukwa Box sends feed to cluster one.
>>> Agent two on Chukwa box sends feed to cluster two.
>>> Does anyone know of a way to do this without hacking this baby to =
shreds?
>>>=20
>>> Thanks!
>>=20
>>=20
>>=20
>> --=20
>> Ari Rabkin asrabkin@gmail.com
>> UC Berkeley Computer Science Department