Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of asrabkin@gmail.com designates
 209.85.161.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAB9+7yjuC18JyZyaV5+7cFZg0UJaOqrax99MRuaxTJ0fgjD5fg@mail.gmail.com>
References: 
 <CAB9+7yjuC18JyZyaV5+7cFZg0UJaOqrax99MRuaxTJ0fgjD5fg@mail.gmail.com>
Date: Fri, 29 Jul 2011 18:54:46 -0400
Message-ID: 
 <CAJcOcz6PptenwVxPyS8H+aKfD=FRzD9wzVRdDk4mAzNTchaJsA@mail.gmail.com>
Subject: Re: Agent and collector
From: Ariel Rabkin <asrabkin@gmail.com>
To: chukwa-user@incubator.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Yes, the agent-->collector path is HTTP.

This was done precisely to allow load balancers. I don't know how
tested that configuration is, though. I think most sites had Chukwa
itself do the load balancing by specifying multiple collectors.

There is a notion of end-to-end reliability; the so-called
asynchronous ack mechanism. It's off by default and hasn't been tried
much in production. See
http://www.usenix.org/events/lisa10/tech/full_papers/Rabkin.pdf for
the detailed design of it.

--Ari

On Fri, Jul 29, 2011 at 11:04 AM, T. A. Smooth <catdaaaady@gmail.com> wrote=
:
> Hello I am checking out Chukwa. I have a few questions I was hoping the m=
ail
> list could answer :-)
>
> 1)Does Chukwa agents communicate to collectors over http? Or some other
> protocol?
>
> The agent configuration makes me believe that:
> http://incubator.apache.org/chukwa/docs/r0.4.0/admin.html#Configuration
>
> 2) And the docs it seems an Agent will pick a collector at random and the=
n
> use that collect until there is a problem in communicating with it. How d=
o
> you think the agent/collector would act if they have a load balancer betw=
een
> them? For example, the agent configuration would have just one url
> http://collector-loadbalancer. example.com:8080/
>
> The load balancer would have 1 or more collectors behind it saving the
> chunks it receives to disk or hadoop.
>
> 3) Does chukwa have any =93end-to-end=94 reliability features for message
> delivery? For example, a collector may receive the chunk from the agent b=
ut
> it may have a problem writing it to the data store. (ie. Disk space full,
> connection to hadoop down) . Will the agent be notified that the chunk wa=
s
> not processed for a certain reason and the agent is told to cache to disk
> the missed message?
>
> Thanks for the info!
>
> -tp-


--=20
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department