Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (athena.apache.org: domain of terreyshih@gmail.com
 designates 209.85.192.173 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1990.1\))
Subject: Re: flume avro event overflow ?
From: terreyshih <terreyshih@gmail.com>
In-Reply-To: <3FE0B617-ED05-468D-A920-3AF5EE0A11B9@gmail.com>
Date: Sun, 19 Oct 2014 01:38:29 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <828FBF37-7433-40F3-9EF9-9F9B19C926E3@gmail.com>
References: <3FE0B617-ED05-468D-A920-3AF5EE0A11B9@gmail.com>
To: user@flume.apache.org

In other words, I would like to explicitly drop the events if the =
channel/event queue if full.

thx

> On Oct 18, 2014, at 11:42 PM, terreyshih <terreyshih@gmail.com> wrote:
>=20
> Hi,
>=20
> If I have a upstream agent duplicating the event to two channels and =
subsequently to two downstream sinks.  If one of the downstream sinks =
for some reason can not process the events fast enough and the event =
overflows back over the buffer limit of the upstream agent, would it =
cause any issues for the other channel since the events are duplicated ?
>=20
> In general, what happens if the downstream agent is not able to handle =
the speed of the upstream agent sending the events ?  I have seen =
exceptions that says unable to send events or memory channel commit =
issues. =20
>=20
> However, the flume agent does not seem to handle that gracefully and =
the whole agent just stops processing the events for all channels, even =
for the channels that can keep up with events.
>=20
> What if I separate the two channels into two different agents, would =
that solve the problem.
>=20
> Can upstream flume agent not just drop the events when one of the its =
sinks or downstream agents cannot catch up  ?
>=20
> thx