flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlotta Hicks <Carlotta.Hi...@sas.com>
Subject RE: Monitoring the progress of events
Date Mon, 26 Jan 2015 22:29:58 GMT
Are these the counters from MonitoredCounterGroup?  What is the scope of these counters?  Can
you reset these counters? 

-----Original Message-----
From: Joey Echeverria [mailto:joey@cloudera.com] 
Sent: Monday, January 26, 2015 5:20 PM
To: user@flume.apache.org
Subject: Re: Monitoring the progress of events

Assuming you're using the AvroSource, the application will have to do something like this:

1. Send all of the events to the source
2. Send a specially crafted canary event to the source 3. Monitor the destination where the
sink writes looking for the canary event

You might also be able to monitor the counters that the agent maintains to see if the expected
number of events have been processed by the sink.

-Joey

On Mon, Jan 26, 2015 at 2:17 PM, Carlotta Hicks <Carlotta.Hicks@sas.com> wrote:
> I've created a simple agent with a source, a channel and a sink.  How can an application
know when all events sent to the source have been processed by the sink?
>
> -----Original Message-----
> From: Joey Echeverria [mailto:joey@cloudera.com]
> Sent: Monday, January 26, 2015 5:06 PM
> To: user@flume.apache.org
> Subject: Re: Monitoring the progress of events
>
> @Seshu: If you emailed user-unsubscribe@flume.apache.org and it still didn't unsubscribe
you you should file a JIRA with Apache's infrastructure team:
>
> https://issues.apache.org/jira/browse/INFRA/?selectedTab=com.atlassian
> .jira.jira-projects-plugin:summary-panel
>
> @Carlotta:
>
> What kind of monitoring are you looking for? I don't believe there is event-level monitoring
as this would be very expensive. What I've seen some folks do is send a canary event to verify
when processing of some batch has completed. This won't work if you have fan out/fan in in
the flow as the ordering between multiple Flume agent's isn't guaranteed.
>
> -Joey
>
> On Mon, Jan 26, 2015 at 12:52 PM, Seshu V <sesh12@gmail.com> wrote:
>> Please take my id off of this group.  Unsubscribe doesn't seem to be 
>> working.
>>
>> thanks.
>>
>> On Mon, Jan 26, 2015 at 2:48 PM, Carlotta Hicks 
>> <Carlotta.Hicks@sas.com>
>> wrote:
>>>
>>> Is there a way to monitor the progress of events from the time they 
>>> are sent to the agent until they have been processed by the sink?  
>>> Is there a status for an agent?
>>>
>>>
>>
>>
>
>
>
> --
> Joey Echeverria



--
Joey Echeverria
Mime
View raw message