flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Buntu Dev <buntu...@gmail.com>
Subject Re: Determine the backlog in File channel
Date Wed, 04 Nov 2015 02:45:23 GMT
Thanks for the info Mike and would be great if you can mention the pipeline
to get the flume metrics into TSDB.

I actually wanted to know how to compute the throughput based on the
various metrics that Flume provides:

{
  "CHANNEL.my-file-channel": {
    "EventPutSuccessCount": "6500",
    "ChannelFillPercentage": "0.0",
    "Type": "CHANNEL",
    "StopTime": "0",
    "EventPutAttemptCount": "6500",
    "ChannelSize": "0",
    "StartTime": "1443053580229",
    "EventTakeSuccessCount": "6500",
    "ChannelCapacity": "1000000",
    "EventTakeAttemptCount": "164715352"
  }

Thanks!


On Tue, Nov 3, 2015 at 5:53 AM, Keane, Mike <mkeane@conversantmedia.com>
wrote:

> We ping our flume metrics every 5 minutes and push the results to a TSDB (
> http://opentsdb.net/) database.  TSDB is great for visualizing your data
> rates.  Depending on your flume configuration, you will get greatly varying
> rates.  If you are using spinning disks with a file channel you'll want to
> make sure your checkpoint directory and  and data directory are on
> physically different disks.
>
>
> ------------------------------
> *From:* Buntu Dev [buntudev@gmail.com]
> *Sent:* Monday, November 02, 2015 7:09 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Determine the backlog in File channel
>
> Awesome thanks!
>
> Whats the right way to calculate the throughput of the channel?
>
> On Mon, Nov 2, 2015 at 4:10 PM, Ahmed Vila <avila@devlogic.eu> wrote:
>
>> To add to it, here is the user manual reference:
>> https://flume.apache.org/FlumeUserGuide.html#json-reporting
>>
>> On Tue, Nov 3, 2015 at 1:06 AM, Keane, Mike <mkeane@conversantmedia.com>
>> wrote:
>>
>>> Enable monitoring and you can get the channelFillPercentage and
>>> channelSize from the metrics:
>>>
>>> VM arguments for http monitoring:
>>> -Dflume.monitoring.type=http -Dflume.monitoring.port=12345
>>>
>>> For JMX
>>> -Dcom.sun.management.jmxremote.port=12346
>>>
>>> or whatever port you choose.
>>>
>>>
>>> ------------------------------
>>> *From:* Buntu Dev [buntudev@gmail.com]
>>> *Sent:* Monday, November 02, 2015 4:33 PM
>>> *To:* user@flume.apache.org
>>> *Subject:* Determine the backlog in File channel
>>>
>>> I got a File channel with HDFS sink. In the case when the sink slows
>>> down and event taken from the channel falls behind while the event puts
>>> continue at the same pace, how would one go about finding the amount of
>>> backlog or time it takes to clear the backlog?
>>>
>>> Thanks!
>>>
>>> This email and any files included with it may contain privileged,
>>> proprietary and/or confidential information that is for the sole use
>>> of the intended recipient(s).  Any disclosure, copying, distribution,
>>> posting, or use of the information contained in or attached to this
>>> email is prohibited unless permitted by the sender.  If you have
>>> received this email in error, please immediately notify the sender
>>> via return email, telephone, or fax and destroy this original
>>> transmission
>>> and its included files without reading or saving it in any manner.
>>> Thank you.
>>>
>>
>>
>>
>> --
>>
>> Best regards,
>> Ahmed Vila | Senior software developer
>> DevLogic | Sarajevo | Bosnia and Herzegovina
>>
>> Office : +387 33 942 123
>> Mobile: +387 62 139 348
>>
>> Website: www.devlogic.eu
>> E-mail   : avila@devlogic.eu
>> ---------------------------------------------------------------------
>> This e-mail and any attachment is for authorised use by the intended
>> recipient(s) only. This email contains confidential information. It should
>> not be copied, disclosed to, retained or used by, any party other than the
>> intended recipient. Any unauthorised distribution, dissemination or copying
>> of this E-mail or its attachments, and/or any use of any information
>> contained in them, is strictly prohibited and may be illegal. If you are
>> not an intended recipient then please promptly delete this e-mail and any
>> attachment and all copies and inform the sender directly via email. Any
>> emails that you send to us may be monitored by systems or persons other
>> than the named communicant for the purposes of ascertaining whether the
>> communication complies with the law and company policies.
>>
>> ---------------------------------------------------------------------
>> This e-mail and any attachment is for authorised use by the intended
>> recipient(s) only. This email contains confidential information. It should
>> not be copied, disclosed to, retained or used by, any party other than the
>> intended recipient. Any unauthorised distribution, dissemination or copying
>> of this E-mail or its attachments, and/or any use of any information
>> contained in them, is strictly prohibited and may be illegal. If you are
>> not an intended recipient then please promptly delete this e-mail and any
>> attachment and all copies and inform the sender directly via email. Any
>> emails that you send to us may be monitored by systems or persons other
>> than the named communicant for the purposes of ascertaining whether the
>> communication complies with the law and company policies.
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s).  Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender.  If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>

Mime
View raw message