Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 594A1EBCF for ; Wed, 23 Jan 2013 21:38:59 +0000 (UTC) Received: (qmail 10875 invoked by uid 500); 23 Jan 2013 21:38:59 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 10841 invoked by uid 500); 23 Jan 2013 21:38:59 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 10828 invoked by uid 99); 23 Jan 2013 21:38:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 21:38:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hshreedharan@cloudera.com designates 209.85.220.47 as permitted sender) Received: from [209.85.220.47] (HELO mail-pa0-f47.google.com) (209.85.220.47) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 21:38:53 +0000 Received: by mail-pa0-f47.google.com with SMTP id fa10so5002203pad.6 for ; Wed, 23 Jan 2013 13:38:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:date:from:to:message-id:in-reply-to:references:subject :x-mailer:mime-version:content-type:x-gm-message-state; bh=rpradV5rw/LjSdgT7l7vFWrPA9UnThZt+Lm45UmO1O4=; b=TYvTNjyXS6IZJm56wtsTvFjPP8WLkFKneGJEOTj+cRBRNd8YPvNlJK6LUlhbRC1sC8 0Vdu+GP05vLc28Brry6e2rmbNF6dcnjW/bZhLdd05gRlbgVzUxn6sGZUm/rb0HkxbAzw znizoZaST1EoZFYbyYyRVkUlYFuAeo328DwzsNV3jchFAEXtcyaV9KRn43+nArFSfLOx qqcFgYjd7OetF34fnoAC9+R3zhegN+BIQoK+Jyv69LBoLcSEhrXa2D+zX9VYwve+68UL ol2MsCyd5oZI/CZvFtaV+7TKCZ9CX7mfcsuYeo79VN1j5WVqb1nse72ahvw2DvT7hzPC STBA== X-Received: by 10.68.233.197 with SMTP id ty5mr7169335pbc.9.1358977113491; Wed, 23 Jan 2013 13:38:33 -0800 (PST) Received: from [10.0.1.8] (c-67-161-24-53.hsd1.ca.comcast.net. [67.161.24.53]) by mx.google.com with ESMTPS id mz10sm13461705pbc.37.2013.01.23.13.38.31 (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 23 Jan 2013 13:38:32 -0800 (PST) Date: Wed, 23 Jan 2013 13:38:29 -0800 From: Hari Shreedharan To: user@flume.apache.org Message-ID: <81B3BAA9F139449CB6CCEF5A12D1A471@cloudera.com> In-Reply-To: References: <50C85E94.5050207@cyberagent.co.jp> Subject: Re: Setting up flume to use ganglia results in a lot of error messages in /var/log/messages X-Mailer: sparrow 1.6.4 (build 1178) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="51005855_7de67713_ce" X-Gm-Message-State: ALoCoQnQ4qjN68JQCRQQFbEx/VD+ajdP4QRE76xsehYes3qiKKf5bQQxBiCnYkISmzRyh9PnAi/8 X-Virus-Checked: Checked by ClamAV on apache.org --51005855_7de67713_ce Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Looks like this is happening because for certain metrics which cannot be = converted into float (for example strings), we are sending the type as fl= oat, causing Ganglia to log those messages. I think it should be fairly e= asy to write a patch to fix this. I filed =46LUME-1870 to track this. =20 Thanks, Hari -- =20 Hari Shreedharan On Wednesday, January 23, 2013 at 1:23 PM, Mike Percy wrote: > Not sure when or how it broke, as I know of people using it in producti= on. There is a way to configure it for different versions of Ganglia, lik= e 3.0, 3.1. Might be worth trying both values to see if it's a problem wi= th one or the other: http://flume.apache.org/=46lumeUserGuide.html=23gang= lia-reporting > =20 > =20 > On Wed, Jan 23, 2013 at 1:12 PM, Connor Woodson wrote: > > I have not had success with Ganglia, due to the same issue I think as= you've encountered. > > =20 > > - Connor > > =20 > > =20 > > On Wed, Jan 23, 2013 at 6:04 AM, Christian Schroer wrote: > > > Hi, > > > =20 > > > i have the same problem here, using =46lume-NG 1.2 from CDH4.1.2. > > > I deleted all related RRDs, gmetad recreates them and i see those e= rrors again. > > > =20 > > > Disk space, inodes, ... are fine. All RRDs not related to =46lume-N= G are fine, too. > > > =20 > > > These errors result in a gmetad crash after while (more metrics =3D= > earlier crash). If I disable ganglia support gmetad runs without any pr= oblem. > > > =20 > > > Regards > > > Christian > > > =20 > > > -----Urspr=C3=BCngliche Nachricht----- > > > Von: Alexander Alten-Lorenz =5Bmailto:wget.null=40gmail.com=5D > > > Gesendet: Mittwoch, 12. Dezember 2012 15:43 > > > An: user=40flume.apache.org (mailto:user=40flume.apache.org) > > > Betreff: Re: Setting up flume to use ganglia results in a lot of er= ror messages in /var/log/messages > > > =20 > > > Looks like the RRD's are damaged, maybe the harddisk full=3F > > > =20 > > > - Alex > > > =20 > > > On Dec 12, 2012, at 11:38 AM, Juhani Connolly wrote: > > > =20 > > > > We just noticed that ganglia's gmetad is spamming messages like t= he following > > > > > > > > Dec 9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5Fu= pdate (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.SINK.avro2.Typ= e.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.SINK.avro2.Ty= pe.rrd: conversion of 'SINK' to float not complete: tail 'SINK' > > > > Dec 9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5Fu= pdate (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.CHANNEL.ch1.Ty= pe.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.CHANNEL.ch1.= Type.rrd: conversion of 'CHANNEL' to float not complete: tail 'CHANNEL' > > > > Dec 9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5Fu= pdate (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SINK.avro1.Typ= e.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SINK.avro1.Ty= pe.rrd: conversion of 'SINK' to float not complete: tail 'SINK' > > > > Dec 9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5Fu= pdate (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SOURCE.scribe.= Type.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SOURCE.scr= ibe.Type.rrd: conversion of 'SOURCE' to float not complete: tail 'SOURCE'= > > > > > > > > The counters are tracked fine on the web interface, but we've had= issues(gmetad crashing or not starting up, I'm still trying to get speci= fics from the responsible people). I can't say for sure if this is a gang= lia problem or a problem with flumes ganglia support. Anyone else feeding= their counters to ganglia getting similar=3F Perhaps the ganglia ml may = be more appropriate, not sure on this one. > > > =20 > > > -- > > > Alexander Alten-Lorenz > > > http://mapredit.blogspot.com > > > German Hadoop LinkedIn Group: http://goo.gl/N8pC=46 > > > =20 > > =20 > =20 --51005855_7de67713_ce Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Looks like this is happening because for certain metr= ics which cannot be converted into float (for example strings), we are se= nding the type as float, causing Ganglia to log those messages. I think i= t should be fairly easy to write a patch to fix this. I filed =46LUME-187= 0 to track this.


Thanks,
Hari

-- 
Hari Shreedha= ran

=20

On Wednesday, January = 23, 2013 at 1:23 PM, Mike Percy wrote:

Not sure when or= how it broke, as I know of people using it in production. There is a way= to configure it for different versions of Ganglia, like 3.0, 3.1. Might = be worth trying both values to see if it's a problem with one or the othe= r: http://flume.apache.org/=46lumeUserGuide.html=23ganglia-= reporting


On Wed, Jan 23, 2013 at 1:12 PM, Connor Woodson <cwoodson.dev=40gmail.com> wrote:
I have not had success= with Ganglia, due to the same issue I think as you've encountered.=

- Connor


<= div>On Wed, Jan 23, 2013 at 6:04 AM, Christian Schroer <cschroer=40autoscout24.com> wrote:
Hi,

i have the same problem here, using =46lume-NG 1.2 from CDH4.1.2.
I deleted all related RRDs, gmetad recreates them and i see those errors = again.

Disk space, inodes, ... are fine. All RRDs not related to =46lume-NG are = fine, too.

These errors result in a gmetad crash after while (more metrics =3D> e= arlier crash). If I disable ganglia support gmetad runs without any probl= em.

Regards
Christian

-----Urspr=C3=BCngliche Nachricht-----
Von: Alexander Alten-Lorenz =5Bmailto:wget.null=40gmail.com=5D
Gesendet: Mittwoch, 12. Dezember 2012 15:43
An: user=40flume.apache.org
Betreff: Re: Setting up flume to use ganglia results in a lot of error me= ssages in /var/log/messages

Looks like the RRD's are damaged, maybe the harddisk full=3F

- Alex

On Dec 12, 2012, at 11:38 AM, Juhani Connolly <juhani=5Fcon= nolly=40cyberagent.co.jp> wrote:

> We just noticed that ganglia's gmetad is spamming messages like the = following
>
> Dec  9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5F= update (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.SINK.avro2.Ty= pe.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.SINK.avro2.T= ype.rrd: conversion of 'SINK' to float not complete: tail 'SINK'
> Dec  9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5F= update (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.CHANNEL.ch1.T= ype.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap02/flume.CHANNEL.ch1= .Type.rrd: conversion of 'CHANNEL' to float not complete: tail 'CHANNEL'<= br> > Dec  9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5F= update (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SINK.avro1.Ty= pe.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SINK.avro1.T= ype.rrd: conversion of 'SINK' to float not complete: tail 'SINK'
> Dec  9 03:13:20 om-pat-obs01 /usr/sbin/gmetad=5B17407=5D: RRD=5F= update (/var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SOURCE.scribe= .Type.rrd): /var/lib/ganglia/rrds/=46lume KDDI/blog-wap11/flume.SOURCE.sc= ribe.Type.rrd: conversion of 'SOURCE' to float not complete: tail 'SOURCE= '
>
> The counters are tracked fine on the web interface, but we've had is= sues(gmetad crashing or not starting up, I'm still trying to get specific= s from the responsible people). I can't say for sure if this is a ganglia= problem or a problem with flumes ganglia support. Anyone else feeding th= eir counters to ganglia getting similar=3F Perhaps the ganglia ml may be = more appropriate, not sure on this one.

--
Alexander Alten-Lorenz
http= ://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pC=46



=20 =20 =20 =20 =20

--51005855_7de67713_ce--