Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of eric818@gmail.com designates
 74.125.82.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAP=R=qhA45PEz=COHacCqpJkJPrYu9P47Xd_x-oTUnVjqtSTsg@mail.gmail.com>
References: 
 <CAJ1-Qf7+ZH_b_skyzgRjUkHEyYRNLQ3pE+iZTS5aXiycQkfRUg@mail.gmail.com>
	<CADfbTpHiHDiwCGSbTwSTQLhCHXy2CVKuaCGcfCi=fmE0=5dNAw@mail.gmail.com>
	<CAP=R=qhrP-YLf53Hj=+x48sjMa=xzet9H=jsW4k=7RCNiTHckQ@mail.gmail.com>
	<CAP=R=qhA45PEz=COHacCqpJkJPrYu9P47Xd_x-oTUnVjqtSTsg@mail.gmail.com>
Date: Wed, 4 Apr 2012 20:04:59 -0700
Message-ID: 
 <CADfbTpE5g7Nv42tG7Y3n1fa_LrDDP=nmTjkFUAzNe+9+9d==mw@mail.gmail.com>
Subject: Re: Hadoop metric are not collected on amazon EMR
From: Eric Yang <eric818@gmail.com>
To: chukwa-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Which version of Hadoop are you using?  This looks like a mismatch
between how Chukwa
calls DFS client, and Hadoop jar file.

regards,
Eric

On Wed, Apr 4, 2012 at 1:43 AM, meena k.s <imsmiling.meena@gmail.com> wrote=
:
> Hi Eric,
>
> Regarding my earlier query, i see the following error message in
> chukwa-collector.log
>
> =A0WARN main SeqFileWriter - Got an exception trying to rotate. Will try =
again
> in 300 seconds.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.protocol.ClientProtocol.create(java.lang.String,
> org.apache.hadoop.fs.permission.FsPermission, java.lang.String, boolean,
> boolean, short, long)
> =A0 =A0 =A0 =A0 at java.lang.Class.getMethod(Class.java:1605)
> =A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> =A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java=
:1388)
> =A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java=
:1384)
> =A0 =A0 =A0 =A0 at java.security.AccessController.doPrivileged(Native Met=
hod)
> =A0 =A0 =A0 =A0 at javax.security.auth.Subject.doAs(Subject.java:396)
> Thanks,
> Meena
>
> On Wed, Apr 4, 2012 at 1:33 PM, meena k.s <imsmiling.meena@gmail.com> wro=
te:
>>
>> Hi Eric,
>>
>> Thanks for the quick reply.I have tried to install chukwa 0.5 accordingl=
y.
>> but there seems to be a problem while starting chukwa-collector. In the =
log
>> files i could see something related to hbase. I haven't installed any hb=
ase
>> in my cluster. Therefore I commented out the parameters related to hbase=
 in
>> my chukwa-collector-conf.xml. Is it necessary to have a hbase installati=
on
>> for chukwa 0.5 to work.
>>
>> Thanks,
>> Meena
>>
>> On Tue, Apr 3, 2012 at 9:36 PM, Eric Yang <eric818@gmail.com> wrote:
>>>
>>> Hi Meena,
>>>
>>> Hadoop 0.20.205 and Chukwa 0.4 is a mismatch. =A0Hadoop 0.20.20x has
>>> been modified to use MetricsSink instead of MentricsContext.
>>> Therefore, you will not see Hadoop metrics by using Chukwa 0.4. =A0You
>>> will need Chukwa 0.5 to monitor Hadoop 0.20.205+. =A0In addition, you
>>> should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor
>>> Hadoop. =A0Hope this helps.
>>>
>>> regards,
>>> Eric
>>>
>>> On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <ksmeena1806@gmail.com> wrote=
:
>>> > Hi,
>>> >
>>> >
>>> >
>>> > We want to use chukwa for hadoop log collection. We are running hive
>>> > queries on EMR. So for the purpose of experimenting with chukwa we
>>> > configured chukwa on a 2 node EMR cluster. =A0The collector and the
>>> > agent are running and also the system logs are generated but the
>>> > hadoop mapred and rpc /jvm logs are not being generated. Chukwa
>>> > version : 0.4 and the =A0hadoop version :0.20.205
>>> >
>>> >
>>> >
>>> > My hadoop-metrics.properties is as below :
>>> >
>>> >
>>> > # Licensed to the Apache Software Foundation (ASF) under one or more
>>> >
>>> > # contributor license agreements.=A0 See the NOTICE file distributed =
with
>>> >
>>> > # this work for additional information regarding copyright ownership.
>>> >
>>> > # The ASF licenses this file to You under the Apache License, Version
>>> > 2.0
>>> >
>>> > # (the "License"); you may not use this file except in compliance wit=
h
>>> >
>>> > # the License.=A0 You may obtain a copy of the License at
>>> >
>>> > #
>>> >
>>> > #=A0=A0=A0=A0 http://www.apache.org/licenses/LICENSE-2.0
>>> >
>>> > #
>>> >
>>> > # Unless required by applicable law or agreed to in writing, software
>>> >
>>> > # distributed under the License is distributed on an "AS IS" BASIS,
>>> >
>>> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>> > implied.
>>> >
>>> > # See the License for the specific language governing permissions and
>>> >
>>> > # limitations under the License.
>>> >
>>> >
>>> >
>>> > log4j.appender.chukwa.rpc.recordType=3DHadoopMetricsProcessor
>>> >
>>> > log4j.appender.chukwa.rpc.chukwaClientHostname=3Dlocalhost
>>> >
>>> > log4j.appender.chukwa.rpc.chukwaClientPortNum=3D9093
>>> >
>>> > log4j.appender.chukwa.rpc.DatePattern=3D.yyyy-MM-dd
>>> >
>>> > log4j.appender.chukwa.rpc.layout=3Dorg.apache.log4j.PatternLayout
>>> >
>>> > log4j.appender.chukwa.rpc.layout.ConversionPattern=3D%d{ISO8601} %p %=
c:
>>> > %m%n
>>> >
>>> > log4j.appender.chukwa.rpc.Dir=3D/tmp/chukwa/log/metrics
>>> >
>>> >
>>> >
>>> > log4j.appender.chukwa.dfs.recordType=3DHadoopMetricsProcessor
>>> >
>>> > log4j.appender.chukwa.dfs.chukwaClientHostname=3Dlocalhost
>>> >
>>> > log4j.appender.chukwa.dfs.chukwaClientPortNum=3D9093
>>> >
>>> > log4j.appender.chukwa.dfs.DatePattern=3D.yyyy-MM-dd
>>> >
>>> > log4j.appender.chukwa.dfs.layout=3Dorg.apache.log4j.PatternLayout
>>> >
>>> > log4j.appender.chukwa.dfs.layout.ConversionPattern=3D%d{ISO8601} %p %=
c:
>>> > %m%n
>>> >
>>> > log4j.appender.chukwa.dfs.Dir=3D/tmp/chukwa/log/metrics
>>> >
>>> >
>>> >
>>> > log4j.appender.chukwa.mapred.recordType=3DHadoopMetricsProcessor
>>> >
>>> > log4j.appender.chukwa.mapred.chukwaClientHostname=3Dlocalhost
>>> >
>>> > log4j.appender.chukwa.mapred.chukwaClientPortNum=3D9093
>>> >
>>> > log4j.appender.chukwa.mapred.DatePattern=3D.yyyy-MM-dd
>>> >
>>> > log4j.appender.chukwa.mapred.layout=3Dorg.apache.log4j.PatternLayout
>>> >
>>> > log4j.appender.chukwa.mapred.layout.ConversionPattern=3D%d{ISO8601} %=
p
>>> > %c: %m%n
>>> >
>>> > log4j.appender.chukwa.mapred.Dir=3D/tmp/chukwa/log/metrics
>>> >
>>> >
>>> >
>>> > I have also copied chukwa-hadoop-*-client.jar and json.jar to
>>> > HADOOP_HOME/lib, and restarted hadoop, but still the logs were not
>>> > generated.
>>> >
>>> > Also when I run telnet localhost 9093, the jvm or other hadoop metric=
s
>>> > are not running, but the other system adapters are running correctly.
>>> >
>>> > Can someone help me here..
>>> >
>>> > Thanks
>>> > Meena
>>
>>
>