incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "meena k.s" <imsmiling.me...@gmail.com>
Subject Re: Hadoop metric are not collected on amazon EMR
Date Wed, 04 Apr 2012 08:03:02 GMT
Hi Eric,

Thanks for the quick reply.I have tried to install chukwa 0.5 accordingly.
but there seems to be a problem while starting chukwa-collector. In the log
files i could see something related to hbase. I haven't installed any hbase
in my cluster. Therefore I commented out the parameters related to hbase in
my chukwa-collector-conf.xml. Is it necessary to have a hbase installation
for chukwa 0.5 to work.

Thanks,
Meena

On Tue, Apr 3, 2012 at 9:36 PM, Eric Yang <eric818@gmail.com> wrote:

> Hi Meena,
>
> Hadoop 0.20.205 and Chukwa 0.4 is a mismatch.  Hadoop 0.20.20x has
> been modified to use MetricsSink instead of MentricsContext.
> Therefore, you will not see Hadoop metrics by using Chukwa 0.4.  You
> will need Chukwa 0.5 to monitor Hadoop 0.20.205+.  In addition, you
> should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor
> Hadoop.  Hope this helps.
>
> regards,
> Eric
>
> On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <ksmeena1806@gmail.com> wrote:
> > Hi,
> >
> >
> >
> > We want to use chukwa for hadoop log collection. We are running hive
> > queries on EMR. So for the purpose of experimenting with chukwa we
> > configured chukwa on a 2 node EMR cluster.  The collector and the
> > agent are running and also the system logs are generated but the
> > hadoop mapred and rpc /jvm logs are not being generated. Chukwa
> > version : 0.4 and the  hadoop version :0.20.205
> >
> >
> >
> > My hadoop-metrics.properties is as below :
> >
> >
> > # Licensed to the Apache Software Foundation (ASF) under one or more
> >
> > # contributor license agreements.  See the NOTICE file distributed with
> >
> > # this work for additional information regarding copyright ownership.
> >
> > # The ASF licenses this file to You under the Apache License, Version 2.0
> >
> > # (the "License"); you may not use this file except in compliance with
> >
> > # the License.  You may obtain a copy of the License at
> >
> > #
> >
> > #     http://www.apache.org/licenses/LICENSE-2.0
> >
> > #
> >
> > # Unless required by applicable law or agreed to in writing, software
> >
> > # distributed under the License is distributed on an "AS IS" BASIS,
> >
> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> >
> > # See the License for the specific language governing permissions and
> >
> > # limitations under the License.
> >
> >
> >
> > log4j.appender.chukwa.rpc.recordType=HadoopMetricsProcessor
> >
> > log4j.appender.chukwa.rpc.chukwaClientHostname=localhost
> >
> > log4j.appender.chukwa.rpc.chukwaClientPortNum=9093
> >
> > log4j.appender.chukwa.rpc.DatePattern=.yyyy-MM-dd
> >
> > log4j.appender.chukwa.rpc.layout=org.apache.log4j.PatternLayout
> >
> > log4j.appender.chukwa.rpc.layout.ConversionPattern=%d{ISO8601} %p %c:
> %m%n
> >
> > log4j.appender.chukwa.rpc.Dir=/tmp/chukwa/log/metrics
> >
> >
> >
> > log4j.appender.chukwa.dfs.recordType=HadoopMetricsProcessor
> >
> > log4j.appender.chukwa.dfs.chukwaClientHostname=localhost
> >
> > log4j.appender.chukwa.dfs.chukwaClientPortNum=9093
> >
> > log4j.appender.chukwa.dfs.DatePattern=.yyyy-MM-dd
> >
> > log4j.appender.chukwa.dfs.layout=org.apache.log4j.PatternLayout
> >
> > log4j.appender.chukwa.dfs.layout.ConversionPattern=%d{ISO8601} %p %c:
> %m%n
> >
> > log4j.appender.chukwa.dfs.Dir=/tmp/chukwa/log/metrics
> >
> >
> >
> > log4j.appender.chukwa.mapred.recordType=HadoopMetricsProcessor
> >
> > log4j.appender.chukwa.mapred.chukwaClientHostname=localhost
> >
> > log4j.appender.chukwa.mapred.chukwaClientPortNum=9093
> >
> > log4j.appender.chukwa.mapred.DatePattern=.yyyy-MM-dd
> >
> > log4j.appender.chukwa.mapred.layout=org.apache.log4j.PatternLayout
> >
> > log4j.appender.chukwa.mapred.layout.ConversionPattern=%d{ISO8601} %p %c:
> %m%n
> >
> > log4j.appender.chukwa.mapred.Dir=/tmp/chukwa/log/metrics
> >
> >
> >
> > I have also copied chukwa-hadoop-*-client.jar and json.jar to
> > HADOOP_HOME/lib, and restarted hadoop, but still the logs were not
> > generated.
> >
> > Also when I run telnet localhost 9093, the jvm or other hadoop metrics
> > are not running, but the other system adapters are running correctly.
> >
> > Can someone help me here..
> >
> > Thanks
> > Meena
>

Mime
View raw message