incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: Re: Re: Re: hadoop metrics isn't collected into mysql
Date Sat, 02 Jul 2011 05:04:33 GMT
Chukwa 0.4 only works with hadoop 0.20.x.  If you are using hadoop
0.2, then it probably doesn't work.

On the nodes that you see the log files, try:

telnet localhost 9093
list

Check if the jvm and mapred files are being streamed over chukwa agent.

If it is not, you might need to restart the mapreduce cluster.

regards,
Eric

2011/7/1 孙长 <smart02_ren@163.com>:
> yes, there are rpc and mapred logs in  log file
>
> hadoop-metrics.properties is like this, and hadoop version is 0.2
>
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License.  You may obtain a copy of the License at
> #
> #     http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> chukwaAgent.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> chukwaAgent.period=60
> chukwaAgent.directory=/tmp/chukwa/log/metrics/
>
> chukwaHttpSender.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> chukwaHttpSender.period=60
> chukwaHttpSender.directory=/tmp/chukwa/log/metrics/
> chunkQueue.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> chunkQueue.period=60
> chunkQueue.directory=/tmp/chukwa/log/metrics/
> dfs.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> dfs.period=60
> dfs.directory=/tmp/chukwa/log/metrics/
> dfs.uuid=true
> jvm.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> jvm.period=60
> jvm.directory=/tmp/chukwa/log/metrics/
> jvm.uuid=true
> mapred.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> mapred.period=60
> mapred.directory=/tmp/chukwa/log/metrics/
> mapred.uuid=true
> rpc.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> rpc.period=60
> rpc.directory=/tmp/chukwa/log/metrics/
> rpc.uuid=true
>
> At 2011-07-02 11:39:32,"Eric Yang" <eric818@gmail.com> wrote:
>
>>Check the log directory and see if you see any log file with rpc or
>>mapred in the file name?
>>What does the hadoop-metrics.properties look like?  What version of
>>hadoop are you running?
>>
>>regards,
>>Eric
>>
>>2011/7/1 孙长 <smart02_ren@163.com>:
>>> i know where the problem is , i didn't put log4j and hadoop-metrics file
>>> into all  nodes' hadoop conf dir.
>>> Now the problem disapears
>>> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc
>>> data, why?
>>> thank you~
>>>
>>>
>>> At 2011-07-01 22:59:41,"Eric Yang" <eric818@gmail.com> wrote:
>>>
>>>>Looks like mapreduce can not find json.jar in its class path.  Which
>>>>version of Hadoop are you using?
>>>>
>>>>regards,
>>>>Eric
>>>>
>>>>2011/7/1 孙长 <smart02_ren@163.com>:
>>>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>>>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
>>>>> error:
>>>>> task error is as follows:
>>>>>
>>>>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>>>> 	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>>>> 	at java.lang.Class.forName0(Native Method)
>>>>> 	at java.lang.Class.forName(Class.java:169)
>>>>> 	at
>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>>>>> 	at
>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>>>>> 	at
>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>>>>> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>>> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>>
>>>>> At 2011-07-01 13:17:42,"Eric Yang" <eric818@gmail.com> wrote:
>>>>>
>>>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>>>>HADOOP_HOME/lib, and restart hadoop?
>>>>>>
>>>>>>regards,
>>>>>>Eric
>>>>>>
>>>>>>2011/6/30 孙长 <smart02_ren@163.com>:
>>>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>>>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>>>>>> dfs, mapred
>>>>>>> dbAdmin.sh is running background, log4j.properties
>>>>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>>>>>> guide
>>>>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>>>>> thank you ~
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>
>
>

Mime
View raw message