incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 孙长 <smart02_...@163.com>
Subject hadoop metrics isn't collected into mysql
Date Sat, 02 Jul 2011 07:16:45 GMT


my hadoop version is 0.20.1
i telnet localhost 9093 and list, then shows:


adaptor_8567b8b00a5dc746ccd5b8d179873db1)  org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor
Top 60 /usr/bin/top -b -n 1 -c 2110354484
adaptor_4be0d8af3bc253eef29983e5dbd4f0d5)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309528269962.log 150865
adaptor_b5382452cfdc2ae81bd45d8126804843)  org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor
Iostat 60 /usr/bin/iostat -x -k 55 2 4771609
adaptor_f0b4a4e9d4549f38f633c451e06e290f)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-jobtracker-hadoop10-131-4-70.log
63614063
adaptor_8890f1df2c09c514450025c0502f473d)  org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor
Sar 60 /usr/bin/sar -q -r -n ALL 55 129031858
adaptor_bbdbf9f9ccd112b1d2973aabf4e7c422)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578850828.log 18561
adaptor_5df3de68e136f337c5a46b9bf04637d1)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-secondarynamenode-hadoop10-131-4-70.log
51747
adaptor_31842d1822977c3c492c725cc8b81bda)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527999599.log 380835
adaptor_0e10bfd94287c9a1c2983c7e103e39f0)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527883608.log 1060785
adaptor_46d675cf8ccd92e245509b19f28dcea8)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309527883619.log 952265
adaptor_1cd9e4134b790becae04026c2fda49ff)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582186581.log 76459
adaptor_5a8f9af70b0e591b034f672f4340371c)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582370136.log 74779
adaptor_fa36c905bcb3701ae4183185ebb8af01)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527883616.log 380397
adaptor_a20e4ffdb7d3fad38732f13fc5aa54a5)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527918692.log 801761
adaptor_72174dc3ec7b8f3871c5d277a0f45125)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582273055.log 143061
adaptor_e24367b38aa7970ea475f67a850fd019)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
chukwaAgent 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaAgent.log 366838
adaptor_1ec2f74ed77d549837b7d5d33db64c7e)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
mapred 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309527999602.log 10253607
adaptor_b74c14fdd101db4a7c9bd7190c230428)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578883752.log 18568
adaptor_16b8b916c39b571a2d38c7ec5bda0a62)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578884932.log 31245
adaptor_4b887b4d345650292baa10290cd34693)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
mapred 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309582370140.log 2216270
adaptor_a2e4104972a498bee2619f6ee0b48efa)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
chukwaHttpSender 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaHttpSender.log
449552
adaptor_9532d27cd32d9c64757583ea28b37664)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-namenode-hadoop10-131-4-70.log
353185023
adaptor_12d05a003a847c3fc1b2c850decbc5bd)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578850821.log 46459
adaptor_d096625c34f55d7d9c5bc9950aa51f04)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309500469601.log 242968
adaptor_ea4c5487a8a14a91da47640303f58eb7)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309578850831.log 45630
adaptor_52e2b02867360ff8b60da5fc4acbcf6e)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582186584.log 190797
adaptor_17e122f11752d4f85bb3771962f4ed0b)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
chunkQueue 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chunkQueue.log 376527
adaptor_6be04642c8e5f3bf89bb12333baaa678)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527917408.log 382148
adaptor_136ad19fb21c0d398072457a231baba2)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582271839.log 75547
adaptor_b6ac14ec507719e8e01a5e9af8cc0cfc)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582633863.log 28576
adaptor_6dd77c53faf6f9224a4ffdf3cb77796b)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582186572.log 202433
adaptor_ffdb380f632a5efb0aa3532df247502f)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309500469604.log 607138
adaptor_e69999b07d7023e6ba08060c85bd9ad7)  org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor
Df 60 /bin/df -l 1437897063
adaptor_ddad8a9e25eb3e3502d7b4d399207ea8)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/audit.log
172088797
adaptor_469551943c09f2cda60cbe3965c35614)  org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309500469592.log 797132


you see, the adaptors are running correctly, so ,which mysql tables will save mapred data?
At 2011-07-02 13:04:33,"Eric Yang" <eric818@gmail.com> wrote:

>Chukwa 0.4 only works with hadoop 0.20.x.  If you are using hadoop
>0.2, then it probably doesn't work.
>
>On the nodes that you see the log files, try:
>
>telnet localhost 9093
>list
>
>Check if the jvm and mapred files are being streamed over chukwa agent.
>
>If it is not, you might need to restart the mapreduce cluster.
>
>regards,
>Eric
>
>2011/7/1 孙长 <smart02_ren@163.com>:
>> yes, there are rpc and mapred logs in  log file
>>
>> hadoop-metrics.properties is like this, and hadoop version is 0.2
>>
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> chukwaAgent.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> chukwaAgent.period=60
>> chukwaAgent.directory=/tmp/chukwa/log/metrics/
>>
>> chukwaHttpSender.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> chukwaHttpSender.period=60
>> chukwaHttpSender.directory=/tmp/chukwa/log/metrics/
>> chunkQueue.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> chunkQueue.period=60
>> chunkQueue.directory=/tmp/chukwa/log/metrics/
>> dfs.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> dfs.period=60
>> dfs.directory=/tmp/chukwa/log/metrics/
>> dfs.uuid=true
>> jvm.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> jvm.period=60
>> jvm.directory=/tmp/chukwa/log/metrics/
>> jvm.uuid=true
>> mapred.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> mapred.period=60
>> mapred.directory=/tmp/chukwa/log/metrics/
>> mapred.uuid=true
>> rpc.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> rpc.period=60
>> rpc.directory=/tmp/chukwa/log/metrics/
>> rpc.uuid=true
>>
>> At 2011-07-02 11:39:32,"Eric Yang" <eric818@gmail.com> wrote:
>>
>>>Check the log directory and see if you see any log file with rpc or
>>>mapred in the file name?
>>>What does the hadoop-metrics.properties look like?  What version of
>>>hadoop are you running?
>>>
>>>regards,
>>>Eric
>>>
>>>2011/7/1 孙长 <smart02_ren@163.com>:
>>>> i know where the problem is , i didn't put log4j and hadoop-metrics file
>>>> into all  nodes' hadoop conf dir.
>>>> Now the problem disapears
>>>> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm,
rpc
>>>> data, why?
>>>> thank you~
>>>>
>>>>
>>>> At 2011-07-01 22:59:41,"Eric Yang" <eric818@gmail.com> wrote:
>>>>
>>>>>Looks like mapreduce can not find json.jar in its class path.  Which
>>>>>version of Hadoop are you using?
>>>>>
>>>>>regards,
>>>>>Eric
>>>>>
>>>>>2011/7/1 孙长 <smart02_ren@163.com>:
>>>>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>>>>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by
demux runs
>>>>>> error:
>>>>>> task error is as follows:
>>>>>>
>>>>>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>>>>> 	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>>>>> 	at java.lang.Class.forName0(Native Method)
>>>>>> 	at java.lang.Class.forName(Class.java:169)
>>>>>> 	at
>>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>>>>>> 	at
>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>>>>>> 	at
>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>>>>>> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>>>> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>>>
>>>>>> At 2011-07-01 13:17:42,"Eric Yang" <eric818@gmail.com> wrote:
>>>>>>
>>>>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>>>>>HADOOP_HOME/lib, and restart hadoop?
>>>>>>>
>>>>>>>regards,
>>>>>>>Eric
>>>>>>>
>>>>>>>2011/6/30 孙长 <smart02_ren@163.com>:
>>>>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a
agent on two
>>>>>>>> nodes, but i only got system metrics in mysql, no hadoop
metrics like jvm,
>>>>>>>> dfs, mapred
>>>>>>>> dbAdmin.sh is running background, log4j.properties
>>>>>>>> and hadoop-metrics.properties in hadoop conf dir are set
according to user
>>>>>>>> guide
>>>>>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>>>>>> thank you ~
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>
Mime
View raw message