incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Preetam Patil <pvpatil.i...@gmail.com>
Subject Re: A demo setup on a single linux server
Date Wed, 01 Jun 2011 17:32:09 GMT
Don't know if it matters--I'm using CDH3 on Ubuntu 10.04. My
collector.log attached now.
Thanks,
-preetam

On Wed, Jun 1, 2011 at 10:33 PM, Preetam Patil <pvpatil.iitb@gmail.com> wrote:
> Hi Eric and DKN,
>
> I am facing a similar (maybe a noob) problem: chukwa collector is able
> to write to a sequence file, but when I enabl eHBaseWriter, the
> HBaseTable entries are not added. I am not sure whether it's a problem
> with my HBase setup (it's in standalone mode) or Chukwa Demux parsers.
> I get two errors: "ERROR main HBaseWriter - Hbase schema mismatch with
> demux parser" and "INFO main HConnectionManager$TableServers -
> getMaster at tempt 6 of 10 failed; retrying after sleep of 4000".
> Can't figure out where the problem lies.
>
> Attaching the collector.log to this mail and below is my
> chukwa-collector-conf.xml:
>
> Thanks,
> -preetam
>
> <?xml version="1.0"?>
> <!--
>   Licensed to the Apache Software Foundation (ASF) under one or more
>   contributor license agreements.  See the NOTICE file distributed with
>   this work for additional information regarding copyright ownership.
>   The ASF licenses this file to You under the Apache License, Version 2.0
>   (the "License"); you may not use this file except in compliance with
>   the License.  You may obtain a copy of the License at
>
>       http://www.apache.org/licenses/LICENSE-2.0
>
>   Unless required by applicable law or agreed to in writing, software
>   distributed under the License is distributed on an "AS IS" BASIS,
>   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>   See the License for the specific language governing permissions and
>   limitations under the License.
> -->
> <?xml-stylesheet type="text/xsl"  href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>
>  <property>
>    <name>chukwaCollector.writerClass</name>
>    <value>org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter</value>
>  </property>
>
> <!-- peetam: replaced by HBaseWriter pipeline below
>  <property>
>    <name>chukwaCollector.pipeline</name>
>    <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter,org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter</value>
>  </property>
> -->
>
> <!-- LocalWriter parameters
>  <property>
>    <name>chukwaCollector.localOutputDir</name>
>    <value>/tmp/chukwa/dataSink/</value>
>    <description>Chukwa local data sink directory, see
> LocalWriter.java</description>
>  </property>
>
>  <property>
>    <name>chukwaCollector.writerClass</name>
>    <value>org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalWriter</value>
>    <description>Local chukwa writer, see LocalWriter.java</description>
>  </property>
> -->
>
> <!-- When writing to HBase, uncomment the following parameters. If
> you're running
> HBase in distributed mode, you'll also need to copy your
> hbase-site.xml file with
> your hbase.zookeeper.quorum setting to the conf/ dir. -->
>
>  <property>
>    <name>chukwaCollector.pipeline</name>
>    <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter,org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter</value>
>  </property>
>
>  <property>
>    <name>hbase.demux.package</name>
>    <value>org.apache.hadoop.chukwa.extraction.demux.processor</value>
>    <description>Demux parser class package, HBaseWriter uses this
> package name to validate HBase for annotated demux parser
> classes.</description>
>  </property>
>
>  <property>
>    <name>hbase.writer.verify.schema</name>
>    <value>true</value>
>    <description>Verify HBase Table schema with demux parser schema, log
>    warning if there are mismatch between hbase schema and demux parsers.
>    </description>
>  </property>
>
>  <property>
>    <name>hbase.writer.halt.on.schema.mismatch</name>
>    <value>false</value>
>    <description>If this option is set to true, and HBase table schema
>    is mismatched with demux parser, collector will shut down itself.
>    </description>
>  </property>
>
>  <property>
>    <name>writer.hdfs.filesystem</name>
>    <value>hdfs://localhost:8020/</value>
>    <description>HDFS to dump to</description>
>  </property>
>
>  <property>
>    <name>chukwaCollector.outputDir</name>
>    <value>/chukwa/logs/</value>
>    <description>Chukwa data sink directory</description>
>  </property>
>
>  <property>
>    <name>chukwaCollector.rotateInterval</name>
>    <value>300000</value>
>    <description>Chukwa rotate interval (ms)</description>
>  </property>
>
>  <property>
>    <name>chukwaCollector.http.port</name>
>    <value>8080</value>
>    <description>The HTTP port number the collector will listen on</description>
>  </property>
> </configuration>
>
>
> On Wed, Jun 1, 2011 at 10:11 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
>> From the current configuration, it is setup to write to Hbase only, it does
>> not write to HDFS.  If you want the data to be written to HDFS as well, use
>> this line in chukwa-collector-conf.xml.
>>
>>   <property>
>>     <name>chukwaCollector.pipeline</name>
>>     <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter,
>> org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter,
>> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter</value>
>>   </property>
>>
>> Make sure that hadoop-site.xml and hbase-site.xml are in the chukwa
>> collector class path.  This can be done by copying hadoop-site.xml and
>> hbase-site.xml to CHUKWA_CONF_DIR.
>>
>> For HICC listening to localhost issue, this is a bug in Chukwa.  Feel free
>> to file a jira.  Before compile, modify web/hicc/WEB-INF/jetty.xml and set
>> jetty.host ip address to 0.0.0.0.  Without this setup, jetty would choose to
>> bind to first available IP address, it could randomly bind to localhost or
>> eth0 depending on the OS.
>>
>> Regards,
>> Eric
>>
>

Mime
View raw message