incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: A demo setup on a single linux server
Date Wed, 01 Jun 2011 18:10:29 GMT
Hi Preetam,

Have you run hbase shell < hbase.schema ?  The problem description has two possible outcomes.

1, Hbase schema has not been applied to hbase. Solution:

Hbase shell < /path/to/chukwa/conf/hbase.schema

2, hbase-site.xml and hadoop-site.xml are not in chukwa configuration directory or nor in
the chukwa collector class path.  Solution:

cp hadoop-site.xml hbase-site.xml /path/to/chukwa/conf

Hope this helps.

Regards,
Eric


On 6/1/11 10:03 AM, "Preetam Patil" <pvpatil.iitb@gmail.com> wrote:

Hi Eric and DKN,

I am facing a similar (maybe a noob) problem: chukwa collector is able
to write to a sequence file, but when I enabl eHBaseWriter, the
HBaseTable entries are not added. I am not sure whether it's a problem
with my HBase setup (it's in standalone mode) or Chukwa Demux parsers.
I get two errors: "ERROR main HBaseWriter - Hbase schema mismatch with
demux parser" and "INFO main HConnectionManager$TableServers -
getMaster at tempt 6 of 10 failed; retrying after sleep of 4000".
Can't figure out where the problem lies.

Attaching the collector.log to this mail and below is my
chukwa-collector-conf.xml:

Thanks,
-preetam

<?xml version="1.0"?>
<!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<?xml-stylesheet type="text/xsl"  href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
    <name>chukwaCollector.writerClass</name>
    <value>org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter</value>
  </property>

<!-- peetam: replaced by HBaseWriter pipeline below
  <property>
    <name>chukwaCollector.pipeline</name>
    <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter,org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter</value>
  </property>
-->

<!-- LocalWriter parameters
  <property>
    <name>chukwaCollector.localOutputDir</name>
    <value>/tmp/chukwa/dataSink/</value>
    <description>Chukwa local data sink directory, see
LocalWriter.java</description>
  </property>

  <property>
    <name>chukwaCollector.writerClass</name>
    <value>org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalWriter</value>
    <description>Local chukwa writer, see LocalWriter.java</description>
  </property>
-->

<!-- When writing to HBase, uncomment the following parameters. If
you're running
HBase in distributed mode, you'll also need to copy your
hbase-site.xml file with
your hbase.zookeeper.quorum setting to the conf/ dir. -->

  <property>
    <name>chukwaCollector.pipeline</name>
    <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter,org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter</value>
  </property>

  <property>
    <name>hbase.demux.package</name>
    <value>org.apache.hadoop.chukwa.extraction.demux.processor</value>
    <description>Demux parser class package, HBaseWriter uses this
package name to validate HBase for annotated demux parser
classes.</description>
  </property>

  <property>
    <name>hbase.writer.verify.schema</name>
    <value>true</value>
    <description>Verify HBase Table schema with demux parser schema, log
    warning if there are mismatch between hbase schema and demux parsers.
    </description>
  </property>

  <property>
    <name>hbase.writer.halt.on.schema.mismatch</name>
    <value>false</value>
    <description>If this option is set to true, and HBase table schema
    is mismatched with demux parser, collector will shut down itself.
    </description>
  </property>

  <property>
    <name>writer.hdfs.filesystem</name>
    <value>hdfs://localhost:8020/</value>
    <description>HDFS to dump to</description>
  </property>

  <property>
    <name>chukwaCollector.outputDir</name>
    <value>/chukwa/logs/</value>
    <description>Chukwa data sink directory</description>
  </property>

  <property>
    <name>chukwaCollector.rotateInterval</name>
    <value>300000</value>
    <description>Chukwa rotate interval (ms)</description>
  </property>

  <property>
    <name>chukwaCollector.http.port</name>
    <value>8080</value>
    <description>The HTTP port number the collector will listen on</description>
  </property>
</configuration>


On Wed, Jun 1, 2011 at 10:11 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
> From the current configuration, it is setup to write to Hbase only, it does
> not write to HDFS.  If you want the data to be written to HDFS as well, use
> this line in chukwa-collector-conf.xml.
>
>   <property>
>     <name>chukwaCollector.pipeline</name>
>     <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter,
> org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter,
> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter</value>
>   </property>
>
> Make sure that hadoop-site.xml and hbase-site.xml are in the chukwa
> collector class path.  This can be done by copying hadoop-site.xml and
> hbase-site.xml to CHUKWA_CONF_DIR.
>
> For HICC listening to localhost issue, this is a bug in Chukwa.  Feel free
> to file a jira.  Before compile, modify web/hicc/WEB-INF/jetty.xml and set
> jetty.host ip address to 0.0.0.0.  Without this setup, jetty would choose to
> bind to first available IP address, it could randomly bind to localhost or
> eth0 depending on the OS.
>
> Regards,
> Eric
>


Mime
View raw message