flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Baranau <alex.barano...@gmail.com>
Subject Re: HBase and Flume
Date Mon, 06 Feb 2012 18:15:50 GMT
Hi,

Hm. Looks like there are too many open connections to the ZK from that
host. How many flume nodes which write to Hbase did you start?

Also, make sure your ZK allows proper number of connections from one host
(I remember there was some issue with having default less than expected(30)
in one of ZK versions). You can add this to zoo.cfg (and restart ZK)
manually  to be sure about how many is allowed:
maxClientCnxns=30

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase

On Thu, Feb 2, 2012 at 2:59 PM, Gaurav Khanna <khannapost@yahoo.com> wrote:

>
> Got the jar for the hbase plugin and updated the flume-site.xml with the
> plugin configuration. Now I can see the plugin and use it as well. But here
> is the problem:
>
> This is on a client box that is not part of the hbase or hadoop cluster
> and am getting the following exception in the flume node. These are the env
> variables:
> # HADOOP RELATED STUFF
> export HADOOP_CONF_DIR=/app/hadoop/hadoop-conf
> export HBASE_CONF_DIR=/app/hadoop/hbase-conf
> export JAVA_HOME=/app/downloads/cloudera/jdk1.6.0_25
> export PATH=/app/downloads/cloudera/jdk1.6.0_25/bin:$PATH
> export ZOOKEEPER_CONF=$HBASE_CONF_DIR
> export
> HIVE_AUX_JARS_PATH=/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar
>
> And this is the exception that I get:
>
> 012-02-02 13:52:08,721 [logicalNode
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2012-02-02 13:52:08,722 [logicalNode
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] INFO
> connector.DirectDriver: Connector logicalNode
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22 exited with error: HBase is
> able to connect to ZooKeeper but the connection closes immediately. This
> could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and then
> make sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:160)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:175)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:153)
>         at com.cloudera.flume.hbase.HBaseSink.open(HBaseSink.java:144)
>         at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
>   ------------------------------
> *From:* Gaurav Khanna <khannapost@yahoo.com>
> *To:* "flume-user@incubator.apache.org" <flume-user@incubator.apache.org>;
> Gaurav Khanna <khannapost@yahoo.com>
> *Sent:* Thursday, February 2, 2012 10:57 AM
> *Subject:* Re: HBase and Flume
>
> Got the information at:
>
> https://github.com/cloudera/flume/tree/master/plugins/flume-plugin-hbasesink
>
> Thanks
> Gaurav
>
>   ------------------------------
> *From:* Gaurav Khanna <khannapost@yahoo.com>
> *To:* "flume-user@incubator.apache.org" <flume-user@incubator.apache.org>
> *Sent:* Thursday, February 2, 2012 10:54 AM
> *Subject:* HBase and Flume
>
> Hi,
> Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink.
>
> Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1",
> "%{body}")
>
> and got the following in the master logs (now I know that I am missing
> some configuration steps - please can anybody point me to where I can get
> the Hbase related plugin and how to configure that?):
>
> 2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager:
> Executing command: config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com,
> tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}",
> "colfam1", "q1", "%{body}")]
> 2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager:
> Unexpected exception during cmdid:0 config [
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com,
> tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}",
> "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source:
> Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
> java.lang.IllegalArgumentException: Attempted to write an invalid
> sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1",
> "%{body}" )
>         at
> com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
>         at
> com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
>         at
> com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
>         at
> com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
>         at
> com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
>         at
> com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
>         at
> com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
> Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase(
> "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
>         at
> com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
>         at
> com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
>         at
> com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
>         at
> com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)
>
>
>
>
>
>

Mime
View raw message