flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Khanna <khannap...@yahoo.com>
Subject Re: HBase and Flume
Date Thu, 02 Feb 2012 19:59:12 GMT


Got the jar for the hbase plugin and updated the flume-site.xml with the plugin configuration.
Now I can see the plugin and use it as well. But here is the problem:

This is on a client box that is not part of the hbase or hadoop cluster and am getting the
following exception in the flume node. These are the env variables:
# HADOOP RELATED STUFF 
export HADOOP_CONF_DIR=/app/hadoop/hadoop-conf
export HBASE_CONF_DIR=/app/hadoop/hbase-conf
export JAVA_HOME=/app/downloads/cloudera/jdk1.6.0_25
export PATH=/app/downloads/cloudera/jdk1.6.0_25/bin:$PATH
export ZOOKEEPER_CONF=$HBASE_CONF_DIR
export HIVE_AUX_JARS_PATH=/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar


And this is the exception that I get:

012-02-02 13:52:08,721 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2012-02-02 13:52:08,722 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] INFO
connector.DirectDriver: Connector logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22
exited with error: HBase is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration
as often as you can. See HTable's javadoc for more information.
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper
but the connection closes immediately. This could be a sign that the server has too many connections
(30 is the default). Consider inspecting your ZK server logs for that error and then make
sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more
information.
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:160)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:175)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:153)
        at com.cloudera.flume.hbase.HBaseSink.open(HBaseSink.java:144)
        at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /hbase
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)



________________________________
 From: Gaurav Khanna <khannapost@yahoo.com>
To: "flume-user@incubator.apache.org" <flume-user@incubator.apache.org>; Gaurav Khanna
<khannapost@yahoo.com> 
Sent: Thursday, February 2, 2012 10:57 AM
Subject: Re: HBase and Flume
 

Got the information at:
https://github.com/cloudera/flume/tree/master/plugins/flume-plugin-hbasesink

Thanks
Gaurav


________________________________
 From: Gaurav Khanna <khannapost@yahoo.com>
To: "flume-user@incubator.apache.org" <flume-user@incubator.apache.org> 
Sent: Thursday, February 2, 2012 10:54 AM
Subject: HBase and Flume
 

Hi,
Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink. 

Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")

and got the following in the master logs (now I know that I am missing some configuration
steps - please can anybody point me to where I can get the Hbase related plugin and how to
configure that?):

2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager: Executing command: config
[POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable",
"%{nanos}", "colfam1", "q1", "%{body}")]
2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager: Unexpected exception during
cmdid:0 config
 [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable",
"%{nanos}", "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source: Invalid
sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
java.lang.IllegalArgumentException: Attempted to write an invalid sink/source: Invalid sink:
hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
        at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
        at com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
        at com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
        at
 com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
        at com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
        at com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
        at com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase( "testtable", "%{nanos}",
"colfam1", "q1", "%{body}" )
        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
        at
 com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
        at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)
Mime
View raw message