flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kris Ogirri <kani...@gmail.com>
Subject Re: Issue with HBase Sink in Flume ( 1.3.0)
Date Mon, 17 Feb 2014 15:38:01 GMT
Hello Jeff,

Please find below requested logs.. Initiation part of the logs were
unfortunately not included. I can run these again if necessary but the
Zookeeper connection is included in the logs.



On 17 February 2014 16:05, Jeff Lord <jlord@cloudera.com> wrote:

> Logs ?
>
> On Mon, Feb 17, 2014 at 5:51 AM, Kris Ogirri <kanirip@gmail.com> wrote:
> > Dear Mailing Group,
> >
> > I am currently having issues with the Hbase sink function. I have
> developed
> > an agent with a fanout channel setup ( single source, multiple channels,
> > multiple sinks) sinking to a HDFS cluster and Hbase deployment.
> >
> >  The issue is that although the HDFS is working well, the Hbase flow is
> > simply not working. There are no errors being reported by Flume for the
> > Hbase channel but there are never any records being written to the HBase
> > store. The Hbase table as stipulated in the config always remains empty.
> > Studying the Flume startup logs I observe that the session connection to
> > Zookeeper is always successfully established
> >
> > Are there any special configurations I am missing out?
> >
> > I am using the Async Event Serializer to persist the txns.
> >
> > Any assistance will be greatly appreciated.
> >
> >
> > Please see below for the flume configuration:
> >
> > [biadmin@bivm bin]$ cat flume-conf.properties.bigdemo
> > agent.sources=exec-source
> > agent.sinks=hdfs-sink hbase-sink
> > agent.channels=ch1 ch2
> >
> > agent.sources.exec-source.type=exec
> > agent.sources.exec-source.command=tail -F
> > /home/biadmin/bigdemo/data/rec_telco.cdr
> >
> > agent.sinks.hdfs-sink.type=hdfs
> > agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/
> > agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec
> > # File size to trigger roll, in bytes (0: never roll based on file size)
> > agent.sinks.hdfs-sink.hdfs.rollSize = 134217728
> > agent.sinks.hdfs-sink.hdfs.rollCount = 0
> > # number of events written to file before it flushed to HDFS
> > agent.sinks.hdfs-sink.hdfs.batchSize = 10000
> > agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000
> >
> >
> > agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink
> >
> agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
> > agent.sinks.hbase-sink.table=telco_cdr_rec
> > agent.sinks.hbase-sink.columnFamily = colfam
> > agent.sinks.hbase-sink.channels = ch2
> > #agent.sinks.hbase-sink.hdfs.batchSize = 10000
> > #agent.sinks.hbase-sink.hdfs.txnEventMax = 40000
> >
> >
> > agent.channels.ch1.type=file
> > agent.channels.ch1.checkpointInterval=3000
> > agent.channels.ch1.transactionCapacity=10000
> >
> agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint
> > agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data
> > agent.channels.ch1.write-timeout=30
> > agent.channels.ch1.keep-alive=30
> > #agent.channels.ch1.capacity=1000
> >
> > agent.channels.ch2.type=file
> > agent.channels.ch2.checkpointInterval=300
> > agent.channels.ch2.transactionCapacity=10000
> >
> agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint
> > agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data
> > agent.channels.ch2.write-timeout=30
> > agent.channels.ch2.keep-alive=30
> > #agent.channels.ch2.capacity=1000
> >
> >
> > agent.sources.exec-source.channels=ch1 ch2
> > agent.sinks.hdfs-sink.channel=ch1
> > agent.sinks.hbase-sink.channel=ch2
> >
>

Mime
View raw message