flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Issue with HBase Sink in Flume ( 1.3.0)
Date Mon, 17 Feb 2014 15:05:12 GMT
Logs ?

On Mon, Feb 17, 2014 at 5:51 AM, Kris Ogirri <kanirip@gmail.com> wrote:
> Dear Mailing Group,
>
> I am currently having issues with the Hbase sink function. I have developed
> an agent with a fanout channel setup ( single source, multiple channels,
> multiple sinks) sinking to a HDFS cluster and Hbase deployment.
>
>  The issue is that although the HDFS is working well, the Hbase flow is
> simply not working. There are no errors being reported by Flume for the
> Hbase channel but there are never any records being written to the HBase
> store. The Hbase table as stipulated in the config always remains empty.
> Studying the Flume startup logs I observe that the session connection to
> Zookeeper is always successfully established
>
> Are there any special configurations I am missing out?
>
> I am using the Async Event Serializer to persist the txns.
>
> Any assistance will be greatly appreciated.
>
>
> Please see below for the flume configuration:
>
> [biadmin@bivm bin]$ cat flume-conf.properties.bigdemo
> agent.sources=exec-source
> agent.sinks=hdfs-sink hbase-sink
> agent.channels=ch1 ch2
>
> agent.sources.exec-source.type=exec
> agent.sources.exec-source.command=tail -F
> /home/biadmin/bigdemo/data/rec_telco.cdr
>
> agent.sinks.hdfs-sink.type=hdfs
> agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/
> agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec
> # File size to trigger roll, in bytes (0: never roll based on file size)
> agent.sinks.hdfs-sink.hdfs.rollSize = 134217728
> agent.sinks.hdfs-sink.hdfs.rollCount = 0
> # number of events written to file before it flushed to HDFS
> agent.sinks.hdfs-sink.hdfs.batchSize = 10000
> agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000
>
>
> agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink
> agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
> agent.sinks.hbase-sink.table=telco_cdr_rec
> agent.sinks.hbase-sink.columnFamily = colfam
> agent.sinks.hbase-sink.channels = ch2
> #agent.sinks.hbase-sink.hdfs.batchSize = 10000
> #agent.sinks.hbase-sink.hdfs.txnEventMax = 40000
>
>
> agent.channels.ch1.type=file
> agent.channels.ch1.checkpointInterval=3000
> agent.channels.ch1.transactionCapacity=10000
> agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint
> agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data
> agent.channels.ch1.write-timeout=30
> agent.channels.ch1.keep-alive=30
> #agent.channels.ch1.capacity=1000
>
> agent.channels.ch2.type=file
> agent.channels.ch2.checkpointInterval=300
> agent.channels.ch2.transactionCapacity=10000
> agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint
> agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data
> agent.channels.ch2.write-timeout=30
> agent.channels.ch2.keep-alive=30
> #agent.channels.ch2.capacity=1000
>
>
> agent.sources.exec-source.channels=ch1 ch2
> agent.sinks.hdfs-sink.channel=ch1
> agent.sinks.hbase-sink.channel=ch2
>

Mime
View raw message