flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olivier brobecker (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLUME-2956) hive sink not sending heartbeat correctly
Date Fri, 15 Jul 2016 11:42:20 GMT
Olivier brobecker created FLUME-2956:
----------------------------------------

             Summary: hive sink not sending heartbeat correctly
                 Key: FLUME-2956
                 URL: https://issues.apache.org/jira/browse/FLUME-2956
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: v1.5.2
         Environment: linux CentOS 6.6
            Reporter: Olivier brobecker


Flume has been configured in order to populate a hive table as follow :

agentSCDR.sinks.hive1.type = hive
agentSCDR.sinks.hive1.channel = channel1
agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
agentSCDR.sinks.hive1.hive.database = myDatabase
agentSCDR.sinks.hive1.hive.table = my_table
agentSCDR.sinks.hive1.hive.partition = %Y
agentSCDR.sinks.hive1.heartBeatInterval = 60
agentSCDR.sinks.hive1.serializer = DELIMITED
agentSCDR.sinks.hive1.serializer.delimiter = ;
agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...

My dataflow is irregular at best and I can have more than 30 minutes of inactivity, so I have
a heartbeatInterval at 60s in order to keep my txn alive.

The issue is that this heartBeat is only sent when flume is trying to inject some datas into
hive instead of having one every 60s.

# grep -i heartbeat flume-agentSCDR.log
15 juil. 2016 13:40:43,008 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:238)
 - Sending heartbeat on batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
15 juil. 2016 14:12:21,001 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231)
 - Sending heartbeat on batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
15 juil. 2016 14:27:56,963 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231)
 - Sending heartbeat on batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message