flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2956) hive sink not sending heartbeat correctly
Date Thu, 14 Dec 2017 05:17:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290349#comment-16290349
] 

He Jiang commented on FLUME-2956:
---------------------------------

It seems there's an bug in flume-hive-sink after hours code review. I did some change and
it works. I could send you the modified flume-hive-sink-1.8.0.jar if you want to have a try.

> hive sink not sending heartbeat correctly
> -----------------------------------------
>
>                 Key: FLUME-2956
>                 URL: https://issues.apache.org/jira/browse/FLUME-2956
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.5.2
>         Environment: linux CentOS 6.6
>            Reporter: Olivier brobecker
>
> Flume has been configured in order to populate a hive table as follow :
> agentSCDR.sinks.hive1.type = hive
> agentSCDR.sinks.hive1.channel = channel1
> agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
> agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
> agentSCDR.sinks.hive1.hive.database = myDatabase
> agentSCDR.sinks.hive1.hive.table = my_table
> agentSCDR.sinks.hive1.hive.partition = %Y
> agentSCDR.sinks.hive1.heartBeatInterval = 60
> agentSCDR.sinks.hive1.serializer = DELIMITED
> agentSCDR.sinks.hive1.serializer.delimiter = ;
> agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
> agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...
> My dataflow is irregular at best and I can have more than 30 minutes of inactivity, so
I have a heartbeatInterval at 60s in order to keep my txn alive.
> The issue is that this heartBeat is only sent when flume is trying to inject some datas
into hive instead of having one every 60s.
> # grep -i heartbeat flume-agentSCDR.log
> 15 juil. 2016 13:40:43,008 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:238)
 - Sending heartbeat on batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:12:21,001 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231)
 - Sending heartbeat on batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:27:56,963 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231)
 - Sending heartbeat on batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
> ...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message