flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olivier brobecker (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2956) hive sink not sending heartbeat correctly
Date Thu, 14 Dec 2017 05:00:10 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290339#comment-16290339

Olivier brobecker commented on FLUME-2956:


No, I didn't have a valid solution: I did a workaround with an hdfs sink + spark batch in
order to analyse my data but our flume version is quite old.

Maybe the newer one do have a fix.

-----Message d'origine-----
De : Abhishek Choudhary (JIRA) [mailto:jira@apache.org] 
Envoyé : jeudi 14 décembre 2017 07:08
Objet : [jira] [Commented] (FLUME-2956) hive sink not sending heartbeat correctly

    [ https://issues.apache.org/jira/browse/FLUME-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290271#comment-16290271

Abhishek Choudhary commented on FLUME-2956:

I am also facing the same issue. did you find a way to work for the same ?

This message was sent by Atlassian JIRA


Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees
et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par
erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant
susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may
be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message
and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed
or falsified.
Thank you.

> hive sink not sending heartbeat correctly
> -----------------------------------------
>                 Key: FLUME-2956
>                 URL: https://issues.apache.org/jira/browse/FLUME-2956
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.5.2
>         Environment: linux CentOS 6.6
>            Reporter: Olivier brobecker
> Flume has been configured in order to populate a hive table as follow :
> agentSCDR.sinks.hive1.type = hive
> agentSCDR.sinks.hive1.channel = channel1
> agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
> agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
> agentSCDR.sinks.hive1.hive.database = myDatabase
> agentSCDR.sinks.hive1.hive.table = my_table
> agentSCDR.sinks.hive1.hive.partition = %Y
> agentSCDR.sinks.hive1.heartBeatInterval = 60
> agentSCDR.sinks.hive1.serializer = DELIMITED
> agentSCDR.sinks.hive1.serializer.delimiter = ;
> agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
> agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...
> My dataflow is irregular at best and I can have more than 30 minutes of inactivity, so
I have a heartbeatInterval at 60s in order to keep my txn alive.
> The issue is that this heartBeat is only sent when flume is trying to inject some datas
into hive instead of having one every 60s.
> # grep -i heartbeat flume-agentSCDR.log
> 15 juil. 2016 13:40:43,008 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:238)
 - Sending heartbeat on batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:12:21,001 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231)
 - Sending heartbeat on batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:27:56,963 INFO  [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231)
 - Sending heartbeat on batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
> ...

This message was sent by Atlassian JIRA

View raw message