flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Teena Kappen // BPRISE <teena.kap...@bprise.com>
Subject RE: Multiple Elasticsearch sinks not working in Flink
Date Tue, 23 Jan 2018 12:32:18 GMT
Thanks Fabian. I will go through it and add info if required.

From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: 23 January 2018 15:20
To: Teena Kappen // BPRISE <teena.kappen@bprise.com>
Cc: Timo Walther <twalthr@apache.org>; user@flink.apache.org
Subject: Re: Multiple Elasticsearch sinks not working in Flink

Hi Teena,
I created FLINK-8489 [1] to track the issue.
Please have a look and add information that might be relevant.

Best, Fabian

[1] https://issues.apache.org/jira/browse/FLINK-8489<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0>

2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <teena.kappen@bprise.com<mailto:teena.kappen@bprise.com>>:
Hi Timo,

It works fine when the second sink is a Cassandra Sink. The data gets read from KafkaTopic2
and it gets written to Cassandra as expected.


From: Timo Walther [mailto:twalthr@apache.org<mailto:twalthr@apache.org>]
Sent: 18 January 2018 18:41
To: user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: Multiple Elasticsearch sinks not working in Flink

Hi Teena,

what happens if you replace the second sink with a non-ElasticSearchSink? Is there the same
result? Is the data read from the KafkaTopic2?

We should determine which system is the bottleneck.


Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:

I am running flink 1.4 in single node. My job has two Kafka consumers reading from separate
topics. After fetching the data, the job writes it to two separate Elasticsearch sinks. So
the process is like this

KafkaTopic1 -> Kafkaconsumer1 -> create output record -> Elasticsearchsink1
KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2

Both the streams and their processing are completely unrelated. The first sink works as expected
and it writes the output for all input records. The second sink writes to Elasticsearch only
once and after that it stops writing to Elasticsearch even if there is more data that gets
fed into Kafka. Sometimes, it does not even write once. We tested this in two other jobs and
the same issue is there in all of them.

I have attached a sample code I had created to illustrate the issue. We are using Elasticsearch
version 5.6.4 and hence the dependency used is ‘flink-connector-elasticsearch5_2.11’.


View raw message