flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5487) Proper at-least-once support for ElasticsearchSink
Date Fri, 13 Jan 2017 21:16:26 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822356#comment-15822356

Tzu-Li (Gordon) Tai commented on FLINK-5487:

Restructuring for the ES connectors is included as part of this issue. We should block this
issue until FLINK-4988 is resolved so at-least-once support can be simultaneously included
in all ES versions.

> Proper at-least-once support for ElasticsearchSink
> --------------------------------------------------
>                 Key: FLINK-5487
>                 URL: https://issues.apache.org/jira/browse/FLINK-5487
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>            Reporter: Tzu-Li (Gordon) Tai
>            Priority: Critical
> Discussion in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
> Currently, the Elasticsearch Sink actually doesn't offer any guarantees for message delivery.
> For proper support of at-least-once, the sink will need to participate in Flink's checkpointing:
when snapshotting is triggered at the {{ElasticsearchSink}}, we need to synchronize on the
pending ES requests by flushing the internal bulk processor. For temporary ES failures (see
FLINK-5122) that may happen on the flush, we should retry them before returning from snapshotting
and acking the checkpoint. If there are non-temporary ES failures on the flush, the current
snapshot should fail.

This message was sent by Atlassian JIRA

View raw message