beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <>
Subject [jira] [Created] (BEAM-223) KafkaIO: don't use SerializableCoder
Date Mon, 25 Apr 2016 15:01:12 GMT
Daniel Halperin created BEAM-223:

             Summary: KafkaIO: don't use SerializableCoder
                 Key: BEAM-223
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
            Reporter: Daniel Halperin
            Assignee: Raghu Angadi

Reuven says:

I noticed that we're using SerializableCoder for the checkpoint mark in KafkaIO. This is generally
highly discouraged in streaming pipelines. Partially because it's inefficient, but more importantly
because Java serialization is not guaranteed to be stable. If a user updates their pipeline,
the new pipeline may not be able to decode the existing checkpoint marks; this will either
cause exceptions to be thrown, or data loss.

This message was sent by Atlassian JIRA

View raw message