beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davor Bonaci (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-53) PubSubIO: reimplement in Java
Date Tue, 22 Mar 2016 15:14:25 GMT

    [ https://issues.apache.org/jira/browse/BEAM-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206522#comment-15206522
] 

Davor Bonaci commented on BEAM-53:
----------------------------------

Another user question: http://stackoverflow.com/questions/36149102/how-would-you-retrieve-message-and-optional-attributes-from-pubsubio/36158225#36158225

> PubSubIO: reimplement in Java
> -----------------------------
>
>                 Key: BEAM-53
>                 URL: https://issues.apache.org/jira/browse/BEAM-53
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-core
>            Reporter: Daniel Halperin
>            Assignee: Mark Shields
>
> PubSubIO is currently only partially implemented in Java: the DirectPipelineRunner uses
a non-scalable API in a single-threaded manner.
> In contrast, the DataflowPipelineRunner uses an entirely different code path implemented
in the Google Cloud Dataflow service.
> We need to reimplement PubSubIO in Java in order to support other runners in a scalable
way.
> Additionally, we can take this opportunity to add new features:
> * getting timestamp from an arbitrary lambda in arbitrary formats rather than from a
message attribute in only 2 formats.
> * exposing metadata and attributes in the elements produced by PubSubIO.Read
> * setting metadata and attributes in the messages written by PubSubIO.Write



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message