beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-849) Redesign PipelineResult API
Date Wed, 01 Mar 2017 19:34:45 GMT

    [ https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890885#comment-15890885
] 

Amit Sela commented on BEAM-849:
--------------------------------

I disagree with stating that "Create.of(filename) + ParDo(tail file) + ParDo(process records)"
in streaming leverages "low-latency", that's very runner specific, and generally inaccurate
- how is sending a "filename" to worker/s + data locality faster than streaming the file to
processors ? 

> Redesign PipelineResult API
> ---------------------------
>
>                 Key: BEAM-849
>                 URL: https://issues.apache.org/jira/browse/BEAM-849
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Pei He
>
> Current state: 
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses waitUntilFinish() and cancel().

> However, there are additional work around PipelineResult: 
> need clearly defined contract and verification across all runners 
> need to revisit how to handle metrics/aggregators 
> need to be able to get logs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message