beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davor Bonaci (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-638) Add sink transform to write bounded data per window, pane, [and key] even when PCollection is unbounded
Date Sun, 07 May 2017 15:30:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999898#comment-15999898
] 

Davor Bonaci commented on BEAM-638:
-----------------------------------

[~Arqu], windowing specified at Window.into applies when performing GroupByKey operation.
Without a GroupByKey in the pipeline, Window.into does little to affect the outcome.

There were various discussions in the past that the system should either detect this pattern
and throw an error, or automatically insert a GroupByKey on your behalf. I don't remember
we ever moved on either of these. If you cannot find an existing JIRA on this, feel free to
file a new one to somehow address this scenario.

> Add sink transform to write bounded data per window, pane, [and key] even when PCollection
is unbounded
> -------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-638
>                 URL: https://issues.apache.org/jira/browse/BEAM-638
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Jean-Baptiste Onofré
>            Assignee: Davor Bonaci
>
> Today, if the pipeline source is unbounded, and the sink expects a bounded collection,
there's no way to use a single pipeline. Even a window creates a chunk on the unbounded PCollection,
but the "sub" PCollection is still unbounded.
> It would be helpful for users to have a Window function that create a bounded PCollection
(on the window) from an unbounded PCollection coming from the source.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message