spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jose Torres (JIRA)" <>
Subject [jira] [Commented] (SPARK-23703) Collapse sequential watermarks
Date Sat, 05 May 2018 18:11:00 GMT


Jose Torres commented on SPARK-23703:

Up to you. It might be worth asking if there are use cases for that kind of thing, but on
the other hand I don't know of other systems that support it.

> Collapse sequential watermarks 
> -------------------------------
>                 Key: SPARK-23703
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Jose Torres
>            Priority: Major
> When there are two sequential EventTimeWatermark nodes in a query plan, the topmost one
overrides the column tracking metadata from its children, but leaves the nodes themselves
untouched. When there is no intervening stateful operation to consume the watermark, we should
remove the lower node entirely.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message