beam-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <>
Subject [jira] [Work logged] (BEAM-12118) QueuingBeamFnDataClient adds polling latency to completing bundle processing
Date Mon, 12 Apr 2021 20:06:00 GMT


ASF GitHub Bot logged work on BEAM-12118:

                Author: ASF GitHub Bot
            Created on: 12/Apr/21 20:05
            Start Date: 12/Apr/21 20:05
    Worklog Time Spent: 10m 
      Work Description: amaliujia closed pull request #14480:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 581312)
    Time Spent: 4h  (was: 3h 50m)

> QueuingBeamFnDataClient adds polling latency to completing bundle processing
> ----------------------------------------------------------------------------
>                 Key: BEAM-12118
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: java-fn-execution
>            Reporter: Sam Whittle
>            Assignee: Sam Whittle
>            Priority: P2
>          Time Spent: 4h
>  Remaining Estimate: 0h
> Currently the inboundDataClients are registered with recieve, and they add data to a
queue. There is no explicit indication from the clients that they are no longer going to add
values to the queue.
> Within QueueingBeamFnDataClient.drainAndBlock the queue is therefore polled and if nothing
is present all clients are polled to see if they are complete.
> This design makes for unfortunate tradeoffs on poll timeout:
> - cpu wasted with small timeout
> - additional latency in noticing we have completed with larger timeout
> With the existing InboundDataClient interface, we could have a separate thread call awaitCompletion
on all of the clients and then shutdown the queue (adding a poison pill perhaps)
> Or we could modify InboundDataClient interface to allow registering iterest in when the
client is done producing elements.  The existing clients all seem based upon futures which
allow that.

This message was sent by Atlassian Jira

View raw message