beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviem Zur (JIRA)" <>
Subject [jira] [Resolved] (BEAM-1294) Long running UnboundedSource Readers
Date Sun, 09 Apr 2017 19:52:41 GMT


Aviem Zur resolved BEAM-1294.
       Resolution: Implemented
    Fix Version/s: First stable release

> Long running UnboundedSource Readers
> ------------------------------------
>                 Key: BEAM-1294
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Aviem Zur
>             Fix For: First stable release
> When reading from an UnboundedSource, current implementation will cause each split to
create a new Reader every micro-batch.
> As long as the overhead of creating a reader is relatively low, it's reasonable (though
I'd still be happy to get rid of), but in cases where the creation overhead is large it becomes
unreasonable forcing large batches.
> One way to solve this could be to create a pool of lazy-init readers to serve each executor,
maybe via Broadcast variables. 

This message was sent by Atlassian JIRA

View raw message