beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <>
Subject [jira] [Resolved] (BEAM-1146) Decrease spark runner startup overhead
Date Mon, 02 Jan 2017 14:27:58 GMT


Amit Sela resolved BEAM-1146.
       Resolution: Fixed
    Fix Version/s: 0.5.0

> Decrease spark runner startup overhead
> --------------------------------------
>                 Key: BEAM-1146
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>            Reporter: Aviem Zur
>            Assignee: Aviem Zur
>             Fix For: 0.5.0
> BEAM-921 introduced a lazy singleton instantiated once in each machine (driver &
executors) which utilizes reflection to find all subclasses of Source and Coder
> While this is beneficial in it's own right, the change added about one minute of overhead
in spark runner startup time (which cause the first job/stage to take up to a minute).
> The change is in class {{BeamSparkRunnerRegistrator}}
> The reason reflection (specifically reflections library) was used here is because  there
is no current way of knowing all the source and coder classes at runtime.

This message was sent by Atlassian JIRA

View raw message