beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilya Ganelin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-88) DataflowPipelineOptions.tempLocation doesn't really default to stagingLocation
Date Tue, 24 May 2016 20:51:12 GMT

    [ https://issues.apache.org/jira/browse/BEAM-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298882#comment-15298882
] 

Ilya Ganelin commented on BEAM-88:
----------------------------------

I've spent some more time digging into this issue and it may actually be rather tricky to
fix. Because the method definitions are created in an interface, we cannot define default
values or behaviors for the methods. Even Java 8 default methods won't solve this problem
because tempLocation is defined in the base PipelineOptions Interface and stagingLocation
is defined within DataflowPipelineOptions, thus while we could default from stagingLocation
to tempLocation, the reverse would not work.

Further, because these interfaces are never implemented explicitly but are instead executed
via a complex reflection mechanism based on defined configuration properties, we don't even
have the option of defining the correct default behavior in classes that implement this interface.


The only way I could think of to get the right behavior is to create a specific class that
implements the DataflowPipelineOptions and have all classes that use DataflowPipelineOptions
use that class instead. However, because of the use of multiple inheritance (via Interfaces)
used throughout the project, that seems like a non-starter. 

I would love any insight as to how we could resolve this from those more familiar with the
project or perhaps from someone who has a deeper understanding of the more advanced Java mechanisms.
 

 

> DataflowPipelineOptions.tempLocation doesn't really default to stagingLocation
> ------------------------------------------------------------------------------
>
>                 Key: BEAM-88
>                 URL: https://issues.apache.org/jira/browse/BEAM-88
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>            Reporter: Eugene Kirpichov
>            Priority: Trivial
>
> The documentation of DataflowPipelineOptions.tempLocation says: ..."defaults to using
stagingLocation."...
> However calling .getTempLocation() when only --stagingLocation is specified on the command
line gives null.
> The "defaulting" is really done in DataflowPipelineRunner.fromOptions():
> {code}
>     if (Strings.isNullOrEmpty(dataflowOptions.getTempLocation())) {
>       dataflowOptions.setTempLocation(dataflowOptions.getStagingLocation());
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message