flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chesnay Schepler (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3290) [py] Generalize OperationInfo transfer
Date Tue, 26 Jan 2016 11:41:39 GMT
Chesnay Schepler created FLINK-3290:

             Summary: [py] Generalize OperationInfo transfer
                 Key: FLINK-3290
                 URL: https://issues.apache.org/jira/browse/FLINK-3290
             Project: Flink
          Issue Type: Bug
          Components: Python API
    Affects Versions: 0.10.1
            Reporter: Chesnay Schepler
            Assignee: Chesnay Schepler
             Fix For: 1.00

A set number of arguments is transferred whenever a user defines an operation. For a CSV Source
for example these are delimiters/filepath, for a map function only the set ID'S are transferred.
As such, for all operators a separate routine is defined that governs which arguments are

While working on FLINK-3275 I realized that adding a new argument/parameter, in this case
parallelism, is not as straightforward as it could be. Most newly added operators will require
a new routine; whereas adding new arguments may require the modification of multiple routines.
Over times, this is bound to become a big mess.

All arguments are stored in an OperationInfo object, which also contains default values for
all unused arguments. I want to generalize the whole affair by transferring all arguments,
used or not. 

This will reduce clutter, make it easier to add new parameters (only 4 new lines needed, 2
for defining new fields inside Java/Python OperationInfo Classes; 1 each for sending/receiving
the new argument) and will make the transfer consistent across all operations.

This message was sent by Atlassian JIRA

View raw message