beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-3204) Coders only should have a FunctionSpec, not an SdkFunctionSpec
Date Thu, 16 Nov 2017 19:58:00 GMT
Kenneth Knowles created BEAM-3204:
-------------------------------------

             Summary: Coders only should have a FunctionSpec, not an SdkFunctionSpec
                 Key: BEAM-3204
                 URL: https://issues.apache.org/jira/browse/BEAM-3204
             Project: Beam
          Issue Type: Improvement
          Components: beam-model
            Reporter: Kenneth Knowles
            Assignee: Henning Rohde


We added environments to coders to account for "custom" coders where it is only really possible
for one SDK to understand them, like this:

{code}
Coder {
  spec: SdkFunctionSpec {
    environment: "java_sdk_docker_container",
    spec: FunctionSpec {
      urn: "beam:coder:java_custom_coder",
      payload: <serialized java bytes>
    }
  }
}
{code}

But a coder must be understood by both the producer of a PCollection and its consumers. A
coder is not the same as other UDF, though these are user-defined.

A pipeline where either the producer or consumer cannot handle the coder is invalid, and we
will have to build our cross-language APIs to prevent construction of such a pipeline. So
we can drop the environment.

I think there are some folks who want to reserve the ability to add an environment later,
perhaps, to not pain ourselves into a corner. In this case, we can just add a field to Coder.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message