beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <>
Subject [jira] [Created] (BEAM-3204) Coders only should have a FunctionSpec, not an SdkFunctionSpec
Date Thu, 16 Nov 2017 19:58:00 GMT
Kenneth Knowles created BEAM-3204:

             Summary: Coders only should have a FunctionSpec, not an SdkFunctionSpec
                 Key: BEAM-3204
             Project: Beam
          Issue Type: Improvement
          Components: beam-model
            Reporter: Kenneth Knowles
            Assignee: Henning Rohde

We added environments to coders to account for "custom" coders where it is only really possible
for one SDK to understand them, like this:

Coder {
  spec: SdkFunctionSpec {
    environment: "java_sdk_docker_container",
    spec: FunctionSpec {
      urn: "beam:coder:java_custom_coder",
      payload: <serialized java bytes>

But a coder must be understood by both the producer of a PCollection and its consumers. A
coder is not the same as other UDF, though these are user-defined.

A pipeline where either the producer or consumer cannot handle the coder is invalid, and we
will have to build our cross-language APIs to prevent construction of such a pipeline. So
we can drop the environment.

I think there are some folks who want to reserve the ability to add an environment later,
perhaps, to not pain ourselves into a corner. In this case, we can just add a field to Coder.

This message was sent by Atlassian JIRA

View raw message