beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henning Korsholm Rohde (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (BEAM-2600) Python SDK harness container
Date Mon, 11 Sep 2017 16:27:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Henning Korsholm Rohde updated BEAM-2600:
-----------------------------------------
    Description: 
In order to build a pipeline that invokes a Python UDF, we need to be able to construct something
like this:

{code}
SdkFunctionSpec {
  environment = <python SDK harness>,
  spec = {
    urn = <python SDK pickled DoFn>,
    data = <pickled DoFn>
  }
}
{code}

I could be out of date, but based on a couple of conversations I do not know that there exists
anything we can put for "<python SDK harness>" today. For prototyping, it could be just
a symbol that runners have to know. But eventually it should be something that runners can
instantiate without knowing anything about the SDK that put it there. I imagine it may encompass
"custom containers" eventually, though that doesn't block anything immediately.

Update 9/11/2017: this issue will track the first portable SDK harness as per , incl shared
boot code and  

  was:
In order to build a pipeline that invokes a Python UDF, we need to be able to construct something
like this:

{code}
SdkFunctionSpec {
  environment = <python SDK harness>,
  spec = {
    urn = <python SDK pickled DoFn>,
    data = <pickled DoFn>
  }
}
{code}

I could be out of date, but based on a couple of conversations I do not know that there exists
anything we can put for "<python SDK harness>" today. For prototyping, it could be just
a symbol that runners have to know. But eventually it should be something that runners can
instantiate without knowing anything about the SDK that put it there. I imagine it may encompass
"custom containers" eventually, though that doesn't block anything immediately.


> Python SDK harness container
> ----------------------------
>
>                 Key: BEAM-2600
>                 URL: https://issues.apache.org/jira/browse/BEAM-2600
>             Project: Beam
>          Issue Type: Sub-task
>          Components: sdk-py
>            Reporter: Kenneth Knowles
>            Assignee: Henning Korsholm Rohde
>              Labels: portability
>
> In order to build a pipeline that invokes a Python UDF, we need to be able to construct
something like this:
> {code}
> SdkFunctionSpec {
>   environment = <python SDK harness>,
>   spec = {
>     urn = <python SDK pickled DoFn>,
>     data = <pickled DoFn>
>   }
> }
> {code}
> I could be out of date, but based on a couple of conversations I do not know that there
exists anything we can put for "<python SDK harness>" today. For prototyping, it could
be just a symbol that runners have to know. But eventually it should be something that runners
can instantiate without knowing anything about the SDK that put it there. I imagine it may
encompass "custom containers" eventually, though that doesn't block anything immediately.
> Update 9/11/2017: this issue will track the first portable SDK harness as per , incl
shared boot code and  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message