beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <>
Subject [jira] [Commented] (BEAM-3171) convert a join into lookup
Date Mon, 13 Nov 2017 18:01:00 GMT


Kenneth Knowles commented on BEAM-3171:

I am not strongly opposed to this, actually. I think the current design for side inputs in
Beam's portable model already has the needed features designed, but there is a lot of implementation
to finish in all the runners. Part of the concern is what the scope of this interface is.
It really has little to do with Beam if it is just a common interface across a few KV stores.
If it is experimental and part of the SQL package then I really think it is fine but perhaps
a more detailed design and scoping doc would help. SQL may or may not have the same needs
as other Beam uses of KV stores.

> convert a join into lookup
> --------------------------
>                 Key: BEAM-3171
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: dsl-sql
>            Reporter: Xu Mingmin
>            Assignee: Xu Mingmin
>              Labels: experimental
> We use BeamSQL to run streaming jobs mostly, and  add a join_as_lookup improvement(internal
branch) to cover the streaming-to-batch case(similar as [1]). I could submit a PR as experimental
if people are interested. 
> The rough solution is, if one source of join node implements {{BeamSeekableTable}} and
the other is not, then the join node is converted to a fact-lookup operation.
> Ref:
> [1]

> [~xumingming] [~takidau] for any comments

This message was sent by Atlassian JIRA

View raw message