beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Kedin (JIRA)" <>
Subject [jira] [Resolved] (BEAM-3292) Remove BeamRecordSqlType
Date Sun, 04 Feb 2018 05:58:00 GMT


Anton Kedin resolved BEAM-3292.
       Resolution: Fixed
    Fix Version/s: Not applicable

> Remove BeamRecordSqlType
> ------------------------
>                 Key: BEAM-3292
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: dsl-sql
>            Reporter: Anton Kedin
>            Assignee: Anton Kedin
>            Priority: Major
>             Fix For: Not applicable
> [BeamRecordType|]
is implemented as 2 lists: the list of field names, and the list of the coders for those fields.
Both lists are ordered.
> [BeamRecordSqlType|]
additionally has a list of [java.sql.Types|]
ints to define types of those fields. It is used to map between Java types, Calcite types,
and Beam Coders.
> This information is not used for anything except for that mapping, which in turn is only
used to create records and map back to Calcite types.
> But because of this indirect mapping we cannot rely on core BeamRecordType and are forced
to have BeamRecordSqlType. This introduces additional complexity, when, for example, generating
record types based on pojo classes.
> If we could find another mechanism to map Calcite types and java classes to Beam Coders
bypassing java.sql.Types then we can just use the core BeamRecordType and remove the BeamRecordSqlType
> One approach is to have a predefined set of coders which are then used like types, e.g.:
> {code:java}
> public static class SqlCoders {
>    public Coder INTEGER = VarIntCoder.of();
>    public Coder VARCHAR = StringUtf8COder.of();
>    public Coder TIMESTAMP = DateCoder.of();
> }
> {code}
> Problem with that approach is establishing the coders identity. That is, when a coder
is serialized and then deserialized, it becomes a different instance, so we need a mechanism
to know the identity or maybe just equality of the coders. If this is solved then replacing
java.sql.Types with predefined SQL coders like above becomes trivial.
> Few links on this:
>  -
> -
>  -

This message was sent by Atlassian JIRA

View raw message