From commits-return-57845-archive-asf-public=cust-asf.ponee.io@beam.apache.org Sun Feb 4 06:58:05 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 638E1180676 for ; Sun, 4 Feb 2018 06:58:05 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 527FE160C38; Sun, 4 Feb 2018 05:58:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A1F7E160C58 for ; Sun, 4 Feb 2018 06:58:04 +0100 (CET) Received: (qmail 28878 invoked by uid 500); 4 Feb 2018 05:58:03 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 28745 invoked by uid 99); 4 Feb 2018 05:58:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Feb 2018 05:58:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 21116C018B for ; Sun, 4 Feb 2018 05:58:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id adVrKqK159su for ; Sun, 4 Feb 2018 05:58:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 1C9815F260 for ; Sun, 4 Feb 2018 05:58:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 55DF8E0114 for ; Sun, 4 Feb 2018 05:58:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1352021301 for ; Sun, 4 Feb 2018 05:58:00 +0000 (UTC) Date: Sun, 4 Feb 2018 05:58:00 +0000 (UTC) From: "Anton Kedin (JIRA)" To: commits@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (BEAM-3292) Remove BeamRecordSqlType MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Kedin resolved BEAM-3292. ------------------------------- Resolution: Fixed Fix Version/s: Not applicable > Remove BeamRecordSqlType > ------------------------ > > Key: BEAM-3292 > URL: https://issues.apache.org/jira/browse/BEAM-3292 > Project: Beam > Issue Type: Bug > Components: dsl-sql > Reporter: Anton Kedin > Assignee: Anton Kedin > Priority: Major > Fix For: Not applicable > > > [BeamRecordType|https://github.com/apache/beam/blob/39e66e953b0f8e16435acb038cad364acf2b3a57/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecordType.java] is implemented as 2 lists: the list of field names, and the list of the coders for those fields. Both lists are ordered. > [BeamRecordSqlType|https://github.com/apache/beam/blob/2eb7de0fe6e96da9805fc827294da1e1329ff716/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java] additionally has a list of [java.sql.Types|https://docs.oracle.com/javase/7/docs/api/java/sql/Types.html] ints to define types of those fields. It is used to map between Java types, Calcite types, and Beam Coders. > This information is not used for anything except for that mapping, which in turn is only used to create records and map back to Calcite types. > But because of this indirect mapping we cannot rely on core BeamRecordType and are forced to have BeamRecordSqlType. This introduces additional complexity, when, for example, generating record types based on pojo classes. > If we could find another mechanism to map Calcite types and java classes to Beam Coders bypassing java.sql.Types then we can just use the core BeamRecordType and remove the BeamRecordSqlType functionality. > One approach is to have a predefined set of coders which are then used like types, e.g.: > {code:java} > public static class SqlCoders { > public Coder INTEGER = VarIntCoder.of(); > public Coder VARCHAR = StringUtf8COder.of(); > public Coder TIMESTAMP = DateCoder.of(); > } > {code} > Problem with that approach is establishing the coders identity. That is, when a coder is serialized and then deserialized, it becomes a different instance, so we need a mechanism to know the identity or maybe just equality of the coders. If this is solved then replacing java.sql.Types with predefined SQL coders like above becomes trivial. > Few links on this: > - https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java#L56 > - https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslator.java#L34 > - https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L391 -- This message was sent by Atlassian JIRA (v7.6.3#76005)