From issues-return-146398-archive-asf-public=cust-asf.ponee.io@flink.apache.org Fri Jan 5 18:41:04 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 089E4180647 for ; Fri, 5 Jan 2018 18:41:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id ECE9E160C27; Fri, 5 Jan 2018 17:41:03 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3C8AD160C15 for ; Fri, 5 Jan 2018 18:41:03 +0100 (CET) Received: (qmail 14109 invoked by uid 500); 5 Jan 2018 17:41:02 -0000 Mailing-List: contact issues-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list issues@flink.apache.org Received: (qmail 14100 invoked by uid 99); 5 Jan 2018 17:41:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jan 2018 17:41:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id CB9FAC2B0E for ; Fri, 5 Jan 2018 17:41:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.911 X-Spam-Level: X-Spam-Status: No, score=-99.911 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 22DJE2n9IVT5 for ; Fri, 5 Jan 2018 17:41:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id C17725F1BA for ; Fri, 5 Jan 2018 17:41:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 54B81E0383 for ; Fri, 5 Jan 2018 17:41:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 17BDA212F8 for ; Fri, 5 Jan 2018 17:41:00 +0000 (UTC) Date: Fri, 5 Jan 2018 17:41:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@flink.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FLINK-8203) Make schema definition of DataStream/DataSet to Table conversion more flexible MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/FLINK-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313513#comment-16313513 ] ASF GitHub Bot commented on FLINK-8203: --------------------------------------- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5132 > Make schema definition of DataStream/DataSet to Table conversion more flexible > ------------------------------------------------------------------------------ > > Key: FLINK-8203 > URL: https://issues.apache.org/jira/browse/FLINK-8203 > Project: Flink > Issue Type: Bug > Components: Table API & SQL > Affects Versions: 1.4.0, 1.5.0 > Reporter: Fabian Hueske > Assignee: Timo Walther > Fix For: 1.5.0 > > > When converting or registering a {{DataStream}} or {{DataSet}} as {{Table}}, the schema of the table can be defined (by default it is extracted from the {{TypeInformation}}. > The schema needs to be manually specified to select (project) fields, rename fields, or define time attributes. Right now, there are several limitations how the fields can be defined that also depend on the type of the {{DataStream}} / {{DataSet}}. Types with explicit field ordering (e.g., tuples, case classes, Row) require schema definition based on the position of fields. Pojo types which have no fixed order of fields, require to refer to fields by name. Moreover, there are several restrictions on how time attributes can be defined, e.g., event time attribute must replace an existing field or be appended and proctime attributes must be appended. > I think we can make the schema definition more flexible and provide two modes: > 1. Reference input fields by name: All fields in the schema definition are referenced by name (and possibly renamed using an alias ({{as}}). In this mode, fields can be reordered and projected out. Moreover, we can define proctime and eventtime attributes at arbitrary positions using arbitrary names (except those that existing the result schema). This mode can be used for any input type, including POJOs. This mode is used if all field references exist in the input type. > 2. Reference input fields by position: Field references might not refer to existing fields in the input type. In this mode, fields are simply renamed. Event-time attributes can replace the field on their position in the input data (if it is of correct type) or be appended at the end. Proctime attributes must be appended at the end. This mode can only be used if the input type has a defined field order (tuple, case class, Row). > We need to add more tests the check for all combinations of input types and schema definition modes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)