From commits-return-83698-archive-asf-public=cust-asf.ponee.io@beam.apache.org Tue Jul 17 20:15:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id EE23818077B for ; Tue, 17 Jul 2018 20:15:04 +0200 (CEST) Received: (qmail 53646 invoked by uid 500); 17 Jul 2018 18:15:04 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 53637 invoked by uid 99); 17 Jul 2018 18:15:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jul 2018 18:15:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 87F161A07D1 for ; Tue, 17 Jul 2018 18:15:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id gtT-IxgPjZjs for ; Tue, 17 Jul 2018 18:15:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 5CE985F432 for ; Tue, 17 Jul 2018 18:15:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7F37BE20CD for ; Tue, 17 Jul 2018 18:15:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9DE9D23FA2 for ; Tue, 17 Jul 2018 18:15:00 +0000 (UTC) Date: Tue, 17 Jul 2018 18:15:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: commits@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (BEAM-4794) Move Nexmark and SQL to use the new Schema framework MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-4794?focusedWorklogId=124158&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124158 ] ASF GitHub Bot logged work on BEAM-4794: ---------------------------------------- Author: ASF GitHub Bot Created on: 17/Jul/18 18:14 Start Date: 17/Jul/18 18:14 Worklog Time Spent: 10m Work Description: akedin commented on a change in pull request #5956: [BEAM-4794] Move SQL and SQL Nexmark to the schema framework URL: https://github.com/apache/beam/pull/5956#discussion_r203127548 ########## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java ########## @@ -98,7 +98,7 @@ public void processElement(ProcessContext context) { context.output(jsonToRow(objectMapper(), context.element())); } })) - .setCoder(schema.getRowCoder()); + .setSchema(schema, SerializableFunctions.identity(), SerializableFunctions.identity()); Review comment: +1, some shorthand version would be helpful. Maybe default to `identity()` and have a couple of overloads of `setSchema()` to allow customization? Or wire it up to schema registry and default to `identity()` there? And is `setSchema` the right place to specify these transforms? Isn't it just conflating `.apply(ToRow.withSchema())... .apply(ParDo.of(fromRow()))`? My thought is that if the transforms are non-trivial, then they are probably better be specified as `ParDos` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 124158) Time Spent: 1h 20m (was: 1h 10m) > Move Nexmark and SQL to use the new Schema framework > ---------------------------------------------------- > > Key: BEAM-4794 > URL: https://issues.apache.org/jira/browse/BEAM-4794 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql > Reporter: Reuven Lax > Assignee: Reuven Lax > Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > This will allow SQL to accept user types. It will also allow the deletion of a lot of code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)