From dev-return-24322-archive-asf-public=cust-asf.ponee.io@spark.apache.org Thu Mar 29 21:34:45 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 54772180645 for ; Thu, 29 Mar 2018 21:34:45 +0200 (CEST) Received: (qmail 63122 invoked by uid 500); 29 Mar 2018 19:34:43 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 63105 invoked by uid 99); 29 Mar 2018 19:34:43 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Mar 2018 19:34:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 7DF78180677 for ; Thu, 29 Mar 2018 19:34:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.898 X-Spam-Level: * X-Spam-Status: No, score=1.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=netflix.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id qHjy7BjvKkN6 for ; Thu, 29 Mar 2018 19:34:41 +0000 (UTC) Received: from mail-io0-f180.google.com (mail-io0-f180.google.com [209.85.223.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 511A15F5F8 for ; Thu, 29 Mar 2018 19:34:41 +0000 (UTC) Received: by mail-io0-f180.google.com with SMTP id q84so8920038iod.10 for ; Thu, 29 Mar 2018 12:34:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:from:date:message-id :subject:to; bh=RJ4Sl52/M1C9GpFkgImjjAxbOndz4LQsTKwE96N6czA=; b=pz6YLqDJN/yLVfVIAiL4eeQA4CCbNRC27TK2EbhhoXiIiIpjyIb5up6+r17T1twmKe Nw+dTE5my3aLSACFj1cUTlmnROMTcDlr5RQsi+f4529y/Gnyu+xSGzmpBV000kBlVWEA prwz6HySidO97z7It3azbSEEHpksbCDX1cQ1kV7BUEurH/kgeGBIPS+QQclEwluN8Ggl TUxy/Qi3NgD6ZGRcne25RmeZznTwdXaZi3NzhgMHjEOIvEvxS63FGb+yfoi9gm+KDePa zaHVaeTCUBfi01VTKue05MduLMTGoCxEJu0rxo7wLpXc8DcAV6P1ihoa4+s1FdI9aWMD BtGA== X-Gm-Message-State: AElRT7FrLbi2qyK7i4QQwOKmM3nzgzMxTCXhvYqnfo8YcwdidxoX8bkm tK1DKbb66/MMuhe3n/kAVLeo2oSpEbiApRrHjx6xmznR X-Google-Smtp-Source: AIpwx49bFr967I1LW9oNzg0hEVLxg4VIzAC6mBnZ63tEDgcraDjlBkYFTbosME7MMrhG3lp5xB1B04ysF+Eq4QioJnw= X-Received: by 10.107.171.65 with SMTP id u62mr32778455ioe.73.1522352074935; Thu, 29 Mar 2018 12:34:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.192.136.229 with HTTP; Thu, 29 Mar 2018 12:34:04 -0700 (PDT) Reply-To: rblue@netflix.com From: Ryan Blue Date: Thu, 29 Mar 2018 12:34:04 -0700 Message-ID: Subject: [DISCUSS] Catalog APIs and multi-catalog support To: Spark Dev List Content-Type: multipart/alternative; boundary="001a114fec90a6807d05689236de" --001a114fec90a6807d05689236de Content-Type: text/plain; charset="UTF-8" Hi everyone, As a follow-up to the SPIP to clean up SparkSQL logical plans , I've written up a proposal for catalog APIs that are required for Spark to implement reliable high-level operations like CTAS. This includes an API that is an extension to DataSourceV2 that provides table operations, and proposes a public API to create and alter tables. The proposal is here: Spark Catalog APIs . Comments and feedback are welcome! Feel free to comment on the doc or reply to this thread. rb -- Ryan Blue Software Engineer Netflix --001a114fec90a6807d05689236de Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi everyone,

As a follow-up to the SPIP to clean up S= parkSQL logical plans, I've written up a proposal for catalog APIs = that are required for Spark to implement reliable high-level operations lik= e CTAS. This includes an API that is an extension to DataSourceV2 that prov= ides table operations, and proposes a public API to create and alter tables= .

The proposal is here: Spark Catalog APIs.

Comments and f= eedback are welcome! Feel free to comment on the doc or reply to this threa= d.

rb

--
Ryan Blue
Software Engineer
Netflix
--001a114fec90a6807d05689236de--