From dev-return-28910-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Apr 29 02:34:47 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id D10EB180649 for ; Mon, 29 Apr 2019 04:34:46 +0200 (CEST) Received: (qmail 99286 invoked by uid 500); 29 Apr 2019 02:34:44 -0000 Mailing-List: contact dev-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list dev@flink.apache.org Received: (qmail 99274 invoked by uid 99); 29 Apr 2019 02:34:43 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Apr 2019 02:34:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 182CEC56CE for ; Mon, 29 Apr 2019 02:34:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.122 X-Spam-Level: *** X-Spam-Status: No, score=3.122 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id DY9Hh2tqIN2W for ; Mon, 29 Apr 2019 02:34:40 +0000 (UTC) Received: from mail-vs1-f41.google.com (mail-vs1-f41.google.com [209.85.217.41]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 3B40C5F1F0 for ; Mon, 29 Apr 2019 02:34:40 +0000 (UTC) Received: by mail-vs1-f41.google.com with SMTP id t23so5030943vso.10 for ; Sun, 28 Apr 2019 19:34:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=SrDVkGMpw/w9s2hNCjLxQ8u2NvVdIx0u0yeHmSxMySY=; b=SdW39amXrP1NZXryrT7chq9GQ37blhOnC6fvj+LTUobtXvH84osvVx8UexUXSlu6Ed gIByHOOQDSFHqVUesTJiY/LGUE42PebntUPr+jwBQPsr9MHh0MahW/KsOJedwLupYj87 uIlAJVI49Fy4ksTrV4/kWBETJvHoM5MqO5IqwXEuJ1iVF6atJ0jvQ7iWyRq/fFYH+Ets Vjyqlogu9nJJ0GO4KXdwuqnf8gZAjF6xBcLhPbHbeyHQcAxpf0mdfRO4vXuJRyPSdmOl ne6lTIUVkHxaSJKhiKZB6jdWSHUFfEZI/ai+RzT0B3LutbjwOkyHXVUrFvf2XcfxJ+qE UibQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=SrDVkGMpw/w9s2hNCjLxQ8u2NvVdIx0u0yeHmSxMySY=; b=oj1/JKutYA1WqYYWUuuWdA0Se1rczOWlAH27jwSd4HKS560ifBjq+0Vk1IfUdsz+EC Kv2v83t2A+8mzs1g9pFF/B0zB4hEVkcJUAefs5HnFtf75v3ow1bdynJIi7IL8lXDvHKi HC9mttbZU4QMXjzB5j2BQYdQzGfHRWRxINYZOFZtXnK+42AMDgdlYTJzkh38dbgWMSJ+ KgfAPvAR0D94alo9KtaQ/GtBC74EnWkBiz9xu6icRk+K/Fqt6O2xlh3LIonuzQBBdkjY U++18npL84DSABbD/mH8bjTcKfbGnPmsn2Xx91tvZ5CVY9prq0bb4Ty9wXnVePDKjUS+ yJ+Q== X-Gm-Message-State: APjAAAVDWtrLS0ctrfCSh1ImMDqj0jEtDkIqyoOd9UYPLI05YAvfxUUe A73pxj8QONLpXGzjt7Io5fRHdf76MwVePa5cDF/8pavPhvo= X-Google-Smtp-Source: APXvYqyZzppAFZRsRYkNsjt1iBYlb68L5xBs7WmCOCUTk8/TvsY5qEkSPs4FaalLj7Dw7znT2uwYXFAwDKoqPPjn07A= X-Received: by 2002:a67:ed0f:: with SMTP id l15mr16640751vsp.147.1556505273293; Sun, 28 Apr 2019 19:34:33 -0700 (PDT) MIME-Version: 1.0 From: Shaoxuan Wang Date: Mon, 29 Apr 2019 10:34:22 +0800 Message-ID: Subject: [DISCUSS] FLIP-39: Flink ML pipeline and ML libs To: Dev Content-Type: multipart/alternative; boundary="000000000000e7e1160587a21fdd" --000000000000e7e1160587a21fdd Content-Type: text/plain; charset="UTF-8" Hi everyone, Weihua has proposed to rebuild Flink ML pipeline on top of TableAPI several months ago in this mail thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Embracing-Table-API-in-Flink-ML-td25368.html Luogen, Becket, Xu, Weihua and I have been working on this proposal offline in the past a few months. Now we want to share the first phase of the entire proposal with a FLIP. In this FLIP-39, we want to achieve several things (and hope those can be accomplished and released in Flink-1.9): - Provide a new set of ML core interface (on top of Flink TableAPI) - Provide a ML pipeline interface (on top of Flink TableAPI) - Provide the interfaces for parameters management and pipeline/mode persistence - All the above interfaces should facilitate any new ML algorithm. We will gradually add various standard ML algorithms on top of these new proposed interfaces to ensure their feasibility and scalability. Part of this FLIP has been present in Flink Forward 2019 @ San Francisco by Xu and Me. https://sf-2019.flink-forward.org/conference-program#when-table-meets-ai--build-flink-ai-ecosystem-on-table-api https://sf-2019.flink-forward.org/conference-program#high-performance-ml-library-based-on-flink You can find the videos & slides at https://www.ververica.com/flink-forward-san-francisco-2019 The design document for FLIP-39 can be found here: https://docs.google.com/document/d/1StObo1DLp8iiy0rbukx8kwAJb0BwDZrQrMWub3DzsEo I am looking forward to your feedback. Regards, Shaoxuan --000000000000e7e1160587a21fdd--