From dev-return-2375-archive-asf-public=cust-asf.ponee.io@systemml.apache.org Sat Mar 17 05:21:09 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2D4F0180608 for ; Sat, 17 Mar 2018 05:21:09 +0100 (CET) Received: (qmail 49753 invoked by uid 500); 17 Mar 2018 04:21:08 -0000 Mailing-List: contact dev-help@systemml.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.apache.org Delivered-To: mailing list dev@systemml.apache.org Received: (qmail 49727 invoked by uid 99); 17 Mar 2018 04:21:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Mar 2018 04:21:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 840D31A0330 for ; Sat, 17 Mar 2018 04:21:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.129 X-Spam-Level: ** X-Spam-Status: No, score=2.129 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id hLBjj2KC8Ln0 for ; Sat, 17 Mar 2018 04:21:04 +0000 (UTC) Received: from mail-vk0-f52.google.com (mail-vk0-f52.google.com [209.85.213.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id BE7885F173 for ; Sat, 17 Mar 2018 04:21:03 +0000 (UTC) Received: by mail-vk0-f52.google.com with SMTP id d140so1868791vke.0 for ; Fri, 16 Mar 2018 21:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=2l4QC2i+PPPQphubGtJ4PafRyDZBHT5ETIB5Ef+FR/8=; b=W9NjWxeojevO8JojdkuP0mWWcTG66BeEk7fq2x3sWNgyREXVpUMohhsykrUEBqQHiE ukYQD/LYO0VM32MGgdBCc/OetHNJzjbKjp6HoL+Zpy5HbowGTeXB1rCkXBEGRDWrk/HF mwEs2hINRMEpGcdYhfbVPY8lhnSXqAwUxLZRPpesoSd8POlM0YvPYncVZnv3EP9zrrB3 SwBgnLEBZ3V4zcLw3fHur8wjFrWKExSVzppZFKFC6ItSd34nS6U4D2UMYvv8/L6Gi5dk hQD5mif/xN7AQKUBJxIfqhuC524QwpuEL1y6X/rQKZq46JYXbrwIPTCb3K4TyLmUQiY4 2ZUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=2l4QC2i+PPPQphubGtJ4PafRyDZBHT5ETIB5Ef+FR/8=; b=fRE//yujWA0RnaNzQpk0m/6g/+XcjpGQ/0GZcqR03vMrtuNubyxr2j415XzNruqXFZ qw3uDIdoO2uMqvC9GQr2P2NuC+5yoqpGmppgFlbZfNXOn1rfw/PrpjL4bFVk2e7EgkZz PWBj40O8R+VALSks7KpXeJ3Dy019vFy+RG1MpVY0HWoqMpvA9h/vXZnYcCs2qEor7+/O 06syA2r6dacVcJJ8x/RZyqM9TdbObeZD97VqB+V34uGgfJqSQamIIjBlOWwm0OTzqyQL EoGexO5rFV74RzPEqH2TDFEOY6+3k1uVC29QfqyQQt2fFpxg2oFYDS1g4UOCjXFQHWIu BLJA== X-Gm-Message-State: AElRT7EpOrR8zNUeBXftexlqGg/DczeYQT0yGXXGtovL1Q+BPfncYbDR /zQS7uubeEOAgAEvM4amF2aMlL2TWIjhUlAfxg== X-Google-Smtp-Source: AG47ELvPKc2Q9Zulk7zk2QfdDac8PmB0Z1NpFinTTY0tt7/Z53T1jndLburA3LMf2AOmHYRMysrcOwEPiY49aOP5htA= X-Received: by 10.31.141.80 with SMTP id p77mr2923283vkd.53.1521260463154; Fri, 16 Mar 2018 21:21:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.0.140 with HTTP; Fri, 16 Mar 2018 21:21:02 -0700 (PDT) From: Matthias Boehm Date: Fri, 16 Mar 2018 21:21:02 -0700 Message-ID: Subject: Re: Release Planning To: dev@systemml.apache.org Content-Type: multipart/alternative; boundary="001a11425ac0846b830567940d79" --001a11425ac0846b830567940d79 Content-Type: text/plain; charset="UTF-8" Thanks for the patience. Meanwhile, all known issues have been resolved and we're ready to cut an RC. Until this is done (or we have a 1.1 branch), I would recommend to limit all commits to critical fixes. Regards, Matthias On Tue, Mar 13, 2018 at 12:18 AM, Matthias Boehm wrote: > just a quick update: the ANTLR issue and most other things that came up > during QA are fixed now, except for performance issues with stratified > stats for which I need some more time (SYSTEMML-2181 tracks the open > issues). So it's probably a good idea to postpone the RC1 for a few days to > avoid unnecessary release efforts. > > Regards, > Matthias > > On Sun, Mar 11, 2018 at 5:43 PM, Matthias Boehm wrote: > >> well, after trying to run our perftest suite with Spark 2.3 and Spark 2.2 >> this seems to be more complicated. Although the version update from 4.5.3 >> to 4.7.1 solved the problem with Spark 2.3 (which uses 4.7.1), SystemML >> would no longer be backwards compatible with Spark 2.2 and 2.1 (which use >> 4.5.3) because ANTLR checks bidirectional mismatches.Unfortunately, >> removing ANTLR from our jar does not help because the above versions are >> binary incompatible. >> >> We need to hold off the release until we decided whether (1) we directly >> release for Spark 2.3 and drop 2.2 and 2.1, or (2) we release for Spark 2.2 >> and 2.1 with a subsequent ANTLR change and release for 2.3. >> >> Regards, >> Matthias >> >> On Thu, Mar 8, 2018 at 5:38 PM, Niketan Pansare >> wrote: >> >>> +1. That will preserve current behavior on older Spark versions too. >>> >>> > On Mar 8, 2018, at 5:24 PM, Ted Yu wrote: >>> > >>> > +1 on upgrading >>> > -------- Original message --------From: Matthias Boehm < >>> mboehm7@gmail.com> Date: 3/8/18 5:19 PM (GMT-08:00) To: >>> dev@systemml.apache.org Subject: Re: Release Planning >>> > related to Spark 2.3, we might want to update our ANTLR version because >>> > with Spark 2.3 every parsed DML script (i.e., multiple times with >>> imported >>> > DML files) produces the following warning: >>> > >>> > "ANTLR Tool version 4.5.3 used for code generation does not match the >>> > current runtime version 4.7" >>> > >>> > Regards, >>> > Matthias >>> > >>> >> On Thu, Mar 1, 2018 at 5:22 PM, Matthias Boehm >>> wrote: >>> >> >>> >> Hi all, >>> >> >>> >> I'm sure you've seen that Spark 2.3 just got released. This lines up >>> >> beautifully with our own SystemML 1.1 release. Accordingly, I would >>> >> recommend to use Spark 2.3 for our due diligence algorithm-level Q/A. >>> How >>> >> about we shoot for an RC1 by March 12? This should give enough time >>> to run >>> >> over reasonably large data and fix all related issues. >>> >> >>> >> Regards, >>> >> Matthias >>> >> >>> >>> On Tue, Feb 6, 2018 at 12:51 PM, Matthias Boehm >>> wrote: >>> >>> >>> >>> yes, absolutely. Here is a list of new features and improvements - >>> please >>> >>> feel free to extend as needed: >>> >>> >>> >>> 1) Extended Caffe2DML and Keras2DML APIs >>> >>> 2) Support for large-dense blocks >16GB in CP >>> >>> 3) New builtin functions ifelse, xor, as well as and/or/not over >>> matrices >>> >>> 4) Single-precision support for native conv2d and mm operations. >>> >>> 5) Performance features and correctness of ultra-sparse operations >>> >>> 6) Codegen: new plan cache, nary cbind >>> >>> 7) Parfor: result merge with accumulators +=, reduced initialization >>> >>> overhead >>> >>> 8) Compiler improvements: avoid unnecessary spark instructions, >>> corrected >>> >>> memory estimates >>> >>> >>> >>> Until the RC, I'd like to support all deep learning builtin >>> functions in >>> >>> codegen, add a couple of pending parfor improvements, and >>> potentially do a >>> >>> first cut of the compiler/runtime integration for parameter servers. >>> >>> >>> >>> Regards, >>> >>> Matthias >>> >>> >>> >>> >>> >>> >>> >>> On Tue, Feb 6, 2018 at 12:36 PM, Niketan Pansare >> > >>> >>> wrote: >>> >>> >>> >>>> +1. >>> >>>> >>> >>>> We should consider including single precision native BLAS in the >>> release >>> >>>> notes as well. If possible, we should add JNI wrappers for PowerPC, >>> Windows >>> >>>> and Mac too in this release. >>> >>>> >>> >>>>> On Feb 6, 2018, at 12:27 PM, Berthold Reinwald < >>> reinwald@us.ibm.com> >>> >>>> wrote: >>> >>>>> >>> >>>>> sure. >>> >>>>> >>> >>>>> Makes sense. Codegen and Keras2DML made good progress, and many >>> other >>> >>>>> fixes/improvements. >>> >>>>> >>> >>>>> What else do we want time/track/highlight for it? >>> >>>>> >>> >>>>> Regards, >>> >>>>> Berthold Reinwald >>> >>>>> IBM Almaden Research Center >>> >>>>> office: (408) 927 2208; T/L: 457 2208 >>> >>>>> e-mail: reinwald@us.ibm.com >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> From: Matthias Boehm >>> >>>>> To: dev@systemml.apache.org >>> >>>>> Date: 02/05/2018 11:05 PM >>> >>>>> Subject: Release Planning >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> Hi all, >>> >>>>> >>> >>>>> since our 1.0 release in Dec, we already got a number enhancements >>> and >>> >>>> new >>> >>>>> features in, so I think it would be good to discuss the timeline >>> for >>> >>>> our >>> >>>>> next SystemML 1.1 release. How about, we target mid March for a >>> first >>> >>>> RC? >>> >>>>> Also, Berthold would you be willing to serve again as the release >>> >>>> manager? >>> >>>>> >>> >>>>> Regards, >>> >>>>> Matthias >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>> >>> >>>> >>> >>> >>> >> >>> >>> >> > --001a11425ac0846b830567940d79--