From dev-return-2354-archive-asf-public=cust-asf.ponee.io@systemml.apache.org Tue Mar 6 19:25:13 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 52C6B180652 for ; Tue, 6 Mar 2018 19:25:12 +0100 (CET) Received: (qmail 66294 invoked by uid 500); 6 Mar 2018 18:25:11 -0000 Mailing-List: contact dev-help@systemml.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.apache.org Delivered-To: mailing list dev@systemml.apache.org Received: (qmail 66274 invoked by uid 99); 6 Mar 2018 18:25:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Mar 2018 18:25:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 123AC180374 for ; Tue, 6 Mar 2018 18:25:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.889 X-Spam-Level: * X-Spam-Status: No, score=1.889 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 3j4btz4dhSqY for ; Tue, 6 Mar 2018 18:25:07 +0000 (UTC) Received: from mail-pl0-f50.google.com (mail-pl0-f50.google.com [209.85.160.50]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E0FDB5F3F0 for ; Tue, 6 Mar 2018 18:25:06 +0000 (UTC) Received: by mail-pl0-f50.google.com with SMTP id s13-v6so12325081plq.6 for ; Tue, 06 Mar 2018 10:25:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=222L2OXu8LN+Lo3kDm/muzyoVBEZbUblET6CoZfEiHU=; b=dc4MESYEKxNEbpZeUALfr8u/6Yqu9CFN4iqs6xEMUz1QC8nPF5cm5fhO0c2TjQRvbm fPLWeUVLJThBEzPbNG7xXrO5HITiwwdOHvqDvcuIzh4ui56tmpKi9wpDdAGeXZ2nRGYq VH4exmfxq0SOApN+fcLQeUyp788TbjFSB2AHew8f+1PjplKzDTEiwGS0VbgzLfE5vOpE ae3E4Kzs8YCv4roL8QzyUqIwNR7BTlQUH5e5iOh5NSMG3kXRiycNZ5enZMEHry9Lv+6w jJX8hxsHpX8vYh+bV0suwpCcr3l0WF479K96rJAj8ij/HUZ++Utpq6Ovgku+qTO3AIfv yt0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=222L2OXu8LN+Lo3kDm/muzyoVBEZbUblET6CoZfEiHU=; b=NNgONVuL1BnkSk4nMFgIfUZoDKYPSwQ/ZxrdIKlATpkXfNT2ddoAXHf6doGtyROqhW VoVwfRQ7K9/JjRH/g5A4Tj78LYzrwZesOPEv0uAKUfvkr4ZoVDdscTAKzV4f2P0njrAc OakE6Ul7ahR8mOXd9usb+zJhbEGlXyWLDWQBT5Q1PPEm1+V+RKMxA74PhuX3LLznueyC zI3E+oSbcle5YVPMT+rnp3UMObsMgBFvt5P4FfMFcIeZUR5KOhN46qRFJg1xDHmq0BbJ ifHtGTP021NNROvO5D6pm3eCnG0JMLrpEC9bo48cjDTJpA+EAVmslp5WuUtBa/vTahWw GbcA== X-Gm-Message-State: APf1xPASdUFtyEMLPpTErnqWoH44NgmcTxnKuopZinq0KT+9g6zrLwSp ARLaHi/qFmkdj76EGs7j0k231jot7M7b275qG7o= X-Google-Smtp-Source: AG47ELvyYPCGVAWAYxNNZ+rfM0Fwdv4gsRHQk54vKrcvwK+0PVaUFKMaaOVfKXCzzv233GtBxnv733hFcY+XclLXGPU= X-Received: by 2002:a17:902:243:: with SMTP id 61-v6mr17540787plc.202.1520360699627; Tue, 06 Mar 2018 10:24:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.100.144.66 with HTTP; Tue, 6 Mar 2018 10:24:58 -0800 (PST) In-Reply-To: References: From: Chamath Abeysinghe Date: Tue, 6 Mar 2018 23:54:58 +0530 Message-ID: Subject: Re: Extending Codegen algorithm tests for heuristics To: Matthias Boehm Cc: dev@systemml.apache.org, Janardhana Ponnatapura Content-Type: multipart/alternative; boundary="0000000000006e8ce90566c28f1e" --0000000000006e8ce90566c28f1e Content-Type: text/plain; charset="UTF-8" Hi, I have sent a pull request for this issue. As a next step, could you suggest any new issue? or anything I have to do to familiarize with Language and run time for parameter servers project. And regarding writing the project proposal I have few questions. * In the epic there are few sub tasks, is it enough to focus on a single task through out the summer? Would it have enough work load or should I go for multiple tasks? * What is the linkage between sub tasks? Do tasks like, Distributed Spark Back-end or Local multi threaded back ends; need previous tasks completed before starting work? I am glad if you could suggests some issues related to Distributed spark back-end or multi threaded backend tasks. Thanks. Regards, Chamath On Fri, Mar 2, 2018 at 6:46 AM, Matthias Boehm wrote: > Hi Chamath, > > in general, you're absolutely right - you can enable -stats and > programmatically probe the heavy hitter statistics for certain opcodes. > However, uamin and uamax stand for "unary aggregate minimum" and "unary > aggregation maximum" which correspond to min(X) and max(X) on script level. > Instead all generated fused operators are prefixed with spoof or sp_spoof > (for distributed spark operations). The related junit assertion should > already be in the existing tests, I just mentioned it for completeness. > > Regards, > Matthias > > On Thu, Mar 1, 2018 at 4:30 AM, Chamath Abeysinghe < > abeysinghechamath@gmail.com> wrote: > >> Thanks for your detailed reply. >> I did some coding >> >> [1] for this issue SYSTEMML-2159 to extend test cases for FA & FNR . I got >> a problem regarding success criteria's, "generating at least one fused >> operator" condition, I think this means I have to look into stats of Heavy >> hitter instructions and check if there are any fused operators. (my guess >> is uamin and uamax are the operators what I have to look for, but I am not >> sure about this because I don't know the meaning of these instructions). >> >> Please help me to clarify this. If my approach is correct I could send a >> PR after fixing tests for other algorithms. Thanks. >> >> Regards, >> Chamath >> >> [1] https://github.com/apache/systemml/compare/master...cham >> athabeysinghe:SYSTEMML-2159?diff=split&name=SYSTEMML-2159 >> >> >> On Tue, Feb 27, 2018 at 1:54 AM, Matthias Boehm >> wrote: >> >>> ---------- Forwarded message ---------- >>> From: Matthias Boehm >>> Date: Mon, Feb 26, 2018 at 11:59 AM >>> Subject: Re: Extending Codegen algorithm tests for heuristics >>> To: Chamath Abeysinghe >>> >>> >>> great - thanks for taking this over Chamath. >>> >>> In general, I would recommend to use this task to explore SystemML a >>> little. For example, take one of the codegen algorithm tests from >>> org.apache.sysml.test.integration.functions.codegenalg (e.g., >>> AlgorithmL2SVM) and pass different flags such as -stats, -explain, >>> -explain >>> recompile_hops, -explain recompile_runtime to programArgs and try to >>> understand the output. If you come over specific questions, please just >>> ask. >>> >>> To answer your detailed questions: >>> >>> 1) We recently added a code generation framework that automatically >>> identifies opportunities for fused operators and subsequently generates >>> code for these operators. A major part is the selection of fusion plans, >>> for which we provide heuristics and a cost-based optimizer. By default >>> (and >>> thus also in our testsuite), we use the cost-based optimizer, but it >>> would >>> be good regularly test the heuristics as well. >>> >>> 2) You can configure the used optimizer in your SystemML-config.xml file >>> as >>> follows: >>> fuse_all >>> Valid alternatives are: fuse_all, fuse_no_redundancy, fuse_cost_based, >>> and >>> fuse_cost_based_v2 (default). You can provide alternative config xml >>> files >>> and switch them dynamically via getConfigTemplateFile. >>> >>> 3) Similar to the existing tests, it needs to (1) run without errors, (2) >>> produce correct results as compared to R, and (3) generate at least one >>> fused operator. >>> >>> Regards, >>> Matthias >>> >>> On Mon, Feb 26, 2018 at 6:54 AM, Chamath Abeysinghe < >>> abeysinghechamath@gmail.com> wrote: >>> >>> > Hi All, >>> > As per the guidelines given to GSoC students, I would like to work on >>> the >>> > SYSTEMML-2159 [1] issue as a starting point. But I don't understand the >>> > background of the issue. Can someone help me with understanding the >>> context >>> > of this issue? >>> > >>> > Few problems I got are, >>> > >>> > 1) What are fusion heuristics, fuse-all and fuse-no-redundancy? >>> > 2) Can I pass those heuristic related configurations as args to execute >>> > DMLScript? >>> > 3) What is the success criteria for a test that use those heuristics? >>> > >>> > Thank you in advance >>> > >>> > Regards, >>> > Chamath >>> > >>> > [1] https://issues.apache.org/jira/browse/SYSTEMML-2159 >>> > >>> > -- >>> > Chamath Abeysinghe >>> > Department of Computer Science and Engineering >>> > University of Moratuwa >>> > [image: >>> > https://www.linkedin.com/in/kaushalya-gayan-batawala-bbb5927 >>> a?trk=hp-identity-name] >>> > >>> > Mobile : +94752930548 >>> > >>> >> >> >> >> -- >> Chamath Abeysinghe >> Department of Computer Science and Engineering >> University of Moratuwa >> [image: >> https://www.linkedin.com/in/kaushalya-gayan-batawala-bbb5927a?trk=hp-identity-name] >> >> Mobile : +94752930548 >> > > -- Chamath Abeysinghe Department of Computer Science and Engineering University of Moratuwa Mobile: +94712803295 --0000000000006e8ce90566c28f1e--