From issues-return-7705-archive-asf-public=cust-asf.ponee.io@systemml.apache.org Sun Jan 28 09:18:11 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 1D34118064F for ; Sun, 28 Jan 2018 09:18:11 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 0D12F160C43; Sun, 28 Jan 2018 08:18:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 30F0B160C2A for ; Sun, 28 Jan 2018 09:18:10 +0100 (CET) Received: (qmail 66610 invoked by uid 500); 28 Jan 2018 08:18:09 -0000 Mailing-List: contact issues-help@systemml.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.apache.org Delivered-To: mailing list issues@systemml.apache.org Received: (qmail 66601 invoked by uid 99); 28 Jan 2018 08:18:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Jan 2018 08:18:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A921E1A0AF3 for ; Sun, 28 Jan 2018 08:18:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -102.311 X-Spam-Level: X-Spam-Status: No, score=-102.311 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id vr8hJ2xm0u1Q for ; Sun, 28 Jan 2018 08:18:07 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 2B34660EC8 for ; Sun, 28 Jan 2018 08:18:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 582A5E00B0 for ; Sun, 28 Jan 2018 08:18:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1733D21301 for ; Sun, 28 Jan 2018 08:18:00 +0000 (UTC) Date: Sun, 28 Jan 2018 08:18:00 +0000 (UTC) From: "Krishna Kalyan (JIRA)" To: issues@systemml.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (SYSTEMML-1451) Automate performance testing and reporting MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SYSTEMML-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Kalyan updated SYSTEMML-1451: ------------------------------------- Description: As part of a release (and in general), performance tests are run for SystemML. Currently, running and reporting on these performance tests are a manual process. There are helper scripts, but largely the process is manual. The aim of this GSoC 2017 project is to automate performance testing and its reporting. These are the tasks that this entails 1. Automate running of the performance tests, including generation of test data 2. Detect errors and report if any 3. Record performance benchmarking information 4. Automatically compare this performance to previous version to check for performance regressions 5. Automatically compare to Spark MLLib, R?, Julia? 6. Prepare report with all the information about failed jobs, performance information, perf info against other comparable projects/algorithms (plotted/in plain text in CSV, PDF or other common format) 7. Create scripts to automatically run this process on a cloud provider that spins up machines, runs the test, saves the reports and spins down the machines. 8. Create a web application to do this interactively without dropping down into a shell. As part of this project, the student will need to know scripting (in Bash, Python, etc). It may also involve changing error reporting and performance reporting code in SystemML. Rating - Medium (for the amount of work) Mentor - [~nakul02] (Other co-mentors will join in) was: As part of a release (and in general), performance tests are run for SystemML. Currently, running and reporting on these performance tests are a manual process. There are helper scripts, but largely the process is manual. The aim of this GSoC 2017 project is to automate performance testing and its reporting. These are the tasks that this entails 1. Automate running of the performance tests, including generation of test data 2. Detect errors and report if any 3. Record performance benchmarking information 4. Automatically compare this performance to previous version to check for performance regressions 5. Automatically compare to Spark MLLib, R?, Julia? 6. Prepare report with all the information about failed jobs, performance information, perf info against other comparable projects/algorithms (plotted/in plain text in CSV, PDF or other common format) 7. Create scripts to automatically run this process on a cloud provider that spins up machines, runs the test, saves the reports and spins down the machines. 8. Create a web application to do this interactively without dropping down into a shell. As part of this project, the student will need to know scripting (in Bash, Python, etc). It may also involve changing error reporting and performance reporting code in SystemML. Rating - Medium (for the amount of work) Mentor - [~nakul02] (Other co-mentors will join in) > Automate performance testing and reporting > ------------------------------------------ > > Key: SYSTEMML-1451 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1451 > Project: SystemML > Issue Type: Improvement > Components: Infrastructure, Test > Reporter: Nakul Jindal > Assignee: Krishna Kalyan > Priority: Major > Labels: gsoc2017, mentor, performance, reporting, testing > > As part of a release (and in general), performance tests are run for SystemML. > Currently, running and reporting on these performance tests are a manual process. There are helper scripts, but largely the process is manual. > The aim of this GSoC 2017 project is to automate performance testing and its reporting. > These are the tasks that this entails > 1. Automate running of the performance tests, including generation of test data > 2. Detect errors and report if any > 3. Record performance benchmarking information > 4. Automatically compare this performance to previous version to check for performance regressions > 5. Automatically compare to Spark MLLib, R?, Julia? > 6. Prepare report with all the information about failed jobs, performance information, perf info against other comparable projects/algorithms (plotted/in plain text in CSV, PDF or other common format) > 7. Create scripts to automatically run this process on a cloud provider that spins up machines, runs the test, saves the reports and spins down the machines. > 8. Create a web application to do this interactively without dropping down into a shell. > As part of this project, the student will need to know scripting (in Bash, Python, etc). It may also involve changing error reporting and performance reporting code in SystemML. > Rating - Medium (for the amount of work) > Mentor - [~nakul02] (Other co-mentors will join in) -- This message was sent by Atlassian JIRA (v7.6.3#76005)