Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0573B2009DC for ; Tue, 2 May 2017 20:26:10 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 040C7160BBE; Tue, 2 May 2017 18:26:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 54926160B9D for ; Tue, 2 May 2017 20:26:09 +0200 (CEST) Received: (qmail 85351 invoked by uid 500); 2 May 2017 18:26:08 -0000 Mailing-List: contact issues-help@systemml.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.incubator.apache.org Delivered-To: mailing list issues@systemml.incubator.apache.org Received: (qmail 85314 invoked by uid 99); 2 May 2017 18:26:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 May 2017 18:26:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0E796C1431 for ; Tue, 2 May 2017 18:26:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id bMMgb3at-R9M for ; Tue, 2 May 2017 18:26:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id C7A765FC96 for ; Tue, 2 May 2017 18:26:05 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 1ECA1E0DA5 for ; Tue, 2 May 2017 18:26:05 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 5E59621DF2 for ; Tue, 2 May 2017 18:26:04 +0000 (UTC) Date: Tue, 2 May 2017 18:26:04 +0000 (UTC) From: "Glenn Weidner (JIRA)" To: issues@systemml.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (SYSTEMML-1423) OOM on generating ultra-sparse rand data MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 02 May 2017 18:26:10 -0000 [ https://issues.apache.org/jira/browse/SYSTEMML-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1423: ------------------------------------ Fix Version/s: (was: SystemML 1.0) SystemML 0.14 > OOM on generating ultra-sparse rand data > ---------------------------------------- > > Key: SYSTEMML-1423 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1423 > Project: SystemML > Issue Type: Bug > Reporter: Matthias Boehm > Assignee: Matthias Boehm > Fix For: SystemML 0.14 > > > In order to ensure consistency across backends, we first determine the number of non-zeros per block and subsequently generate random data accordingly. However, in case of ultra-sparse data sets, this temporary array can be almost as large as the dataset. Since this memory consumption is unaccounted and even required for distributed operations, there are various possible scenarios where this would cause OOMs. > This task aims to solve this issue for all backends, by determining the nnz per block in a streaming manner without materialization. -- This message was sent by Atlassian JIRA (v6.3.15#6346)