Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E6298200C2E for ; Sun, 19 Feb 2017 07:05:22 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E495E160B71; Sun, 19 Feb 2017 06:05:22 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 16146160B66 for ; Sun, 19 Feb 2017 07:05:21 +0100 (CET) Received: (qmail 76161 invoked by uid 500); 19 Feb 2017 06:05:21 -0000 Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.incubator.apache.org Delivered-To: mailing list dev@systemml.incubator.apache.org Received: (qmail 76150 invoked by uid 99); 19 Feb 2017 06:05:20 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Feb 2017 06:05:20 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5DDC61804C1 for ; Sun, 19 Feb 2017 06:05:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.298 X-Spam-Level: * X-Spam-Status: No, score=1.298 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id da-mWWKd68m0 for ; Sun, 19 Feb 2017 06:05:19 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 99FCD5F36C for ; Sun, 19 Feb 2017 06:05:18 +0000 (UTC) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v1J5sbKv125222 for ; Sun, 19 Feb 2017 01:05:13 -0500 Received: from e18.ny.us.ibm.com (e18.ny.us.ibm.com [129.33.205.208]) by mx0b-001b2d01.pphosted.com with ESMTP id 28pkb2ppru-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Sun, 19 Feb 2017 01:05:12 -0500 Received: from localhost by e18.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sun, 19 Feb 2017 01:05:12 -0500 Received: from d01dlp03.pok.ibm.com (9.56.250.168) by e18.ny.us.ibm.com (146.89.104.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sun, 19 Feb 2017 01:05:09 -0500 Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id EB1FCC90046 for ; Sun, 19 Feb 2017 01:04:49 -0500 (EST) Received: from b01ledav005.gho.pok.ibm.com (b01ledav005.gho.pok.ibm.com [9.57.199.110]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v1J64sEK38469772 for ; Sun, 19 Feb 2017 06:05:09 GMT Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E94E6AE043 for ; Sun, 19 Feb 2017 01:05:06 -0500 (EST) Received: from d50lp31.co.us.ibm.com (unknown [9.17.249.32]) by b01ledav005.gho.pok.ibm.com (Postfix) with ESMTPS id 9F79BAE054 for ; Sun, 19 Feb 2017 01:05:06 -0500 (EST) Received: from localhost by d50lp31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 18 Feb 2017 23:05:07 -0700 Received: from smtp.notes.na.collabserv.com (192.155.248.74) by d50lp31.co.us.ibm.com (192.168.2.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128/128) Sat, 18 Feb 2017 23:05:06 -0700 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Sun, 19 Feb 2017 06:05:05 -0000 Received: from us1a3-smtp07.a3.dal06.isc4sb.com (10.146.103.14) by smtp.notes.na.collabserv.com (10.106.227.92) with smtp.notes.na.collabserv.com ESMTP; Sun, 19 Feb 2017 06:05:04 -0000 Received: from us1a3-mail56.a3.dal09.isc4sb.com ([10.142.3.44]) by us1a3-smtp07.a3.dal06.isc4sb.com with ESMTP id 2017021906045551-18880 ; Sun, 19 Feb 2017 06:04:55 +0000 To: dev@systemml.incubator.apache.org In-Reply-To: <50983993.367150.1487484069099@mail.yahoo.com> Date: Sun, 19 Feb 2017 06:04:56 +0000 From: "Niketan Pansare" Subject: Re: Weighted Statistical Estimates MIME-Version: 1.0 X-Mailer: IBM Traveler 9.0.1.16 Build 201702061301_20 on behalf of device with id FH3...7NC, type 445 (ios) and description iPhone 6s:Apple-iPhone8C1/1401.403 (OS 10) X-KeepSent: 509A1B67:7932F690-002580CC:002168EA; type=4; name=$KeepSent X-LLNOutbound: False X-Disclaimed: 16259 X-TNEFEvaluated: 1 Content-Type: multipart/alternative; boundary=Apple-Mail-770867DE-5826-4B0F-B92F-103213A999EA x-cbid: 17021906-0044-0000-0000-000002987E8A X-IBM-SpamModules-Scores: BY=0.073841; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.387138; ST=0; TS=0; UL=0; ISC=; MB=0.026947 X-IBM-SpamModules-Versions: BY=3.00006643; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000203; SDB=6.00823919; UDB=6.00403261; IPR=6.00601376; BA=6.00005152; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014344; XFM=3.00000011; UTC=2017-02-19 06:05:05 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2017-02-19 03:45:04 - 6.00006321 x-cbparentid: 17021906-7582-0000-0000-000004257EC6 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006643; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000203; SDB=6.00823919; UDB=6.00403261; IPR=6.00601376; BA=6.00005152; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014344; XFM=3.00000011; UTC=2017-02-19 06:05:10 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-02-19_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1702190061 archived-at: Sun, 19 Feb 2017 06:05:23 -0000 --Apple-Mail-770867DE-5826-4B0F-B92F-103213A999EA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 +1 Thanks, Niketan=20 > On Feb 18, 2017, at 10:01 PM, Arvind Surve wr= ote: >=20 > +1 ------------------ Arvind Surve Spark Technology Center htt= p://www.spark.tc/ >=20 > From: Felix Sch=C3=BCler > To: dev@systemml.incubator.apache.org=20 > Sent: Saturday, February 18, 2017 9:42 PM > Subject: Re: Weighted Statistical Estimates >=20 > Sounds good! >=20 > -Felix >=20 >> On 18.02.2017 21:20, Matthias Boehm wrote: >> Going toward to our 1.0 release, I'd like to create consistency across o= ur >> weighted statistics. Conceptually, theses weights represent frequency >> counts, i.e., multiplicities of input values. >>=20 >> So far, our documentation does not state any restrictions on these weigh= ts >> but some runtime operations require integer data (I), while others allow >> arbitrary floating point data as indicated below: >>=20 >> * moment >> * cov >> * aggregate >> * table >> * median (I) >> * quantile (I) >> * interQuartileMean (I) >>=20 >> This can lead to unexpected errors as shown by recent issues such as >> SYSTEMML-1265. Looking back to R and its packages like Hmisc or reldist,= it >> turns out that they all allow arbitrary weights. >>=20 >> So, relaxing any restrictions of integer weights seems like the right >> choice. As this changes the external behavior - albeit in a generalizing >> manner - we should make this change now. If you have any concerns, let me >> know. >>=20 >> Regards, >> Matthias >>=20 >=20 >=20 >=20 --Apple-Mail-770867DE-5826-4B0F-B92F-103213A999EA--