Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 45A0F200C2E for ; Sun, 19 Feb 2017 07:04:26 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 4420C160B71; Sun, 19 Feb 2017 06:04:26 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 67B78160B66 for ; Sun, 19 Feb 2017 07:04:25 +0100 (CET) Received: (qmail 75979 invoked by uid 500); 19 Feb 2017 06:04:24 -0000 Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.incubator.apache.org Delivered-To: mailing list dev@systemml.incubator.apache.org Received: (qmail 75968 invoked by uid 99); 19 Feb 2017 06:04:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Feb 2017 06:04:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B8439C002D for ; Sun, 19 Feb 2017 06:04:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.025 X-Spam-Level: ** X-Spam-Status: No, score=2.025 tagged_above=-999 required=6.31 tests=[HTML_IMAGE_ONLY_28=0.726, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, TVD_FW_GRAPHIC_NAME_MID=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 43G-KAu46GcJ for ; Sun, 19 Feb 2017 06:04:20 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id DF5385F47D for ; Sun, 19 Feb 2017 06:04:19 +0000 (UTC) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v1J5sb8V014062 for ; Sun, 19 Feb 2017 01:04:19 -0500 Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.152]) by mx0b-001b2d01.pphosted.com with ESMTP id 28pk9nxmg8-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Sun, 19 Feb 2017 01:04:19 -0500 Received: from localhost by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 18 Feb 2017 23:04:18 -0700 Received: from d03dlp03.boulder.ibm.com (9.17.202.179) by e34.co.us.ibm.com (192.168.1.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sat, 18 Feb 2017 23:04:16 -0700 Received: from b03cxnp08027.gho.boulder.ibm.com (b03cxnp08027.gho.boulder.ibm.com [9.17.130.19]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id C1F9619D803F for ; Sat, 18 Feb 2017 23:03:28 -0700 (MST) Received: from b03ledav006.gho.boulder.ibm.com (b03ledav006.gho.boulder.ibm.com [9.17.130.237]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v1J64GH210879400 for ; Sat, 18 Feb 2017 23:04:16 -0700 Received: from b03ledav006.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 407A0C612D for ; Sat, 18 Feb 2017 23:04:16 -0700 (MST) Received: from d50lp02.ny.us.ibm.com (unknown [146.89.104.208]) by b03ledav006.gho.boulder.ibm.com (Postfix) with ESMTPS id 04F17C612B for ; Sat, 18 Feb 2017 23:04:15 -0700 (MST) Received: from localhost by d50lp02.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sun, 19 Feb 2017 01:04:15 -0500 Received: from smtp.notes.na.collabserv.com (192.155.248.74) by d50lp02.ny.us.ibm.com (158.87.18.21) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128/128) Sun, 19 Feb 2017 01:04:12 -0500 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Sun, 19 Feb 2017 06:04:11 -0000 Received: from us1a3-smtp01.a3.dal06.isc4sb.com (10.106.154.95) by smtp.notes.na.collabserv.com (10.106.227.92) with smtp.notes.na.collabserv.com ESMTP; Sun, 19 Feb 2017 06:04:10 -0000 Received: from us1a3-mail52.a3.dal06.isc4sb.com ([10.146.77.168]) by us1a3-smtp01.a3.dal06.isc4sb.com with ESMTP id 2017021906040933-33343 ; Sun, 19 Feb 2017 06:04:09 +0000 MIME-Version: 1.0 In-Reply-To: <50983993.367150.1487484069099@mail.yahoo.com> Subject: Re: Weighted Statistical Estimates To: dev@systemml.incubator.apache.org From: "Glenn Weidner" Date: Sat, 18 Feb 2017 22:04:10 -0800 References: <50983993.367150.1487484069099@mail.yahoo.com> X-KeepSent: 4F240E82:E9302E96-002580CC:00214DCA; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1FP5 SHF190 February 24, 2016 X-LLNOutbound: False X-Disclaimed: 60463 X-TNEFEvaluated: 1 Content-type: multipart/related; Boundary="0__=8FBB0A5FDFB2CB5A8f9e8a93df938690918c8FBB0A5FDFB2CB5A" x-cbid: 17021906-0016-0000-0000-000006366BBE X-IBM-SpamModules-Scores: BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.387138; ST=0; TS=0; UL=0; ISC=; MB=0.003746 X-IBM-SpamModules-Versions: BY=3.00006643; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000203; SDB=6.00823919; UDB=6.00403261; IPR=6.00601376; BA=6.00005152; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014344; XFM=3.00000011; UTC=2017-02-19 06:04:11 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2017-02-19 03:45:04 - 6.00006321 x-cbparentid: 17021906-7582-0000-0000-000004257E97 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006643; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000203; SDB=6.00823919; UDB=6.00403261; IPR=6.00601376; BA=6.00005152; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014344; XFM=3.00000011; UTC=2017-02-19 06:04:17 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-02-19_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1702190061 archived-at: Sun, 19 Feb 2017 06:04:26 -0000 --0__=8FBB0A5FDFB2CB5A8f9e8a93df938690918c8FBB0A5FDFB2CB5A Content-type: multipart/alternative; Boundary="1__=8FBB0A5FDFB2CB5A8f9e8a93df938690918c8FBB0A5FDFB2CB5A" --1__=8FBB0A5FDFB2CB5A8f9e8a93df938690918c8FBB0A5FDFB2CB5A Content-Transfer-Encoding: quoted-printable Content-type: text/plain; charset=ISO-8859-1 +1 Thanks, Glenn From: Arvind Surve To: "dev@systemml.incubator.apache.org" Date: 02/18/2017 10:01 PM Subject: Re: Weighted Statistical Estimates +1=A0------------------ =A0 =A0=A0Arvind Surve =A0 =A0=A0Spark Technology C= enter http://www.spark.tc/ From: Felix Sch=FCler To: dev@systemml.incubator.apache.org Sent: Saturday, February 18, 2017 9:42 PM Subject: Re: Weighted Statistical Estimates Sounds good! -Felix On 18.02.2017 21:20, Matthias Boehm wrote: > Going toward to our 1.0 release, I'd like to create consistency across our > weighted statistics. Conceptually, theses weights represent frequency > counts, i.e., multiplicities of input values. > > So far, our documentation does not state any restrictions on these weights > but some runtime operations require integer data (I), while others allow > arbitrary floating point data as indicated below: > > * moment > * cov > * aggregate > * table > * median (I) > * quantile (I) > * interQuartileMean (I) > > This can lead to unexpected errors as shown by recent issues such as > SYSTEMML-1265. Looking back to R and its packages like Hmisc or reldist, it > turns out that they all allow arbitrary weights. > > So, relaxing any restrictions of integer weights seems like the right > choice. As this changes the external behavior - albeit in a generalizing > manner - we should make this change now. If you have any concerns, let me > know. > > Regards, > Matthias > --1__=8FBB0A5FDFB2CB5A8f9e8a93df938690918c8FBB0A5FDFB2CB5A Content-Transfer-Encoding: quoted-printable Content-type: text/html; charset=ISO-8859-1 Content-Disposition: inline

+1

Thanks,
Glenn


3D"InactiveArvind Surve ---02/18/2017 10:01:31 PM--= -+1=A0------------------ =A0 =A0=A0Arvind Surve =A0 =A0=A0Spark Technology = Center =A0 =A0=A0http://www.spark.tc/ = Fr

From: Arvind Surve <acs=5Fs@yahoo.com.INVALID>To: &qu= ot;dev@systemml.incubator.apache.org" <dev@systemml.incubator.apach= e.org>
Date: = 02/18/2017 10:01 PM
Subject: Re: Weighted Statistical Est= imates




+1=A0------------------ =A0 =A0=A0Arvi= nd Surve =A0 =A0=A0Spark Technology Center =A0 =A0=A0http://www.spark.tc/

   =  From: Felix Sch=FCler <fschueler@posteo.de>
To: dev@system= ml.incubator.apache.org
Sent: Saturday, February 18, 2017 9:42 PM
= Subject: Re: Weighted Statistical Estimates
 
Sounds good!
=
-Felix

On 18.02.2017 21:20, Matthias Boehm wrote:
> Going = toward to our 1.0 release, I'd like to create consistency across our
>= ; weighted statistics. Conceptually, theses weights represent frequency
= > counts, i.e., multiplicities of input values.
>
> So far, = our documentation does not state any restrictions on these weights
> = but some runtime operations require integer data (I), while others allow> arbitrary floating point data as indicated below:
>
> * m= oment
> * cov
> * aggregate
> * table
> * median (I= )
> * quantile (I)
> * interQuartileMean (I)
>
> Th= is can lead to unexpected errors as shown by recent issues such as
> = SYSTEMML-1265. Looking back to R and its packages like Hmisc or reldist, it=
> turns out that they all allow arbitrary weights.
>
> S= o, relaxing any restrictions of integer weights seems like the right
>= ; choice. As this changes the external behavior - albeit in a generalizing<= br>> manner - we should make this change now. If you have any concerns, = let me
> know.
>
> Regards,
> Matthias
>
<= br>

 


--1__=8FBB0A5FDFB2CB5A8f9e8a93df938690918c8FBB0A5FDFB2CB5A-- --0__=8FBB0A5FDFB2CB5A8f9e8a93df938690918c8FBB0A5FDFB2CB5A--