From user-return-17999-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Mon Feb 26 17:26:51 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1C98D18064A for ; Mon, 26 Feb 2018 17:26:50 +0100 (CET) Received: (qmail 63016 invoked by uid 500); 26 Feb 2018 16:26:49 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 63006 invoked by uid 99); 26 Feb 2018 16:26:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Feb 2018 16:26:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 58D6D1A0F63 for ; Mon, 26 Feb 2018 16:26:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.701 X-Spam-Level: X-Spam-Status: No, score=-0.701 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=aegon.onmicrosoft.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 1bU7mkYT_2MN for ; Mon, 26 Feb 2018 16:26:47 +0000 (UTC) Received: from mx0b-00099f01.pphosted.com (mx0a-00099f01.pphosted.com [67.231.149.228]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 56D6B5F1E7 for ; Mon, 26 Feb 2018 16:26:47 +0000 (UTC) Received: from pps.filterd (m0074058.ppops.net [127.0.0.1]) by mx0a-00099f01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w1QGLcaK028218 for ; Mon, 26 Feb 2018 10:26:46 -0600 Received: from crexpp04.us.aegon.com (mail2.aegonusa.com [162.123.17.223]) by mx0a-00099f01.pphosted.com with ESMTP id 2gb2hb4m2e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 26 Feb 2018 10:26:46 -0600 Received: from pps.filterd (crexpp04.us.aegon.com [127.0.0.1]) by crexpp04.us.aegon.com (8.16.0.21/8.16.0.21) with SMTP id w1QG3dLc026092 for ; Mon, 26 Feb 2018 10:26:45 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Aegon.onmicrosoft.com; s=selector1-transamerica-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Yo4WJzfvc5yVVBz5DOWT9N8YE8J8kw4eC9HkqoEDe68=; b=gpnTYhawcTmVMFlPr0iPM1dfdtGFMyPR8NsKCvJV73MbTo7cCicHtZb9wEw9PW9/9rKQJOLdWt0IOma01dvFn51tlzGv3vtjul+jFxOHUF89jtloJX7r5KxyS9OVCO1x1q+AMFxjRXq/B8NdP+w+4MHkvUr9v/IAUWyfTQ0z1t8= From: "Williams, Michael" To: "user@ignite.apache.org" Subject: RE: Slow Group-By Thread-Topic: Slow Group-By Thread-Index: AdOvDjENSM0WG/qnQ3ek//jMKfor9QAAZRNAAANekAAAACmQwA== Date: Mon, 26 Feb 2018 16:26:41 +0000 Message-ID: References: <1519661828620-0.post@n6.nabble.com> In-Reply-To: <1519661828620-0.post@n6.nabble.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [198.39.84.18] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM5PR05MB3161;7:5eRAhlRDPFilabN3M1msY/9PTB+2oe3uYlNtvmfsSNJSoq4o+zM5Fbo27zZAaPzBC/ats8g3XS6j3kmzB/iX+/4rmpIid096Zh15yTb5hn4Y2aDWGpmRtAG6dzbK2GxrUOCtEKQZWKBqnM1aWtgqxWe0IFvJWM7VP7ub9tGASizGxzDK7Aflwbe9PxvADbU4CETvW9lG/4KZXNLnAtcBG8mQt/eC31k64duismoam5PJdu5pF4kTHzubwSdsbb7N;20:YyLPbNa61iknOHGd5RmhdNQZevhbV8Mk76CqRSFvKUJTnTyiUFkdB9f6bxK7klTJoaIj+wXRTusIacuyDbmdQKJlMWSadKsec79khmWO44i33VfyIefZJdmazOLeeMxN4tAl9RdNqsjaFfhmcvewU/3w01YQtAoxH56jF2anwYSb0lVhg0IB6VSafUbGG4u6s3tGZdbRTUkGaoTPteEJXe84X9H2sQMhCfS/qcsn7ncMCB5jej1zy6GuCadryTUkmt8AwaYd9ceqq9huj/lgufWnFIBuwjxCvaY6GtBRLKeJD3Q504c5fhmYQv0jbkW+PuXJfIveM7qMfjt+H1zePQnmvsKP3ALbnsOOiqlbu11NcEqx6xQVaXtO9RU4tKDMso6g8HXBWxCYoRFh8p5FV0ki44hBf1y/clQcmZZxbORC7sxynk86rmrD/5VTMIlAK7Fm2BcWyKtZ7eK+ZLfYLyJNWBbJYhQicNlKpUCKGWafi7C5HC70s+hGRnIabQL9 x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: e0c8c20b-09fa-49b7-ab81-08d57d35b7db x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020);SRVR:DM5PR05MB3161; x-ms-traffictypediagnostic: DM5PR05MB3161: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(10436049006162)(85827821059158); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(8211001083)(6040501)(2401047)(8121501046)(5005006)(3002001)(3231220)(944501187)(93006095)(93001095)(10201501046)(6041288)(20161123564045)(20161123560045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(6072148)(201708071742011);SRVR:DM5PR05MB3161;BCL:0;PCL:0;RULEID:;SRVR:DM5PR05MB3161; x-forefront-prvs: 05954A7C45 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(39860400002)(39380400002)(346002)(376002)(396003)(366004)(189003)(51444003)(199004)(13464003)(106356001)(7696005)(2950100002)(6246003)(99286004)(2900100001)(575784001)(86362001)(5250100002)(2351001)(186003)(6916009)(6306002)(6436002)(2501003)(3660700001)(6346003)(53936002)(74316002)(966005)(14454004)(68736007)(316002)(26005)(33656002)(66066001)(2906002)(102836004)(9686003)(6116002)(25786009)(3846002)(5660300001)(7736002)(55016002)(6506007)(478600001)(81156014)(1730700003)(305945005)(105586002)(8676002)(5640700003)(53546011)(76176011)(3280700002)(81166006)(97736004)(59450400001)(229853002)(8936002);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR05MB3161;H:DM5PR05MB2954.namprd05.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; received-spf: None (protection.outlook.com: transamerica.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: n7Zmu+sCHqEo+sTvXscHUX2iTg/L/0RUDc+ZoUcESlij8hqExtKqV+zctgFQlw8gmQ5FRVDccsefXIaUeZ9/YCqYzHTm3fKRGQdzRwhQ8vz5oR2Wr2A2z70aupsI5Y+OX8iu53tmwO22A6WcZb7NnNjJ0zibBgwI5yBZFbNttMs= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: e0c8c20b-09fa-49b7-ab81-08d57d35b7db X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Feb 2018 16:26:41.3568 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46e16835-c804-41de-be3c-55835d14dee4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR05MB3161 X-EXCLAIMER-MD-CONFIG: 7562670a-beab-4c6e-8ed2-ab3b5287c042 X-OriginatorOrg: transamerica.com x-crexppdlp-TriggeredRule: module.access.rule.forcepoint_dlp_reroute v2 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-02-26_06:,, signatures=0 X-VPM-MSG-ID: 44b662f2-d043-408a-b8ca-98924c0ecfbb X-VPM-HOST: CREXZX01.inet.nogea.local X-VPM-GROUP-ID: 731e87f7-34d4-4422-90ad-134033dc6c5a X-VPM-ENC-REGIME: Plaintext X-VPM-IS-HYBRID: 0 x-crexpp01-TriggeredRule: module.access.rule.Strip_Receive_HeadersV2 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-02-26_06:,, signatures=0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-02-26_06:,, signatures=0 Unfortunately, at this stage in dev, I'm only doing runs on one machine, an= d though I am using partitioned data to do query parallelism, it seems I lo= se that in the GROUP BY. Does GROUP_BY distribute at all?=20 Might a spark layer on top give a better distribution path?=20 Mike =09 -----Original Message----- From: slava.koptilin [mailto:slava.koptilin@gmail.com]=20 Sent: Monday, February 26, 2018 11:17 AM To: user@ignite.apache.org Subject: RE: Slow Group-By Hi Mike, It seems that GROUP_BY requires to fetch all dataset into java heap (in ord= er to sort data) and it may lead to long GC pauses. I think that data collocation [1] should improve performance with using GRO= UP BY. [1] https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__apacheignite.rea= dme.io_docs_affinity-2Dcollocation&d=3DDwICAg&c=3D9g4MJkl2VjLjS6R4ei18BA&r= =3DipRRuqPnuP3BWnXGSOR_sLoARpltax56uFYU6n57c3GFvMdyEV-dz2ez2lZZpYl0&m=3DNkZ= 5g5gstJbpAgZaFvdxW5LiH0PKkDt17rQQ1t3pWlM&s=3DHrRyvf4qAOPX9Fc0eEdX83y-EvOBiW= Lqbn5f_aE99Pw&e=3D Thanks! -- Sent from: https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__apache-2Di= gnite-2Dusers.70518.x6.nabble.com_&d=3DDwICAg&c=3D9g4MJkl2VjLjS6R4ei18BA&r= =3DipRRuqPnuP3BWnXGSOR_sLoARpltax56uFYU6n57c3GFvMdyEV-dz2ez2lZZpYl0&m=3DNkZ= 5g5gstJbpAgZaFvdxW5LiH0PKkDt17rQQ1t3pWlM&s=3DU_kuoGAjhwdELc4JAGoFSPc76DNhai= SwpOJCDR3MGZ8&e=3D