Return-Path: X-Original-To: apmail-ignite-user-archive@minotaur.apache.org Delivered-To: apmail-ignite-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3C46518992 for ; Mon, 17 Aug 2015 19:12:50 +0000 (UTC) Received: (qmail 25657 invoked by uid 500); 17 Aug 2015 19:12:28 -0000 Delivered-To: apmail-ignite-user-archive@ignite.apache.org Received: (qmail 25613 invoked by uid 500); 17 Aug 2015 19:12:28 -0000 Mailing-List: contact user-help@ignite.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.incubator.apache.org Delivered-To: mailing list user@ignite.incubator.apache.org Received: (qmail 25603 invoked by uid 99); 17 Aug 2015 19:12:27 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2015 19:12:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 913F2DE86C for ; Mon, 17 Aug 2015 19:12:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.174 X-Spam-Level: ** X-Spam-Status: No, score=2.174 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, NML_ADSP_CUSTOM_MED=1.2, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Y2t-PYB0BagL for ; Mon, 17 Aug 2015 19:12:17 +0000 (UTC) Received: from mbob.nabble.com (mbob.nabble.com [162.253.133.15]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTP id CAF1F207D6 for ; Mon, 17 Aug 2015 19:12:16 +0000 (UTC) Received: from malf.nabble.com (unknown [162.253.133.59]) by mbob.nabble.com (Postfix) with ESMTP id 4315D1361F66 for ; Mon, 17 Aug 2015 12:10:58 -0700 (PDT) Date: Mon, 17 Aug 2015 12:06:30 -0700 (PDT) From: JohnnyS To: user@ignite.incubator.apache.org Message-ID: <1439838390258-1007.post@n6.nabble.com> In-Reply-To: <1439835224216-1000.post@n6.nabble.com> References: <1439811951418-996.post@n6.nabble.com> <1439835224216-1000.post@n6.nabble.com> Subject: Re: Local Node Query Optimization MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi The C++ implementation is almost completely custom. I used zip_iterators from Cuda Thrust to create a "virtual container" in the host memory from a bunch of STL vectors. I then sort this by key and use upper and lower to find the limits for the records in a tile. I then run a simple scan over these to count the number of distinct users. I'm fairly sure that this could be further optimized. The reading of the STL vectors sequentially should be as fast as possible. If I insert more data, the container needs to be sorted again which is not optimal. I could use global queries to count the number of distinct users. I do also a map-reduce like reduction to aggregate the gps tracks in a tile. To do this, I follow http://apache-ignite-users.70518.x6.nabble.com/Closure-method-on-a-Cache-Query-td456.html#a457 and do a query broadcast, reduce locally and this reduced result is returned to the caller. If it is possible to create complex custom aggregate SQL functions I could probably use these. I tried profiling using virtualvm and the bottleneck might be the marshalling, but the results are somewhat conflicting and I'm not sure of this. Hazelcast has binary and object formats for the values http://docs.hazelcast.org/docs/latest/manual/html/map-inmemoryformat.html. I could try this if something similar is available in Ignite. Regards, johnny. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-Node-Query-Optimization-tp996p1007.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.