From dev-return-53067-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Tue Jul 17 03:40:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2C7DB180657 for ; Tue, 17 Jul 2018 03:40:04 +0200 (CEST) Received: (qmail 10280 invoked by uid 500); 17 Jul 2018 01:40:03 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 10269 invoked by uid 99); 17 Jul 2018 01:40:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jul 2018 01:40:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id C46821A0841 for ; Tue, 17 Jul 2018 01:40:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 1gX-JHpTFP0W for ; Tue, 17 Jul 2018 01:40:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 5F9345F27D for ; Tue, 17 Jul 2018 01:40:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B52FBE013F for ; Tue, 17 Jul 2018 01:40:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 4D93823F99 for ; Tue, 17 Jul 2018 01:40:00 +0000 (UTC) Date: Tue, 17 Jul 2018 01:40:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-2405) Improve performance and stability of server side sort for ORDER BY MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-2405?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 545926#comment-16545926 ]=20 ASF GitHub Bot commented on PHOENIX-2405: ----------------------------------------- Github user solzy commented on the issue: https://github.com/apache/phoenix/pull/308 =20 *@JamesRTaylor * Thanks for your clarify. =20 =20 ---------------------------------------- Yun Zhang Best regards! =20 =20 2018-07-17 6:18 GMT+08:00 James Taylor : =20 > *@JamesRTaylor* commented on this pull request. > ------------------------------ > > In phoenix-core/src/main/java/org/apache/phoenix/iterate/ > ClientHashAggregatingResultIterator.java > : > > > + protected ImmutableBytesWritable getGroupingKey(Tuple tuple, I= mmutableBytesWritable ptr) throws SQLException { > + try { > + ImmutableBytesWritable key =3D TupleUtil.getConcatenated= Value(tuple, groupByExpressions); > + ptr.set(key.get(), key.getOffset(), key.getLength()); > + return ptr; > + } catch (IOException e) { > + throw new SQLException(e); > + } > + } > + > + // Copied from ClientGroupedAggregatingResultIterator > + protected Tuple wrapKeyValueAsResult(KeyValue keyValue) { > + return new MultiKeyValueTuple(Collections. singletonLi= st(keyValue)); > + } > + > + private void populateHash() throws SQLException { > > @geraldss - memory management is tracke= d by > the GlobalMemoryManager. Operations that potentially use memory alloc= ate > (and eventually free) a set of MemoryChunk instances. You can see an > example of this in GroupedAggregateRegionObserver (the runtime code f= or > aggregation). If the memory used goes over a threshold (phoenix.query= .maxGlobalMemoryPercentage > and phoenix.query.maxTenantMemoryPercentage as the allowed percentage= of > Java heap across all queries that is allowed to be used), then the qu= ery > will fail. Most typically, this mechanism is used on the server side = as we > don't typically use a lot of memory on the client-side (as we're most= ly > doing merge joins). One example where we use this on the client side = is for > our broadcast join implementation (see HashCacheClient) to track memo= ry > held onto for Hash Join caches. > > Classes you may want to look at (or perhaps you already have?): > OrderedResultIterator and MappedByteBufferSortedQueue. Above a certai= n > configurable threshold (phoenix.query.spoolThresholdBytes defaults to > 20MB), we output results into memory mapped files. Have you tried > decreasing that threshold? > > Couple of JIRAs you may want to take a look at: PHOENIX-2405 (unclear= if > this is still an issue) and PHOENIX-3289. Are you running into issues= with > too many memory mapped files? > > =E2=80=94 > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , o= r mute > the thread > > . > > Improve performance and stability of server side sort for ORDER BY > ------------------------------------------------------------------ > > Key: PHOENIX-2405 > URL: https://issues.apache.org/jira/browse/PHOENIX-2405 > Project: Phoenix > Issue Type: Bug > Reporter: James Taylor > Assignee: Haoran Zhang > Priority: Major > Labels: gsoc2016 > > We currently use memory mapped files to buffer data as it's being sorted = in an ORDER BY (see MappedByteBufferQueue). The following types of exceptio= ns have been seen to occur: > {code} > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904) > {code} > [~apurtell] has read that memory mapped files are not cleaned up after ve= ry well in Java: > {quote} > "Map failed" means the JVM ran out of virtual address space. If you searc= h around stack overflow for suggestions on what to do when your app (in thi= s case Phoenix) encounters this issue when using mapped buffers, the answer= s tend toward manually cleaning up the mapped buffers or explicitly trigger= ing a full GC. See http://stackoverflow.com/questions/8553158/prevent-outof= memory-when-using-java-nio-mappedbytebuffer for example. There are apparent= ly long standing JVM/JRE problems with reclamation of mapped buffers. I thi= nk we may want to explore in Phoenix a different way to achieve what the cu= rrent code is doing. > {quote} > Instead of using memory mapped files, we could use heap memory, or perhap= s there are other mechanisms too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)