phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-2405) Improve performance and stability of server side sort for ORDER BY
Date Tue, 19 Jul 2016 16:34:20 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384466#comment-15384466
] 

ASF GitHub Bot commented on PHOENIX-2405:
-----------------------------------------

GitHub user RCheungIT opened a pull request:

    https://github.com/apache/phoenix/pull/184

    PHOENIX-2405 

    https://issues.apache.org/jira/browse/PHOENIX-2405
    
    Hi @maryannxue, I guess this time it may be closer to what you described. 
    I think the threshold in DeferredResultIterator should be different from the threshold
in DeferredByteBufferSegmentQueue, but I don't know where to get it.
    Also, I don't find a good way to get rid of the offset in Iterator.
    Would you mind giving me any suggestions?
    
    Thanks

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/RCheungIT/phoenix PHOENIX-2405-v3

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/phoenix/pull/184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #184
    
----
commit 27df6878397b6e6c70c44d288f8d89ba35187880
Author: RCheungIT <hubert.zhang@outlook.com>
Date:   2016-07-19T15:54:17Z

    PHOENIX-2405 Improve performance and stability of server side sort for ORDER BY

----


> Improve performance and stability of server side sort for ORDER BY
> ------------------------------------------------------------------
>
>                 Key: PHOENIX-2405
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2405
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: Haoran Zhang
>              Labels: gsoc2016
>             Fix For: 4.9.0
>
>
> We currently use memory mapped files to buffer data as it's being sorted in an ORDER
BY (see MappedByteBufferQueue). The following types of exceptions have been seen to occur:
> {code}
> Caused by: java.lang.OutOfMemoryError: Map failed
>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
> {code}
> [~apurtell] has read that memory mapped files are not cleaned up after very well in Java:
> {quote}
> "Map failed" means the JVM ran out of virtual address space. If you search around stack
overflow for suggestions on what to do when your app (in this case Phoenix) encounters this
issue when using mapped buffers, the answers tend toward manually cleaning up the mapped buffers
or explicitly triggering a full GC. See http://stackoverflow.com/questions/8553158/prevent-outofmemory-when-using-java-nio-mappedbytebuffer
for example. There are apparently long standing JVM/JRE problems with reclamation of mapped
buffers. I think we may want to explore in Phoenix a different way to achieve what the current
code is doing.
> {quote}
> Instead of using memory mapped files, we could use heap memory, or perhaps there are
other mechanisms too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message