phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From solzy <...@git.apache.org>
Subject [GitHub] phoenix issue #308: Client-side hash aggregation
Date Tue, 17 Jul 2018 01:39:04 GMT
Github user solzy commented on the issue:

    https://github.com/apache/phoenix/pull/308
  
    *@JamesRTaylor * Thanks for your clarify.
    
    
    ----------------------------------------
       Yun Zhang
       Best regards!
    
    
    2018-07-17 6:18 GMT+08:00 James Taylor <notifications@github.com>:
    
    > *@JamesRTaylor* commented on this pull request.
    > ------------------------------
    >
    > In phoenix-core/src/main/java/org/apache/phoenix/iterate/
    > ClientHashAggregatingResultIterator.java
    > <https://github.com/apache/phoenix/pull/308#discussion_r202842452>:
    >
    > > +    protected ImmutableBytesWritable getGroupingKey(Tuple tuple, ImmutableBytesWritable
ptr) throws SQLException {
    > +        try {
    > +            ImmutableBytesWritable key = TupleUtil.getConcatenatedValue(tuple, groupByExpressions);
    > +            ptr.set(key.get(), key.getOffset(), key.getLength());
    > +            return ptr;
    > +        } catch (IOException e) {
    > +            throw new SQLException(e);
    > +        }
    > +    }
    > +
    > +    // Copied from ClientGroupedAggregatingResultIterator
    > +    protected Tuple wrapKeyValueAsResult(KeyValue keyValue) {
    > +        return new MultiKeyValueTuple(Collections.<Cell> singletonList(keyValue));
    > +    }
    > +
    > +    private void populateHash() throws SQLException {
    >
    > @geraldss <https://github.com/geraldss> - memory management is tracked by
    > the GlobalMemoryManager. Operations that potentially use memory allocate
    > (and eventually free) a set of MemoryChunk instances. You can see an
    > example of this in GroupedAggregateRegionObserver (the runtime code for
    > aggregation). If the memory used goes over a threshold (phoenix.query.maxGlobalMemoryPercentage
    > and phoenix.query.maxTenantMemoryPercentage as the allowed percentage of
    > Java heap across all queries that is allowed to be used), then the query
    > will fail. Most typically, this mechanism is used on the server side as we
    > don't typically use a lot of memory on the client-side (as we're mostly
    > doing merge joins). One example where we use this on the client side is for
    > our broadcast join implementation (see HashCacheClient) to track memory
    > held onto for Hash Join caches.
    >
    > Classes you may want to look at (or perhaps you already have?):
    > OrderedResultIterator and MappedByteBufferSortedQueue. Above a certain
    > configurable threshold (phoenix.query.spoolThresholdBytes defaults to
    > 20MB), we output results into memory mapped files. Have you tried
    > decreasing that threshold?
    >
    > Couple of JIRAs you may want to take a look at: PHOENIX-2405 (unclear if
    > this is still an issue) and PHOENIX-3289. Are you running into issues with
    > too many memory mapped files?
    >
    > —
    > You are receiving this because you were mentioned.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/phoenix/pull/308#discussion_r202842452>, or mute
    > the thread
    > <https://github.com/notifications/unsubscribe-auth/AD1phzirsKewFaxyFttqCr5ybJNLnvWdks5uHREqgaJpZM4U_6wx>
    > .
    >



---

Mime
View raw message