impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dimitris Tsirogiannis (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] [PREVIEW] Use native allocation while building catalog updates
Date Wed, 06 Sep 2017 05:12:33 GMT
Dimitris Tsirogiannis has posted comments on this change.

Change subject: [PREVIEW] Use native allocation while building catalog updates
......................................................................


Patch Set 3:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/7955/3//COMMIT_MSG
Commit Message:

PS3, Line 20: 4-5GB
You need to comment about the limit (4GB) that currently thrift is imposing.


http://gerrit.cloudera.org:8080/#/c/7955/3/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:

PS3, Line 142: This method serializes
             :    * the output of CatalogService#getCatallogObjects() into native memory allocated
             :    * using a custom TNativeSerializer and returns the corresponding TNativeByteBuffer
             :    * serialized into bytes.
Alternatively: "Uses a custom serializer (TNativeSerializer) to serialize the catalog objects
into a byte buffer that is backed by native memory."


http://gerrit.cloudera.org:8080/#/c/7955/3/fe/src/main/java/org/apache/impala/thrift/NativeByteArrayOutputStream.java
File fe/src/main/java/org/apache/impala/thrift/NativeByteArrayOutputStream.java:

PS3, Line 36: 512MB
update value


Line 54:   // Limit on the size to which the underlying buffer can grow
Comment why that limit exists


PS3, Line 103: off
offset


PS3, Line 115: if (bufferLen_ >= BUFFER_DOUBLING_RESIZE_LIMIT) {
             :         while (newBufferSize < bytesWritten_ + len) {
             :           newBufferSize += BUFFER_RESIZE_INCREMENTS;
             :         }
             :       } else {
             :         while (newBufferSize < bytesWritten_ + len) {
             :           newBufferSize <<= 1;
             :         }
             :       }
I think if you put the if/else block inside the while block it may be a bit easier to follow
plus it will give you a slightly better behavior because you'll be doubling as long as newBufferSize
< 1GB and then you'll switch to incremental increases if you cross the 1GB boundary.


PS3, Line 135: public synchronized void reset() {
             :   }
single line (here and for some of the other functions as well).


-- 
To view, visit http://gerrit.cloudera.org:8080/7955
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I383684effa9524734ce3c6c0fb7ed37de0e15782
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mmokhtar@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message