impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dimitris Tsirogiannis (Code Review)" <>
Subject [Impala-ASF-CR] [PREVIEW] Use native allocation while building catalog updates
Date Wed, 06 Sep 2017 05:12:33 GMT
Dimitris Tsirogiannis has posted comments on this change.

Change subject: [PREVIEW] Use native allocation while building catalog updates

Patch Set 3:

Commit Message:

PS3, Line 20: 4-5GB
You need to comment about the limit (4GB) that currently thrift is imposing.
File fe/src/main/java/org/apache/impala/service/

PS3, Line 142: This method serializes
             :    * the output of CatalogService#getCatallogObjects() into native memory allocated
             :    * using a custom TNativeSerializer and returns the corresponding TNativeByteBuffer
             :    * serialized into bytes.
Alternatively: "Uses a custom serializer (TNativeSerializer) to serialize the catalog objects
into a byte buffer that is backed by native memory."
File fe/src/main/java/org/apache/impala/thrift/

PS3, Line 36: 512MB
update value

Line 54:   // Limit on the size to which the underlying buffer can grow
Comment why that limit exists

PS3, Line 103: off

PS3, Line 115: if (bufferLen_ >= BUFFER_DOUBLING_RESIZE_LIMIT) {
             :         while (newBufferSize < bytesWritten_ + len) {
             :           newBufferSize += BUFFER_RESIZE_INCREMENTS;
             :         }
             :       } else {
             :         while (newBufferSize < bytesWritten_ + len) {
             :           newBufferSize <<= 1;
             :         }
             :       }
I think if you put the if/else block inside the while block it may be a bit easier to follow
plus it will give you a slightly better behavior because you'll be doubling as long as newBufferSize
< 1GB and then you'll switch to incremental increases if you cross the 1GB boundary.

PS3, Line 135: public synchronized void reset() {
             :   }
single line (here and for some of the other functions as well).

To view, visit
To unsubscribe, visit

Gerrit-MessageType: comment
Gerrit-Change-Id: I383684effa9524734ce3c6c0fb7ed37de0e15782
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <>
Gerrit-Reviewer: Alex Behm <>
Gerrit-Reviewer: Bharath Vissapragada <>
Gerrit-Reviewer: Dimitris Tsirogiannis <>
Gerrit-Reviewer: Mostafa Mokhtar <>
Gerrit-Reviewer: Tim Armstrong <>
Gerrit-HasComments: Yes

View raw message