impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharath Vissapragada (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-5881: Use native allocation while building catalog updates
Date Sat, 09 Sep 2017 02:25:52 GMT
Bharath Vissapragada has posted comments on this change.

Change subject: IMPALA-5881: Use native allocation while building catalog updates

Patch Set 14:

File fe/src/main/java/org/apache/impala/thrift/

Line 103:       if (len <= 0) {
> I think it's better to remove and document the preconditions this function 
File fe/src/main/java/org/apache/impala/thrift/

Line 128:     if ((offset < 0) || (offset > b.length) || (len < 0)
> if (b == null) return;
File fe/src/test/java/org/apache/impala/thrift/

Line 42:           && (rand_.nextInt(10) < 8)) {
> Let's avoid non-determinism, otherwise a test failure may be hard to debug 
Fair point, switched to the counter logic.

Line 89:     while (testAllocator_.getAllocationFailures() < 10) {
> Not a big fan of the non-determinism. How about we run this loop a fixed (b
Now that the above logic is switched to a counter based one. this loop is deterministic as

Line 91:       byte[] b = new byte[byteArraySize];
> Can we get away with allocating a single array of size byteArrayMaxSize?
Wanted to check varied allocation sizes, but other tests already do it, so  I'm moving to
a single size. I"m using 1MB rather than 1 GB.

Line 92:       writeAndCheck(b, 0, byteArraySize);
> How does this work? Is the NBAOS expected to be left in a good state after 
Yea this only runs because the write actually doesn't modify the underlying nboas, due to
the TestAllocator. Please correct me if I'm wrong, but isn't the unit test for checking that
we free in all edge cases? TNativeSerialzer test already writes multiple sized objects that
does the underlying allocate/free/copying?

Line 97:     writeAndCheck(b, -1, 10);
> Test that len == 0 works

Line 100:     writeAndCheck(b, 5, 10);
> Test passing a null byte array
File fe/src/test/java/org/apache/impala/thrift/

Line 74:     UnsafeUtil.UNSAFE.freeMemory(result.buffer_ptr);
> put in finally block

Line 86:    * buffer resizing capability in the underlying NativeByteArrayOutputStreami and
> typo: extraneous 'i' after NBAOS

Line 107:     serializeTestObject(test15gb, protocolFactory);
> we also need to test the max 4GB
Added a test for ~3.5GB, made sure we serialize it ok, as we can't deserialize it on the Java
side. However it runs into occassional OOMs due to memory pressure. Still figuring out a way
to fix that. May be run testThriftLimits for only one protocol?

Line 119:     // The reason it is not a problem with Impala's catalog updates is because a
> I'd remove this sentence, it's not really true.
lol :) Done.

Line 139:     }
> Test that calling serialize() twice results in an exception.

To view, visit
To unsubscribe, visit

Gerrit-MessageType: comment
Gerrit-Change-Id: I383684effa9524734ce3c6c0fb7ed37de0e15782
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <>
Gerrit-Reviewer: Alex Behm <>
Gerrit-Reviewer: Bharath Vissapragada <>
Gerrit-Reviewer: Dimitris Tsirogiannis <>
Gerrit-Reviewer: Mostafa Mokhtar <>
Gerrit-Reviewer: Tim Armstrong <>
Gerrit-HasComments: Yes

View raw message