impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Ho (Code Review)" <>
Subject [Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.
Date Tue, 05 Jul 2016 21:30:54 GMT
Michael Ho has uploaded a new change for review.

Change subject: IMPALA-1619: Support 64-bit allocations.

IMPALA-1619: Support 64-bit allocations.

This change extends MemPool, FreePool and StringBuffer to support
64-bit allocations, fixes a bug in decompressor and extends various
places in the code to support 64-bit allocation sizes. With this
change, the text scanner can now decompress compressed files larger
than 1GB.

Note that the UDF interfaces FunctionContext::Allocate() and
FunctionContext::Reallocate() still use 32-bit for the input
argument to avoid breaking compatibility.

In addition, the byte size of a tuple is still assumed to be
within 32-bit. If it needs to be upgraded to 64-bit, it will be
done in a separate change.

A new test has been added to test the decompression of a 2GB
compressed text file.

Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
M be/src/exec/
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/
M be/src/exec/hdfs-scanner.h
M be/src/exec/
M be/src/exec/
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/
M be/src/runtime/
M be/src/runtime/collection-value-builder.h
M be/src/runtime/
M be/src/runtime/free-pool.h
M be/src/runtime/
M be/src/runtime/mem-pool.h
M be/src/runtime/
M be/src/runtime/string-buffer.h
M be/src/udf/udf-internal.h
M be/src/udf/
M be/src/udf/udf.h
M be/src/util/bit-util.h
M be/src/util/
M be/src/util/codec.h
M be/src/util/
M be/src/util/
M common/thrift/
M testdata/bin/
A testdata/compressed_formats/compressed_payload.snap
M tests/query_test/
29 files changed, 366 insertions(+), 340 deletions(-)

  git pull ssh:// refs/changes/75/3575/1
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <>

View raw message