impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Ho (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.
Date Wed, 20 Apr 2016 01:53:15 GMT
Michael Ho has uploaded a new patch set (#2).

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................

IMPALA-1619: Support 64-bit allocations.

This change extends MemPool, FreePool and StringBuffer to support
64-bit allocations, fixes a bug decompressor and extends various
places in the code to support 64-bit allocation sizes. With this
change, the text scanner can now decompress compressed files larger
than 1GB.

This change also adds a function MemPool::TryFree() which will free
the chunk corresponding to the input buffer if the buffer covers the
entire range of a chunk. This is useful for reducing the memory
consumption of StringBuffer which will double its buffer without
freeing the old buffer and the memory is accumulated till the end
of the query.

Note that the UDF interfaces FunctionContext::Allocate() and
FunctionContext::Reallocate() still use 32-bit for the input
argument to avoid breaking compatibility.

In addition, the byte size of a tuple is still assumed to be
within 32-bit. If it needs to be upgraded to 64-bit, it will be
done in a separate change.

Change-Id: I7ed28083d809a86d801a9c063a0aa32c50d32b20
---
M be/src/exec/delimited-text-parser.cc
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/scanner-context.cc
M be/src/runtime/collection-value-builder.h
M be/src/runtime/free-pool-test.cc
M be/src/runtime/free-pool.h
M be/src/runtime/mem-pool-test.cc
M be/src/runtime/mem-pool.cc
M be/src/runtime/mem-pool.h
M be/src/runtime/string-buffer-test.cc
M be/src/runtime/string-buffer.h
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
M be/src/udf/udf.h
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M infra/python/bootstrap_virtualenv.py
M infra/python/deps/download_requirements
M infra/python/deps/requirements.txt
M tests/query_test/test_compressed_formats.py
27 files changed, 343 insertions(+), 265 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/81/2781/2
-- 
To view, visit http://gerrit.cloudera.org:8080/2781
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7ed28083d809a86d801a9c063a0aa32c50d32b20
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kwho@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>

Mime
View raw message