impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Ho (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.
Date Fri, 24 Jun 2016 18:21:33 GMT
Hello Tim Armstrong,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/2781

to look at the new patch set (#9).

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................

IMPALA-1619: Support 64-bit allocations.

This change extends MemPool, FreePool and StringBuffer to support
64-bit allocations, fixes a bug in decompressor and extends various
places in the code to support 64-bit allocation sizes. With this
change, the text scanner can now decompress compressed files larger
than 1GB.

Note that the UDF interfaces FunctionContext::Allocate() and
FunctionContext::Reallocate() still use 32-bit for the input
argument to avoid breaking compatibility.

In addition, the byte size of a tuple is still assumed to be
within 32-bit. If it needs to be upgraded to 64-bit, it will be
done in a separate change.

Change-Id: I7ed28083d809a86d801a9c063a0aa32c50d32b20
---
M be/src/exec/delimited-text-parser.cc
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/scanner-context.cc
M be/src/runtime/buffered-block-mgr.cc
M be/src/runtime/collection-value-builder.h
M be/src/runtime/free-pool-test.cc
M be/src/runtime/free-pool.h
M be/src/runtime/mem-pool-test.cc
M be/src/runtime/mem-pool.h
M be/src/runtime/string-buffer-test.cc
M be/src/runtime/string-buffer.h
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
M be/src/udf/udf.h
M be/src/util/bit-util.h
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M common/thrift/generate_error_codes.py
M infra/python/bootstrap_virtualenv.py
A infra/python/deps/python-snappy-0.5.tar.gz
M infra/python/deps/requirements.txt
M tests/query_test/test_compressed_formats.py
30 files changed, 363 insertions(+), 338 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/81/2781/9
-- 
To view, visit http://gerrit.cloudera.org:8080/2781
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7ed28083d809a86d801a9c063a0aa32c50d32b20
Gerrit-PatchSet: 9
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kwho@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Michael Ho <kwho@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>

Mime
View raw message