impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-5522:Use tracked memory for DictDecoder and DictEncoder
Date Fri, 03 Nov 2017 23:07:31 GMT
Tim Armstrong has posted comments on this change. ( )

Change subject: IMPALA-5522:Use tracked memory for DictDecoder and DictEncoder

Patch Set 15:


Pretty close, have some formatting nits (nothing horrible, just trying to keep the codebase
consistent) and one perf concern.
File be/src/util/dict-encoding.h:
PS15, Line 54:     : dict_encoded_size_(0), pool_(NULL), dict_bytes_cnt_(0), dict_mem_tracker_(NULL)
Ok to leave for now but we've generally been moving towards using the recent C++ extension
that allows initialising member variables to constants at their declartion.
PS15, Line 59: DCHECK
DCHECK_EQ(dict_bytes_cnt_, 0);
PS15, Line 77:   void ClearIndices() {
We generally use the more concise one-line formatting for very short single-statement functions
like this one.
PS15, Line 123:  
formatting nit: * goes on the left with the type name.
PS15, Line 168: inline
nit: inline isn't necessary. It isn't harmful but can be confusing to have unnecessary modifiers.
PS15, Line 269:  *
nit: * should be on left.
PS15, Line 330: inline
nit: unnecessary inline
PS15, Line 374:   ConsumeBytes(sizeof(node));
I'm concerned that calling MemTracker::Consume for every node added to the table could hurt
performance in some workloads since it will end up incrementing the memory consumption counter
in the root process-wide MemTracker up to 40k times per dictionary.

(In contrast, MemPool::Allocate() is generally fine since it allocates memory in chunks).

How about we track the memory for some number of nodes at a time. E.g.

    ConsumeBytes(sizeof(node) * NODE_MEM_TRACKING_GRANULARITY);

To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3b54f6c107d19b62ad9e1c49df94175964299
Gerrit-Change-Number: 8034
Gerrit-PatchSet: 15
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Bikramjeet Vig <>
Gerrit-Reviewer: Joe McDonnell <>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <>
Gerrit-Comment-Date: Fri, 03 Nov 2017 23:07:31 +0000
Gerrit-HasComments: Yes

  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message