impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Ho (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4577: Adjust maximum size of row batch queue with MT DOP.
Date Fri, 02 Dec 2016 21:34:24 GMT
Michael Ho has posted comments on this change.

Change subject: IMPALA-4577: Adjust maximum size of row batch queue with MT_DOP.
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5330/1/be/src/exec/hdfs-scan-node.cc
File be/src/exec/hdfs-scan-node.cc:

PS1, Line 149:     if (state->query_options().__isset.mt_dop && state->query_options().mt_dop
> 0) {
             :       // To avoid a significant memory increase, adjust the number of maximally
queued
             :       // row batches per scan instance based on MT_DOP. The max materialized
row batches
             :       // is at least 2 to allow for some parallelism between the producer/consumer.
             :       max_materialized_row_batches_ =
             :           max(2, per_host_default_max_row_batches / state->query_options().mt_dop);
             :     } else {
             :       max_materialized_row_batches_ = per_host_default_max_row_batches;
             :     }
Just wondering if this should be outside of the if-stmt. For instance, if the user sets FLAG_max_row_batches,
we will still be MT_DOP * FLAGS_max_row_batches_ * row_batch_size amount of memory which seems
to be much larger than before.


-- 
To view, visit http://gerrit.cloudera.org:8080/5330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2aa260f9265ec21173fb703c41934964ece6485
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Michael Ho <kwho@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message