impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailesh Mukil (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3610: Account for memory used by filters in the coordinator
Date Wed, 20 Jul 2016 05:52:38 GMT
Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-3610: Account for memory used by filters in the coordinator
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/3618/2/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:

PS2, Line 2077: state->DiscardFilter(filter_mem_tracker_.get());
Isn't this essentially a no-op, because an always_true_filter will always just consist of
a NULL bloom_filter?

Or can a partitioned join filter have just one ALWAYS_TRUE_FILTER (in which case this is fine,
because state->bloom_filter would have been set by another update)?


PS2, Line 2090: state->pending_count == 0
Wouldn't this always be 0 for a broadcast join filter? Given that we always set state->pending_count
to 1 for a broadcast join filter with a remote target (in UpdateFilterRoutingTable())? Which
makes this check redundant.

One way to handle this is to modify the code in UpdateFilterRoutingTable() to set the pending_count
to 3 (or MAX_BROADCAST_FILTER_PRODUCERS) for broadcast join filters; and change the conditional
in L2065 to be:

if (state->pending_count == 0 || (state->desc.is_broadcast_join && state->pending_count
< MAX_BROADCAST_FILTER_PRODUCERS)) {...}



But this seems a little messy and if you think of a cleaner way, that would be much better.


PS2, Line 2110:     for (const auto& target: state->targets) {
              :       // Don't publish the filter to targets that are in the same fragment
as the join
              :       // that produced it.
              :       if (target.is_local) continue;
              :       target_fragment_instance_idxs.insert(target.fragment_instance_idxs.begin(),
              :           target.fragment_instance_idxs.end());
              :     }
Seems like state->targets is read only at this point, because it's set only in UpdateFilterRoutingTable()
which runs before sending out the remote fragments.

You could move the BloomFilter::ToThrift() before this (i.e. after L2109) and also release
the filter_lock_ before running this for loop.


-- 
To view, visit http://gerrit.cloudera.org:8080/3618
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I3c52c8a1c2e79ef370c77bf264885fc859678d1b
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mmokhtar@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message