Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Message-Id: <201607200552.u6K5qcfj024480@ip-10-146-233-104.ec2.internal>
Date: Wed, 20 Jul 2016 05:52:38 +0000
From: "Sailesh Mukil (Code Review)" <gerrit@cloudera.org>
To: Henry Robinson <henry@cloudera.com>, impala-cr@cloudera.com,
        dev@impala.incubator.apache.org
CC: Dan Hecht <dhecht@cloudera.com>, Mostafa Mokhtar <mmokhtar@cloudera.com>
Reply-To: sailesh@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-CR=5D=28cdh5-trunk=29_IMPALA-3610=3A_Account_for_memory_used_by_filters_in_the_coordinator=0A?=
In-Reply-To: <gerrit.1468282677000.I3c52c8a1c2e79ef370c77bf264885fc859678d1b@gerrit.cloudera.org>
References: <gerrit.1468282677000.I3c52c8a1c2e79ef370c77bf264885fc859678d1b@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Wed, 20 Jul 2016 05:52:45 -0000

Sailesh Mukil has posted comments on this change.

Change subject: IMPALA-3610: Account for memory used by filters in the coordinator
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/3618/2/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:

PS2, Line 2077: state->DiscardFilter(filter_mem_tracker_.get());
Isn't this essentially a no-op, because an always_true_filter will always just consist of a NULL bloom_filter?

Or can a partitioned join filter have just one ALWAYS_TRUE_FILTER (in which case this is fine, because state->bloom_filter would have been set by another update)?


PS2, Line 2090: state->pending_count == 0
Wouldn't this always be 0 for a broadcast join filter? Given that we always set state->pending_count to 1 for a broadcast join filter with a remote target (in UpdateFilterRoutingTable())? Which makes this check redundant.

One way to handle this is to modify the code in UpdateFilterRoutingTable() to set the pending_count to 3 (or MAX_BROADCAST_FILTER_PRODUCERS) for broadcast join filters; and change the conditional in L2065 to be:

if (state->pending_count == 0 || (state->desc.is_broadcast_join && state->pending_count < MAX_BROADCAST_FILTER_PRODUCERS)) {...}


But this seems a little messy and if you think of a cleaner way, that would be much better.


PS2, Line 2110:     for (const auto& target: state->targets) {
              :       // Don't publish the filter to targets that are in the same fragment as the join
              :       // that produced it.
              :       if (target.is_local) continue;
              :       target_fragment_instance_idxs.insert(target.fragment_instance_idxs.begin(),
              :           target.fragment_instance_idxs.end());
              :     }
Seems like state->targets is read only at this point, because it's set only in UpdateFilterRoutingTable() which runs before sending out the remote fragments.

You could move the BloomFilter::ToThrift() before this (i.e. after L2109) and also release the filter_lock_ before running this for loop.


-- 
To view, visit http://gerrit.cloudera.org:8080/3618
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I3c52c8a1c2e79ef370c77bf264885fc859678d1b
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mmokhtar@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-HasComments: Yes