impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henry Robinson (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4905: Reduce coordinator lock contention in RPC handler
Date Fri, 10 Feb 2017 23:59:39 GMT
Henry Robinson has posted comments on this change.

Change subject: IMPALA-4905: Reduce coordinator lock contention in RPC handler
......................................................................


Patch Set 1:

(1 comment)

Changed how the logging works - it didn't quite work in the previous patch because WaitForAllInstance()
isn't called until late on in query execution.

http://gerrit.cloudera.org:8080/#/c/5971/1/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:

PS1, Line 829:   for (const InstanceState* state: fragment_instance_states_) {
             :     files_to_move.insert(
             :         state->insert_status().files_to_move.begin(),
             :         state->insert_status().files_to_move.end());
             : 
             :     for (const PartitionStatusMap::value_type& partition:
             :              state->insert_status().per_partition_status) {
             :       TInsertPartitionStatus* status = &(per_partition_status_[partition.first]);
             :       status->__set_num_modified_rows(
             :           status->num_modified_rows + partition.second.num_modified_rows);
             :       status->__set_kudu_latest_observed_ts(std::max(
             :               partition.second.kudu_latest_observed_ts, status->kudu_latest_observed_ts));
             :       status->__set_id(partition.second.id);
             :       status->__set_partition_base_dir(partition.second.partition_base_dir);
             : 
             :       if (partition.second.__isset.stats) {
             :         if (!status->__isset.stats) status->__set_stats(TInsertStats());
             :         DataSink::MergeDmlStats(partition.second.stats, &status->stats);
             :       }
             :     }
             :   }
> I expect that large INSERT queries will take a while longer to run because 
I don't think it's going to be excessive, even though this is an N^2 loop (the metadata update
times usually dominate). Worth checking to see if it makes a difference though.


-- 
To view, visit http://gerrit.cloudera.org:8080/5971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id7599780785c4e9306711f535bf4726a247873e2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message