Mailing-List: contact issues-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Date: Tue, 25 Jul 2017 00:11:00 +0000 (UTC)
From: "Henry Robinson (JIRA)" <jira@apache.org>
To: issues@impala.incubator.apache.org
Message-ID: <JIRA.13055524.1486602175000.2515.1500941460179@Atlassian.JIRA>
In-Reply-To: <JIRA.13055524.1486602175000@Atlassian.JIRA>
References: <JIRA.13055524.1486602175000@Atlassian.JIRA> <JIRA.13055524.1486602175000@jira-lw-us.apache.org>
Subject: [jira] [Resolved] (IMPALA-4905) Fragments always report insert
 status, even if not insert query
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/IMPALA-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Henry Robinson resolved IMPALA-4905.
------------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

https://github.com/apache/incubator-impala/commit/d25db64f0e17092af9ef60eb37ec9214900c2d1c

> Fragments always report insert status, even if not insert query
> ---------------------------------------------------------------
>
>                 Key: IMPALA-4905
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4905
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Distributed Exec
>    Affects Versions: Impala 2.7.0
>            Reporter: Henry Robinson
>            Assignee: Henry Robinson
>             Fix For: Impala 2.10.0
>
>
> {code}
> if (done) {
>       TInsertExecStatus insert_status;
>       if (runtime_state->hdfs_files_to_move()->size() > 0) {
>         insert_status.__set_files_to_move(*runtime_state->hdfs_files_to_move());
>       }
>       if (runtime_state->per_partition_status()->size() > 0) {
>         insert_status.__set_per_partition_status(*runtime_state->per_partition_status());
>       }
>       params.__set_insert_exec_status(insert_status);
>     }
> {code}
> This means that any fragment will always set {{insert_exec_status}} in its response, even if it's not an INSERT query.
> However, in the RPC handler, {{Coordinator::UpdateFragmentExecStatus()}}, we have:
> {code}
> if (params.done && params.__isset.insert_exec_status) {
>     lock_guard<mutex> l(lock_);
>     // Merge in table update data (partitions written to, files to be moved as part of
>     // finalization)
>     for (const PartitionStatusMap::value_type& partition:
>          params.insert_exec_status.per_partition_status) {
>     // etc
> {code}
> which means that the RPC will always try and take the query exec state lock, for every 'done' report. With lots of fragment instances, this can lead to some severe serialisation of reports when the query finishes.
> The simplest workaround is not to set {{insert_exec_status}} for {{SELECT}} queries. But a better solution (that will help INSERTs as well) is not to try and do the merge here, but instead in {{Coordinator::FinalizeSuccessfulInsert()}}, saving the {{TInsertExecStatus}} in the fragment instance state until that point.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)