From issues-return-130680-archive-asf-public=cust-asf.ponee.io@hive.apache.org Sun Jul 29 07:19:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 61A5018065B for ; Sun, 29 Jul 2018 07:19:04 +0200 (CEST) Received: (qmail 37861 invoked by uid 500); 29 Jul 2018 05:19:03 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 37852 invoked by uid 99); 29 Jul 2018 05:19:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Jul 2018 05:19:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BD64AC050D for ; Sun, 29 Jul 2018 05:19:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id R6YK-EdQ_Txi for ; Sun, 29 Jul 2018 05:19:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 55ECE5F366 for ; Sun, 29 Jul 2018 05:19:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8E030E0041 for ; Sun, 29 Jul 2018 05:19:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 14D5527756 for ; Sun, 29 Jul 2018 05:19:00 +0000 (UTC) Date: Sun, 29 Jul 2018 05:19:00 +0000 (UTC) From: "mahesh kumar behera (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --------------------------------------- Attachment: HIVE-19924.07.patch > Tag distcp jobs run by Repl Load > -------------------------------- > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl > Affects Versions: 3.1.0, 4.0.0 > Reporter: mahesh kumar behera > Assignee: mahesh kumar behera > Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, HIVE-19924.06.patch, HIVE-19924.07.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon issues a kill command. > * one of the tags should definitely be the query_id that starts the job : With this flow beacon before retrying the bootstrap load, will issue a kill command to hs2 with the query id of the previous issued command. hs2 will then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs can be tagged with an additional unique tag_id provided by Beacon in the WITH clause in repl load command to be used to tag distcp jobs ). Enhance the kill api to take the tag as input and kill jobs associated with that tag. Problem here is how do we validate the association of the tag with a hive query id to make sure this api is not used to kill jobs run by other components, however we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)