From issues-return-121914-archive-asf-public=cust-asf.ponee.io@hive.apache.org Sat Jun 2 18:46:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5B9BC180676 for ; Sat, 2 Jun 2018 18:46:04 +0200 (CEST) Received: (qmail 21247 invoked by uid 500); 2 Jun 2018 16:46:03 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 21165 invoked by uid 99); 2 Jun 2018 16:46:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Jun 2018 16:46:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id E3E71C0101 for ; Sat, 2 Jun 2018 16:46:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id lHcGJCVBQrzm for ; Sat, 2 Jun 2018 16:46:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id F2CF35F58F for ; Sat, 2 Jun 2018 16:46:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 158C1E0D3C for ; Sat, 2 Jun 2018 16:46:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 375782109C for ; Sat, 2 Jun 2018 16:46:00 +0000 (UTC) Date: Sat, 2 Jun 2018 16:46:00 +0000 (UTC) From: "Bharathkrishna Guruvayoor Murali (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-19525) Spark task logs print PLAN PATH excessive number of times MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19525: ---------------------------------------------------- Attachment: HIVE-19525.2.patch > Spark task logs print PLAN PATH excessive number of times > --------------------------------------------------------- > > Key: HIVE-19525 > URL: https://issues.apache.org/jira/browse/HIVE-19525 > Project: Hive > Issue Type: Sub-task > Components: Spark > Reporter: Sahil Takiar > Assignee: Bharathkrishna Guruvayoor Murali > Priority: Major > Attachments: HIVE-19525.1.patch, HIVE-19525.2.patch > > > A ton of logs with this {{Utilities - PLAN PATH = hdfs://localhost:59527/.../apache-hive/itests/qtest-spark/target/tmp/scratchdir/stakiar/6ebceb49-7a76-4159-9082-5bba44391e30/hive_2018-05-14_07-28-44_672_8205774950452575544-1/-mr-10006/bf14c0b5-a014-4ee8-8ddf-fdb7453eb0f0/map.xml}} > Seems it print multiple times per task exception, not sure where it is coming from, but its too verbose. It should be changed to DEBUG level. Furthermore, given that we are using {{Utilities#getBaseWork}} anytime we need to access a {{MapWork}} or {{ReduceWork}} object, we should make the method slightly more efficient. Right now it borrows a {{Kryo}} from a pool and does a bunch of stuff to set the classloader, then it checks the cache to see if the work object has already been created. It should check the cache before doing any of that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)