From issues-return-117935-archive-asf-public=cust-asf.ponee.io@hive.apache.org Sun May 13 18:59:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BBA0E180636 for ; Sun, 13 May 2018 18:59:04 +0200 (CEST) Received: (qmail 55936 invoked by uid 500); 13 May 2018 16:59:03 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 55927 invoked by uid 99); 13 May 2018 16:59:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 May 2018 16:59:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6AF09C00D5 for ; Sun, 13 May 2018 16:59:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id fPnOikR1pjm0 for ; Sun, 13 May 2018 16:59:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 4AC9D5F17D for ; Sun, 13 May 2018 16:59:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 49D82E0C50 for ; Sun, 13 May 2018 16:59:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 098E82179F for ; Sun, 13 May 2018 16:59:00 +0000 (UTC) Date: Sun, 13 May 2018 16:59:00 +0000 (UTC) From: "Vihang Karajgaonkar (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-19514) Successful test logs should be copied MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-19514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-19514: --------------------------------------- Description: Ptest tries to copy ~20G worth of logs from the worker nodes to the server. The total aggregate time spent by all the Ptest server in copying these files is close to 15 min (although much of that is in parallel threads). Most of the times we don't check for the logs. We seldom check for logs of successful tests. We should ignore copying successful test logs by default to improve runtime. {noformat} 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:235 Executed 13568 tests 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase took 71 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase.TotalRsyncElapsedTime took 15 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase took 5 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ReportingPhase took 0 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: TestCheckPhase took 0 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase took 0 minutes {noformat} was: Ptest tries to copy ~20G worth of logs from the worker nodes to the server. The total aggregate time spent by all the hosts in copying these files is close to 15 min in copying files. Most of the times we don't check for the logs. We seldom check for logs of successful tests. We should ignore copying successful test logs by default to improve runtime. {noformat} 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:235 Executed 13568 tests 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase took 71 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase.TotalRsyncElapsedTime took 15 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase took 5 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ReportingPhase took 0 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: TestCheckPhase took 0 minutes 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase took 0 minutes {noformat} > Successful test logs should be copied > ------------------------------------- > > Key: HIVE-19514 > URL: https://issues.apache.org/jira/browse/HIVE-19514 > Project: Hive > Issue Type: Sub-task > Reporter: Vihang Karajgaonkar > Assignee: Vihang Karajgaonkar > Priority: Major > > Ptest tries to copy ~20G worth of logs from the worker nodes to the server. The total aggregate time spent by all the Ptest server in copying these files is close to 15 min (although much of that is in parallel threads). Most of the times we don't check for the logs. We seldom check for logs of successful tests. We should ignore copying successful test logs by default to improve runtime. > {noformat} > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:235 Executed 13568 tests > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase took 71 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase.TotalRsyncElapsedTime took 15 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase took 5 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ReportingPhase took 0 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: TestCheckPhase took 0 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase took 0 minutes > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)