Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 83F2911071 for ; Thu, 21 Aug 2014 18:13:49 +0000 (UTC) Received: (qmail 87099 invoked by uid 500); 21 Aug 2014 18:13:49 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 87049 invoked by uid 500); 21 Aug 2014 18:13:49 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 87038 invoked by uid 99); 21 Aug 2014 18:13:48 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Aug 2014 18:13:48 +0000 Received: from localhost (HELO mail-qa0-f50.google.com) (127.0.0.1) (smtp-auth username sseth, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Aug 2014 18:13:48 +0000 Received: by mail-qa0-f50.google.com with SMTP id s7so8421162qap.23 for ; Thu, 21 Aug 2014 11:13:47 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.224.130.201 with SMTP id u9mr548157qas.58.1408644827792; Thu, 21 Aug 2014 11:13:47 -0700 (PDT) Received: by 10.96.76.3 with HTTP; Thu, 21 Aug 2014 11:13:47 -0700 (PDT) In-Reply-To: References: Date: Thu, 21 Aug 2014 11:13:47 -0700 Message-ID: Subject: Re: Hive on Tez Counters From: Siddharth Seth To: dev@hive.apache.org Cc: dev@tez.incubator.apache.org Content-Type: multipart/alternative; boundary=047d7b5d860f934f82050127ae84 --047d7b5d860f934f82050127ae84 Content-Type: text/plain; charset=UTF-8 I'll let Hive folks answer the questions about the Hive counters. In terms of the CPU counter - that was a bug in Tez-0.4.0, which has been fixed in 0.5.0. COMMITTED_HEAP_BYTES just represents the memory available to the JVM (Runtime.getRuntime().totalMemory()). This will only vary if the VM is started with a different Xms and Xmx option. In terms of Tez, the application logs are currently the best place. Hive may expose these in a more accessible manner though. On Wed, Aug 20, 2014 at 11:16 PM, Suma Shivaprasad < sumasai.shivaprasad@gmail.com> wrote: > Hi, > > Needed info on where I can get detailed job counters for Hive on Tez. Am > running this on a HDP cluster with Hive 0.13 and see only the following job > counters through Hive Tez in Yarn application logs which I got through( > yarn logs -applicationId ...) . > > a. Cannot see any ReduceOperator counters and also only DESERIALIZE_ERRORS > is the only counter present in MapOperator > b. The CPU_MILLISECONDS in some cases in -ve. Is CPU_MILLISECONDS accurate > c. What does COMMITTED_HEAP_BYTES indicate? > d. Is there any other place I should be checking the counters? > > [[File System Counters > FILE: BYTES_READ=512, > FILE: BYTES_WRITTEN=3079881, > FILE: READ_OPS=0, FILE: LARGE_READ_OPS=0, FILE: WRITE_OPS=0, HDFS: > BYTES_READ=8215153, HDFS: BYTES_WRITTEN=0, HDFS: READ_OPS=3, HDFS: > LARGE_READ_OPS=0, HDFS: WRITE_OPS=0] > > [org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=222543, > GC_TIME_MILLIS=172, *CPU_MILLISECONDS=-19700*, > PHYSICAL_MEMORY_BYTES=667566080, VIRTUAL_MEMORY_BYTES=1887797248, > COMMITTED_HEAP_BYTES=1011023872, INPUT_RECORDS_PROCESSED=222543, > OUTPUT_RECORDS=222543, > OUTPUT_BYTES=23543896, > OUTPUT_BYTES_WITH_OVERHEAD=23989024, OUTPUT_BYTES_PHYSICAL=3079369, > ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, > ADDITIONAL_SPILL_COUNT=0] > > > [*org.apache.hadoop.hive.ql.exec.MapOperator*$Counter > DESERIALIZE_ERRORS=0]] > > Thanks > Suma > --047d7b5d860f934f82050127ae84--