hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Seth <ss...@apache.org>
Subject Re: Hive on Tez Counters
Date Thu, 21 Aug 2014 18:13:47 GMT
I'll let Hive folks answer the questions about the Hive counters.

In terms of the CPU counter - that was a bug in Tez-0.4.0, which has been
fixed in 0.5.0.

COMMITTED_HEAP_BYTES just represents the memory available to the JVM
(Runtime.getRuntime().totalMemory()). This will only vary if the VM is
started with a different Xms and Xmx option.

In terms of Tez, the application logs are currently the best place. Hive
may expose these in a more accessible manner though.


On Wed, Aug 20, 2014 at 11:16 PM, Suma Shivaprasad <
sumasai.shivaprasad@gmail.com> wrote:

> Hi,
>
> Needed info on where I can get detailed job counters for Hive on Tez. Am
> running this on a HDP cluster with Hive 0.13 and see only the following job
> counters through Hive Tez in Yarn application logs which I got through(
> yarn logs -applicationId ...) .
>
> a. Cannot see any ReduceOperator counters and also only DESERIALIZE_ERRORS
> is the only counter present in MapOperator
> b. The CPU_MILLISECONDS in some cases in -ve. Is CPU_MILLISECONDS accurate
> c. What does COMMITTED_HEAP_BYTES indicate?
> d. Is there any other place I should be checking the counters?
>
> [[File System Counters
> FILE: BYTES_READ=512,
> FILE: BYTES_WRITTEN=3079881,
> FILE: READ_OPS=0, FILE: LARGE_READ_OPS=0, FILE: WRITE_OPS=0, HDFS:
> BYTES_READ=8215153, HDFS: BYTES_WRITTEN=0, HDFS: READ_OPS=3, HDFS:
> LARGE_READ_OPS=0, HDFS: WRITE_OPS=0]
>
> [org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=222543,
> GC_TIME_MILLIS=172, *CPU_MILLISECONDS=-19700*,
> PHYSICAL_MEMORY_BYTES=667566080, VIRTUAL_MEMORY_BYTES=1887797248,
> COMMITTED_HEAP_BYTES=1011023872, INPUT_RECORDS_PROCESSED=222543,
> OUTPUT_RECORDS=222543,
> OUTPUT_BYTES=23543896,
> OUTPUT_BYTES_WITH_OVERHEAD=23989024, OUTPUT_BYTES_PHYSICAL=3079369,
> ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0,
> ADDITIONAL_SPILL_COUNT=0]
>
>
> [*org.apache.hadoop.hive.ql.exec.MapOperator*$Counter
> DESERIALIZE_ERRORS=0]]
>
> Thanks
> Suma
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message