hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10474) LLAP: investigate why TPCH Q1 1k is slow
Date Fri, 24 Apr 2015 16:12:38 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511287#comment-14511287
] 

Gopal V commented on HIVE-10474:
--------------------------------

Huge numbers of int[] arrays - I was expecting to find a huge number of ColumnVectors or something
like that

{code}
 JVM version is 25.25-b02
   5 Object Histogram:
   6 
   7 num       #instances    #bytes  Class description
   8 --------------------------------------------------------------------------
   9 1:              70473   227976176       int[]
  10 2:              852738  88757176        char[]
  11 3:              7152    58638832        double[]
  12 4:              30035   22348928        byte[][]
  13 5:              686743  21975776        java.util.Hashtable$Entry
  14 6:              293811  18803904        java.nio.DirectByteBuffer
  15 7:              762767  18306408        java.lang.String
  16 8:              219901  14073664        org.apache.hadoop.hive.llap.cache.LlapDataBuffer
  17 9:              37270   13536408        boolean[]
  18 10:             357189  11430048        java.util.concurrent.ConcurrentHashMap$Node
  19 11:             457895  10989480        java.lang.Long
  20 12:             129347  9312984 org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics
  21 13:             141337  8986752 java.lang.Object[]
  22 14:             316768  7641760 java.lang.String[]
  23 15:             1481    6373792 java.util.Hashtable$Entry[]
  24 16:             127001  6096048 org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndexEntry
  25 17:             234784  5634816 java.util.concurrent.ConcurrentSkipListMap$Node
  26 18:             857     4361200 java.util.concurrent.ConcurrentHashMap$Node[]
  27 19:             64053   3586968 org.apache.hadoop.hive.ql.io.orc.OrcProto$DoubleStatistics
  28 20:             221391  3542256 java.util.concurrent.atomic.AtomicInteger
  29 21:             133161  3195864 java.util.ArrayList
  30 22:             112042  2689008 java.util.Collections$UnmodifiableRandomAccessList
  31 23:             109038  2616912 java.util.concurrent.ConcurrentSkipListMap$Index
  32 24:             48730   2339040 org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics
  33 25:             66161   1587864 com.google.protobuf.LiteralByteString
  34 26:             32529   1301160 sun.misc.Cleaner
  35 27:             24668   1184064 org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper
  36 28:             22467   1078416 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$BufferChunk
  37 29:             14850   1069200 java.lang.reflect.Field
  38 30:             32529   1040928 java.nio.DirectByteBuffer$Deallocator
  39 31:             14496   927744  org.apache.hive.com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField
  40 32:             516     881544  long[]
{code}

> LLAP: investigate why TPCH Q1 1k is slow
> ----------------------------------------
>
>                 Key: HIVE-10474
>                 URL: https://issues.apache.org/jira/browse/HIVE-10474
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>         Attachments: llap-gc-pauses.png
>
>
> While most queries run faster in LLAP than just Tez with container reuse, TPCH Q1 is
much slower.
> On my run, on tez with container reuse (current default LLAP configuration but mode ==
container and no daemons running)  runs 2-6 (out of 6 consecutive runs in the same session)
finished in 25.5sec average; with 16 LLAP daemons in default config the average was 35.5sec;
same w/o IO elevator (to rule out its impact) it took 59.7sec w/strange distribution (later
runs were slower than earlier runs, still, fastest run was 49.5sec).
> So excluding IO elevator it's more than 2x degradation.
> We need to figure out why this is happening. Is it just slot discrepancy? Regardless,
this needs to be addressed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message