hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-17642) union test OOMs
Date Thu, 28 Sep 2017 22:55:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-17642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sergey Shelukhin updated HIVE-17642:
------------------------------------
    Description: 
Noticed this while investigating some spurious failures due to a combination of errors.
The reason unionDistinct (or whatever) test fails sporadically is OOM:
{noformat}
2017-09-28T01:07:25,804 ERROR [3d4e3f44-40c5-431a-b3de-801d60c1c579 main] ql.Driver: FAILED:
Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
vertexName=Map 61, vertexId=vertex_1506585924598_0001_53_01, diagnostics=[Vertex vertex_1506585924598_0001_53_01
[Map 61] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: src initializer failed,
vertex=vertex_1506585924598_0001_53_01 [Map 61], java.lang.OutOfMemoryError: GC overhead limit
exceeded
	at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
	at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
	at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852)
	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:721)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:442)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:591)
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
]Invalid event V_INTERNAL_ERROR on Vertex vertex_1506585924598_0001_53_00 [Map 60]
2017-09-28T01:07:25,804 DEBUG [3d4e3f44-40c5-431a-b3de-801d60c1c579 main] ql.Driver: Shutting
down query 


SELECT count(1) FROM (
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src) src
{noformat}

Perhaps we should make it smaller, or increase the GC limit.
Or maybe even make it less memory hungry :)


  was:
Noticed this while investigating some spurious failures due to a combination of errors.
The reason unionDistinct (or whatever) test fails sporadically is OOM:
{noformat}
2017-09-28T01:07:25,804 ERROR [3d4e3f44-40c5-431a-b3de-801d60c1c579 main] ql.Driver: FAILED:
Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
vertexName=Map 61, vertexId=vertex_1506585924598_0001_53_01, diagnostics=[Vertex vertex_1506585924598_0001_53_01
[Map 61] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: src initializer failed,
vertex=vertex_1506585924598_0001_53_01 [Map 61], java.lang.OutOfMemoryError: GC overhead limit
exceeded
	at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
	at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
	at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852)
	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:721)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:442)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:591)
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
]Invalid event V_INTERNAL_ERROR on Vertex vertex_1506585924598_0001_53_00 [Map 60]
2017-09-28T01:07:25,804 DEBUG [3d4e3f44-40c5-431a-b3de-801d60c1c579 main] ql.Driver: Shutting
down query 


SELECT count(1) FROM (
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT

  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src UNION DISTINCT
  SELECT key, value FROM src) src
{noformat}

Perhaps we should make it smaller.



> union test OOMs
> ---------------
>
>                 Key: HIVE-17642
>                 URL: https://issues.apache.org/jira/browse/HIVE-17642
>             Project: Hive
>          Issue Type: Bug
>          Components: Test
>            Reporter: Sergey Shelukhin
>
> Noticed this while investigating some spurious failures due to a combination of errors.
> The reason unionDistinct (or whatever) test fails sporadically is OOM:
> {noformat}
> 2017-09-28T01:07:25,804 ERROR [3d4e3f44-40c5-431a-b3de-801d60c1c579 main] ql.Driver:
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex
failed, vertexName=Map 61, vertexId=vertex_1506585924598_0001_53_01, diagnostics=[Vertex vertex_1506585924598_0001_53_01
[Map 61] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: src initializer failed,
vertex=vertex_1506585924598_0001_53_01 [Map 61], java.lang.OutOfMemoryError: GC overhead limit
exceeded
> 	at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
> 	at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
> 	at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852)
> 	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:721)
> 	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:442)
> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:591)
> 	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> 	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> 	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> 	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> 	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> ]Invalid event V_INTERNAL_ERROR on Vertex vertex_1506585924598_0001_53_00 [Map 60]
> 2017-09-28T01:07:25,804 DEBUG [3d4e3f44-40c5-431a-b3de-801d60c1c579 main] ql.Driver:
Shutting down query 
> SELECT count(1) FROM (
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src UNION DISTINCT
>   SELECT key, value FROM src) src
> {noformat}
> Perhaps we should make it smaller, or increase the GC limit.
> Or maybe even make it less memory hungry :)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message