spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mitesh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter
Date Mon, 27 Mar 2017 21:31:41 GMT

     [ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mitesh updated SPARK-20112:
---------------------------
    Description: 
I'm seeing a very weird crash in {{GeneratedIterator.sort_addToSorter}}. The hs_err_pid and
codegen file are attached (with query plans). Its not a deterministic repro, but running a
big query load, I eventually see it come up within a few minutes.

Here is some interesting repro information:
- Using AWS r3.8xlarge machines, which have ephermal attached drives, I can't repro this.
But it does repro with m4.10xlarge with an io1 EBS drive. So I think that means its not an
issue with the code-gen, but I cant figure out what the difference in behavior is.
- The broadcast joins in the plan are all small tables. I have autoJoinBroadcast=-1 because
I always hint which tables should be broadcast.
- As you can see from the plan, all the sources are cached memory tables

{noformat}
# A fatal error has been detected by the Java Runtime Environment:
#
#  [thread 139872345896704 also had an error]
SIGSEGV (0xb) at pc=0x00007f38a378caa3, pid=19271, tid=139872342738688
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed
oops)

[thread 139872348002048 also had an error]# Problematic frame:
# 
J 28454 C1 org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIterator;)V
(369 bytes) @ 0x00007f38a378caa3 [0x00007f38a378b5e0+0x14c3]
{noformat}

  was:
I'm seeing a very weird crash in {{GeneratedIterator.sort_addToSorter}}. The hs_err_pid and
codegen file are attached (with query plans). Its not a deterministic repro, but running a
big query load, I eventually see it come up within a few minutes.

Here is some interesting repro information:
- Using r3.8xlarge machines, which have ephermal attached drives, I can't repro this. So I
think that means its not an issue with the code-gen, but I cant figure out what the difference
in behavior is.
- The broadcast joins in the plan are all small tables. I have autoJoinBroadcast=-1 because
I always hint which tables should be broadcast.
- As you can see from the plan, all the sources are cached memory tables

{noformat}
# A fatal error has been detected by the Java Runtime Environment:
#
#  [thread 139872345896704 also had an error]
SIGSEGV (0xb) at pc=0x00007f38a378caa3, pid=19271, tid=139872342738688
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed
oops)

[thread 139872348002048 also had an error]# Problematic frame:
# 
J 28454 C1 org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIterator;)V
(369 bytes) @ 0x00007f38a378caa3 [0x00007f38a378b5e0+0x14c3]
{noformat}


> SIGSEGV in GeneratedIterator.sort_addToSorter
> ---------------------------------------------
>
>                 Key: SPARK-20112
>                 URL: https://issues.apache.org/jira/browse/SPARK-20112
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.2
>         Environment: AWS m4.10xlarge with EBS (io1 drive, 400g, 4000iops)
>            Reporter: Mitesh
>         Attachments: codegen_sorter_crash.log, hs_err_pid19271.log
>
>
> I'm seeing a very weird crash in {{GeneratedIterator.sort_addToSorter}}. The hs_err_pid
and codegen file are attached (with query plans). Its not a deterministic repro, but running
a big query load, I eventually see it come up within a few minutes.
> Here is some interesting repro information:
> - Using AWS r3.8xlarge machines, which have ephermal attached drives, I can't repro this.
But it does repro with m4.10xlarge with an io1 EBS drive. So I think that means its not an
issue with the code-gen, but I cant figure out what the difference in behavior is.
> - The broadcast joins in the plan are all small tables. I have autoJoinBroadcast=-1 because
I always hint which tables should be broadcast.
> - As you can see from the plan, all the sources are cached memory tables
> {noformat}
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  [thread 139872345896704 also had an error]
> SIGSEGV (0xb) at pc=0x00007f38a378caa3, pid=19271, tid=139872342738688
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed
oops)
> [thread 139872348002048 also had an error]# Problematic frame:
> # 
> J 28454 C1 org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIterator;)V
(369 bytes) @ 0x00007f38a378caa3 [0x00007f38a378b5e0+0x14c3]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message