spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Kleyn (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23801) Consistent SIGSEGV after upgrading to Spark v2.3.0
Date Tue, 27 Mar 2018 15:04:00 GMT
Nathan Kleyn created SPARK-23801:
------------------------------------

             Summary: Consistent SIGSEGV after upgrading to Spark v2.3.0
                 Key: SPARK-23801
                 URL: https://issues.apache.org/jira/browse/SPARK-23801
             Project: Spark
          Issue Type: Bug
          Components: Spark Core, SQL
    Affects Versions: 2.3.0
            Reporter: Nathan Kleyn
         Attachments: spark-executor-failure.coredump.log

After upgrading to Spark v2.3.0 from Spark v2.1.1, we are seeing consistent segfaults in a
large Spark job (18 * r3.4xlarge 16 core boxes with 105G of executor memory). I've attached
the full coredump but here is an except:


{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f1467427fdc, pid=1315, tid=0x00007f1464f2d700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 1.8.0_161-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode linux-amd64 )
# Problematic frame:
# V  [libjvm.so+0x995fdc]  oopDesc* PSPromotionManager::copy_to_survivor_space<false>(oopDesc*)+0x7c
#
# Core dump written. Default location: /var/lib/mesos/slave/slaves/92f50385-a83b-4f36-b1a3-53d9b8716544-S203/frameworks/92f50385-a83b-4f36-b1a3-53d9b8716544-0095/executors/14/runs/2e6b3a6e-b811-47d1-9393-66301d923b98/spark-2.3.0-bin-hadoop2.7/core
or core.1315
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#{code}
{code:java}
---------------  T H R E A D  ---------------

Current thread (0x00007f146005b000):  GCTaskThread [stack: 0x00007f1464e2d000,0x00007f1464f2e000]
[id=1363]

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

Registers:
RAX=0x17e907feccbc6d20, RBX=0x00007ef9c035f8c8, RCX=0x00007f1464f2c9f0, RDX=0x0000000000000000
RSP=0x00007f1464f2c1a0, RBP=0x00007f1464f2c210, RSI=0x0000000000000068, RDI=0x00007ef7bc30bda8
R8 =0x00007f1464f2c3d0, R9 =0x0000000000001741, R10=0x00007f1467a52819, R11=0x00007f14671240e0
R12=0x00007f130912c998, R13=0x17e907feccbc6d20, R14=0x0000000000000002, R15=0x000000000000000d
RIP=0x00007f1467427fdc, EFLAGS=0x0000000000010202, CSGSFS=0x002b000000000033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Top of Stack: (sp=0x00007f1464f2c1a0)
0x00007f1464f2c1a0:   00007f146005b000 0000000000000001
0x00007f1464f2c1b0:   0000000000000004 00007f14600bb640
0x00007f1464f2c1c0:   00007f1464f2c210 00007f14673aeed6
0x00007f1464f2c1d0:   00007f1464f2c2c0 00007f1464f2c250
0x00007f1464f2c1e0:   00007f11bde31b70 00007ef9c035f8c8
0x00007f1464f2c1f0:   00007ef8a80a7060 0000000000001741
0x00007f1464f2c200:   0000000000000002 00000000ffffffff
0x00007f1464f2c210:   00007f1464f2c230 00007f146742b005
0x00007f1464f2c220:   00007ef8a80a7050 0000000000001741
0x00007f1464f2c230:   00007f1464f2c2d0 00007f14673ae9fb
0x00007f1464f2c240:   00007f1467a5d880 00007f14673ad9a0
0x00007f1464f2c250:   00007f1464f2c9f0 00007f1464f2c3d0
0x00007f1464f2c260:   00007f1464f2c3a0 00007f146005b620
0x00007f1464f2c270:   00007ef8b843d7c8 ffff000200000006
0x00007f1464f2c280:   00007f1464f2c340 00007f14600bb640
0x00007f1464f2c290:   17417f1453fb9cec 00007f1453fbffff
0x00007f1464f2c2a0:   00007f1453fb819e 00007f1464f2c3a0
0x00007f1464f2c2b0:   0000000000000001 0000000000000000
0x00007f1464f2c2c0:   00007f1464f2c3d0 00007f1464f2c9d0
0x00007f1464f2c2d0:   00007f1464f2c340 00007f1467025f22
0x00007f1464f2c2e0:   00007f145427cb5c 00007f1464f2c3a0
0x00007f1464f2c2f0:   00007f1464f2c370 00007f146005b000
0x00007f1464f2c300:   00007f1464f2c9f0 00007ef850009800
0x00007f1464f2c310:   00007f1464f2c9f0 00007f1464f2c3a0
0x00007f1464f2c320:   00007f1464f2c3d0 00007f146005b000
0x00007f1464f2c330:   00007f1464f2c9f0 00007ef850009800
0x00007f1464f2c340:   00007f1464f2c9c0 00007f1467508191
0x00007f1464f2c350:   00007ef9c16f7890 00007f1464f2c370
0x00007f1464f2c360:   00007f1464f2c9d0 0000000000000000
0x00007f1464f2c370:   00007ef9c035f8c0 00007f145427cb5c
0x00007f1464f2c380:   00007f145427ba90 00007ef900000000
0x00007f1464f2c390:   0000000000000078 00007ef9c035f8c0 

Instructions: (pc=0x00007f1467427fdc)
0x00007f1467427fbc:   01 0f 85 f5 00 00 00 89 f0 c1 f8 03 41 f6 c5 01
0x00007f1467427fcc:   4c 63 f8 0f 85 04 01 00 00 4c 89 e8 48 83 e0 fd
0x00007f1467427fdc:   48 8b 00 48 c1 e8 03 89 c2 48 8b 05 04 74 5e 00
0x00007f1467427fec:   83 e2 0f 3b 10 0f 82 fd 00 00 00 48 8b 45 a8 4e 

Register to memory mapping:

RAX=0x17e907feccbc6d20 is an unknown value
RBX=0x00007ef9c035f8c8 is pointing into the stack for thread: 0x00007ef850009800
RCX=0x00007f1464f2c9f0 is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x00007f1464f2c1a0 is an unknown value
RBP=0x00007f1464f2c210 is an unknown value
RSI=0x0000000000000068 is an unknown value
RDI=0x00007ef7bc30bda8 is pointing into metadata
R8 =0x00007f1464f2c3d0 is an unknown value
R9 =0x0000000000001741 is an unknown value
R10=0x00007f1467a52819: <offset 0xfc0819> in /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
at 0x00007f1466a92000
R11=0x00007f14671240e0: <offset 0x6920e0> in /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
at 0x00007f1466a92000
R12=0x00007f130912c998 is an oop
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage50

 - klass: 'org/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIteratorForCodegenStage50'
R13=0x17e907feccbc6d20 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x000000000000000d is an unknown value


Stack: [0x00007f1464e2d000,0x00007f1464f2e000],  sp=0x00007f1464f2c1a0,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x995fdc]  oopDesc* PSPromotionManager::copy_to_survivor_space<false>(oopDesc*)+0x7c
V  [libjvm.so+0x999005]  PSRootsClosure<false>::do_oop(oopDesc**)+0x35
V  [libjvm.so+0x91c9fb]  OopMapSet::all_do(frame const*, RegisterMap const*, OopClosure*,
void (*)(oopDesc**, oopDesc**), OopClosure*)+0x2fb
V  [libjvm.so+0x593f22]  frame::oops_do_internal(OopClosure*, CLDClosure*, CodeBlobClosure*,
RegisterMap*, bool)+0xa2
V  [libjvm.so+0xa76191]  JavaThread::oops_do(OopClosure*, CLDClosure*, CodeBlobClosure*)+0x161
V  [libjvm.so+0x99926f]  ThreadRootsTask::do_it(GCTaskManager*, unsigned int)+0x6f
V  [libjvm.so+0x5dbfef]  GCTaskThread::run()+0x12f
V  [libjvm.so+0x92da28]  java_start(Thread*)+0x108

JavaThread 0x00007ef850009800 (nid = 1558) was being processed
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 2336  sun.misc.Unsafe.putLong(Ljava/lang/Object;JJ)V (0 bytes) @ 0x00007f14518c70cc [0x00007f14518c7080+0x4c]
J 20102 C2 org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage50.processNext()V
(1030 bytes) @ 0x00007f145427cb5c [0x00007f145427c020+0xb3c]
J 9304 C2 scala.collection.Iterator$$anon$11.hasNext()Z (10 bytes) @ 0x00007f145280da10 [0x00007f145280d460+0x5b0]
J 15346 C2 org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(Lscala/collection/Iterator;)V
(117 bytes) @ 0x00007f145227172c [0x00007f1452271680+0xac]
J 16755 C1 org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Lorg/apache/spark/scheduler/MapStatus;
(293 bytes) @ 0x00007f14534a1dbc [0x00007f145349f820+0x259c]
J 16754 C1 org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Ljava/lang/Object;
(6 bytes) @ 0x00007f14536cf5cc [0x00007f14536cf540+0x8c]
J 15858 C1 org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;)Ljava/lang/Object;
(399 bytes) @ 0x00007f1452eccd44 [0x00007f1452eca8a0+0x24a4]
J 16786 C1 org.apache.spark.executor.Executor$TaskRunner.run()V (2984 bytes) @ 0x00007f1453a4c97c
[0x00007f1453a495e0+0x339c]
J 18919 C1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(225 bytes) @ 0x00007f1453fb91cc [0x00007f1453fb81c0+0x100c]
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub{code}
Unfortunately, this job is so large that it's pretty impossible for us to narrow down to a
reproducible test case. What I can say though is that:
 * We are running on Mesos using fine grained scheduling.
 * We can make it fail every time, consistently.
 * It only happened after we upgraded to v2.3.0.
 * All inputs and options to the job are _exactly_ the same before as after.

Please let me know if we can provide any other information!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message