From dev-return-100666-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Thu Dec 27 17:00:07 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5909B18077A for ; Thu, 27 Dec 2018 17:00:06 +0100 (CET) Received: (qmail 96420 invoked by uid 500); 27 Dec 2018 16:00:05 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 96409 invoked by uid 99); 27 Dec 2018 16:00:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Dec 2018 16:00:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B199B1808A3 for ; Thu, 27 Dec 2018 16:00:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id zOLvNlqzIhU1 for ; Thu, 27 Dec 2018 16:00:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id B97635FE03 for ; Thu, 27 Dec 2018 16:00:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 1B99FE00CA for ; Thu, 27 Dec 2018 16:00:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 6981A2536E for ; Thu, 27 Dec 2018 16:00:00 +0000 (UTC) Date: Thu, 27 Dec 2018 16:00:00 +0000 (UTC) From: "Ismael Juma (JIRA)" To: dev@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (KAFKA-7625) Kafka Broker node JVM crash - kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleEndTransaction MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KAFKA-7625?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-7625. -------------------------------- Resolution: Not A Bug Closing since it's not a Kafka issue. Someone else reported that changing t= he GC from G1 also made the issue go away. In any case, upgrading to the la= test version of JDK 8 is the recommended path. > Kafka Broker node JVM crash - kafka.coordinator.transaction.TransactionCo= ordinator.$anonfun$handleEndTransaction > -------------------------------------------------------------------------= --------------------------------------- > > Key: KAFKA-7625 > URL: https://issues.apache.org/jira/browse/KAFKA-7625 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 2.0.0 > Environment: environment:os.version=3D2.6.32-754.2.1.el6.x86_64= =20 > java.version=3D1.8.0_92=20 > environment:zookeeper.version=3D3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7= 487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper) > Kafka commitId : 3402a8361b734732=20 > Reporter: Sebastian Puzo=C5=84 > Priority: Critical > Attachments: hs_err_pid10238.log, hs_err_pid15119.log, hs_err_pid= 19131.log, hs_err_pid19405.log, hs_err_pid20124.log, hs_err_pid22373.log, h= s_err_pid22386.log, hs_err_pid22633.log, hs_err_pid24681.log, hs_err_pid255= 13.log, hs_err_pid25701.log, hs_err_pid26844.log, hs_err_pid27156.log, hs_e= rr_pid27290.log, hs_err_pid4194.log, hs_err_pid4299.log > > > I observe broker node JVM crashes with same problematic frame: > {code:java} > # > # A fatal error has been detected by the Java Runtime Environment: > # > #=C2=A0 SIGSEGV (0xb) at pc=3D0x00007ff4a2588261, pid=3D24681, tid=3D0x00= 007ff3b9bb1700 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_= 92-b14) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-= amd64 compressed oops) > # Problematic frame: > # J 9736 C1 kafka.coordinator.transaction.TransactionCoordinator.$anonfun= $handleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinat= or;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;= Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either; (51= 8 bytes) @ 0x00007ff4a2588261 [0x00007ff4a25871a0+0x10c1] > # > # Failed to write core dump. Core dumps have been disabled. To enable cor= e dumping, try "ulimit -c unlimited" before starting Java again > # > # If you would like to submit a bug report, please visit: > #=C2=A0=C2=A0 http://bugreport.java.com/bugreport/crash.jsp > # > ---------------=C2=A0 T H R E A D=C2=A0 --------------- > Current thread (0x00007ff4b356f800):=C2=A0 JavaThread "kafka-request-hand= ler-3" daemon [_thread_in_Java, id=3D24781, stack(0x00007ff3b9ab1000,0x0000= 7ff3b9bb2000)] > {code} > {code:java} > Stack: [0x00007ff3b9ab1000,0x00007ff3b9bb2000],=C2=A0 sp=3D0x00007ff3b9ba= fca0,=C2=A0 free space=3D1019k > Native frames: (J=3Dcompiled Java code, j=3Dinterpreted, Vv=3DVM code, C= =3Dnative code) > J 9736 C1 kafka.coordinator.transaction.TransactionCoordinator.$anonfun$h= andleEndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator= ;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;Lk= afka/coordinator/transaction/TransactionMetadata;)Lscala/util/Either; (518 = bytes) @ 0x00007ff4a2588261 [0x00007ff4a25871a0+0x10c1] > J 10456 C2 kafka.coordinator.transaction.TransactionCoordinator.$anonfun$= handleEndTransaction$5(Lkafka/coordinator/transaction/TransactionCoordinato= r;Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResult;I= Lscala/Option;)Lscala/util/Either; (192 bytes) @ 0x00007ff4a1d413f0 [0x0000= 7ff4a1d41240+0x1b0] > J 9303 C1 kafka.coordinator.transaction.TransactionCoordinator$$Lambda$11= 07.apply(Ljava/lang/Object;)Ljava/lang/Object; (32 bytes) @ 0x00007ff4a245f= 55c [0x00007ff4a245f3c0+0x19c] > J 10018 C2 scala.util.Either$RightProjection.flatMap(Lscala/Function1;)Ls= cala/util/Either; (43 bytes) @ 0x00007ff4a1f242c4 [0x00007ff4a1f24260+0x64] > J 9644 C1 kafka.coordinator.transaction.TransactionCoordinator.sendTxnMar= kersCallback$1(Lorg/apache/kafka/common/protocol/Errors;Ljava/lang/String;J= SLorg/apache/kafka/common/requests/TransactionResult;Lscala/Function1;ILkaf= ka/coordinator/transaction/TxnTransitMetadata;)V (251 bytes) @ 0x00007ff4a1= ef6254 [0x00007ff4a1ef5120+0x1134] > J 9302 C1 kafka.coordinator.transaction.TransactionCoordinator$$Lambda$11= 06.apply(Ljava/lang/Object;)Ljava/lang/Object; (40 bytes) @ 0x00007ff4a2474= 7ec [0x00007ff4a24745a0+0x24c] > J 10125 C2 kafka.coordinator.transaction.TransactionStateManager.updateCa= cheCallback$1(Lscala/collection/Map;Ljava/lang/String;ILkafka/coordinator/t= ransaction/TxnTransitMetadata;Lscala/Function1;Lscala/Function1;Lorg/apache= /kafka/common/TopicPartition;)V (892 bytes) @ 0x00007ff4a27045ec [0x00007ff= 4a2703c60+0x98c] > J 10051 C2 kafka.coordinator.transaction.TransactionStateManager$$Lambda$= 814.apply(Ljava/lang/Object;)Ljava/lang/Object; (36 bytes) @ 0x00007ff4a1a9= cd08 [0x00007ff4a1a9cc80+0x88] > J 9349 C2 kafka.server.DelayedProduce.tryComplete()Z (52 bytes) @ 0x00007= ff4a1e46e5c [0x00007ff4a1e46980+0x4dc] > J 10111 C2 kafka.server.DelayedOperationPurgatory.tryCompleteElseWatch(Lk= afka/server/DelayedOperation;Lscala/collection/Seq;)Z (147 bytes) @ 0x00007= ff4a1c6e000 [0x00007ff4a1c6df20+0xe0] > J 10448 C2 kafka.server.ReplicaManager.appendRecords(JSZZLscala/collectio= n/Map;Lscala/Function1;Lscala/Option;Lscala/Function1;)V (237 bytes) @ 0x00= 007ff4a2340b6c [0x00007ff4a233f3e0+0x178c] > J 10050 C2 kafka.coordinator.transaction.TransactionStateManager.$anonfun= $appendTransactionToLog$13(Lkafka/coordinator/transaction/TransactionStateM= anager;Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransitMetadata= ;Lscala/Function1;Lscala/Function1;Lorg/apache/kafka/common/TopicPartition;= Lscala/collection/immutable/Map;)V (294 bytes) @ 0x00007ff4a25a2384 [0x0000= 7ff4a25a1800+0xb84] > J 10112 C2 kafka.coordinator.transaction.TransactionStateManager$$Lambda$= 812.apply$mcV$sp()V (36 bytes) @ 0x00007ff4a1b24e64 [0x00007ff4a1b24de0+0x8= 4] > J 4800 C2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object;= (10 bytes) @ 0x00007ff4a1ad73dc [0x00007ff4a1ad73a0+0x3c] > J 6365 C2 kafka.utils.CoreUtils$.inReadLock(Ljava/util/concurrent/locks/R= eadWriteLock;Lscala/Function0;)Ljava/lang/Object; (12 bytes) @ 0x00007ff4a1= 3c7794 [0x00007ff4a13c7720+0x74] > J 10037 C2 kafka.coordinator.transaction.TransactionStateManager.appendTr= ansactionToLog(Ljava/lang/String;ILkafka/coordinator/transaction/TxnTransit= Metadata;Lscala/Function1;Lscala/Function1;)V (152 bytes) @ 0x00007ff4a26cd= 90c [0x00007ff4a26c9f40+0x39cc] > J 10447 C2 kafka.server.KafkaApis.handleEndTxnRequest(Lkafka/network/Requ= estChannel$Request;)V (118 bytes) @ 0x00007ff4a1fbb154 [0x00007ff4a1fb9fa0+= 0x11b4] > J 6316 C2 kafka.server.KafkaApis.handle(Lkafka/network/RequestChannel$Req= uest;)V (1154 bytes) @ 0x00007ff4a19538c0 [0x00007ff4a1953120+0x7a0] > J 8822% C2 kafka.server.KafkaRequestHandler.run()V (258 bytes) @ 0x00007f= f4a1d43bc4 [0x00007ff4a1d43a60+0x164] > j=C2=A0 java.lang.Thread.run()V+11 > v=C2=A0 ~StubRoutines::call_stub > {code} > =C2=A0 > =C2=A0Attached crash reports: > =C2=A0 > /apps/solr/kafka/kafka_2.12-2.0.0 $ ls -lth hs_err_pid* > -rwxrwxrwx 1 res2 res 424K Nov 14 14:02 hs_err_pid24681.log > -rwxrwxrwx 1 res2 res 427K Nov 13 05:18 hs_err_pid4194.log > -rwxrwxrwx 1 res2 res 423K Nov 12 17:06 hs_err_pid15119.log > -rwxrwxrwx 1 res2 res 425K Nov 12 04:56 hs_err_pid4299.log > -rwxrwxrwx 1 res2 res 423K Nov 11 16:32 hs_err_pid25701.log > -rwxrwxrwx 1 res2 res 424K Nov 11 03:44 hs_err_pid22373.log > -rwxrwxrwx 1 res2 res 426K Nov 10 15:24 hs_err_pid25513.log > -rwxrwxrwx 1 res2 res 421K Nov 10 02:16 hs_err_pid10238.log > -rwxrwxrwx 1 res2 res 435K Nov=C2=A0 8 11:45 hs_err_pid22386.log > -rwxrwxrwx 1 res2 res 417K Nov=C2=A0 7 20:38 hs_err_pid26844.log > -rwxrwxrwx 1 res2 res 419K Nov=C2=A0 7 08:04 hs_err_pid27290.log > -rwxrwxrwx 1 res2 res 419K Nov=C2=A0 6 17:37 hs_err_pid19131.log > -rwxrwxrwx 1 res2 res 418K Nov=C2=A0 6 03:50 hs_err_pid22633.log > =C2=A0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)