From user-return-12006-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Mon Jul 29 20:14:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7AF4518063F for ; Mon, 29 Jul 2019 22:14:04 +0200 (CEST) Received: (qmail 59255 invoked by uid 500); 29 Jul 2019 20:14:02 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 59243 invoked by uid 99); 29 Jul 2019 20:14:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jul 2019 20:14:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 39208C04F3 for ; Mon, 29 Jul 2019 20:14:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.301 X-Spam-Level: ** X-Spam-Status: No, score=2.301 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_EXCESS_BASE64=0.001, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id TOpsiY_G6c9I for ; Mon, 29 Jul 2019 20:13:58 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::730; helo=mail-qk1-x730.google.com; envelope-from=jornfranke@gmail.com; receiver= Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 1D7417DC04 for ; Mon, 29 Jul 2019 20:13:58 +0000 (UTC) Received: by mail-qk1-x730.google.com with SMTP id d79so44917821qke.11 for ; Mon, 29 Jul 2019 13:13:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=u7/YpHfZJaKRj3UxPPYjMeDcSwIDsizmbiu2aaLKBzc=; b=JTDfzoEgw64G6A2PZPQn8ey0HarhhQ2SHnfIiUsM3E63IcqNF4KJYb2WxS+Npbdayb Xu/BZvCwLo1omxrvXnYhnSPGryFQgHNruNv8Fq9cjqselGigvywBW8wQYrucvvCNQ9Ac hE3aJtpih43QfJhuilp1GEpavWPdXxdJK3d1AuHuAFV9BaowXvHgnU6MqIEDI9kmEAe4 0Zijl7trIv6SXvL4Xl1H9aghYHwoJeAMV8ffzru1PRy2/Uw9QmjfBl22nZQf68eji3vs psWkqMjZsyZJJ/QvKSgq8y9iI3I8jZ3ylPRS6GRez63AhUdg81b3eQMyM7bgDdgqGY2s my/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=u7/YpHfZJaKRj3UxPPYjMeDcSwIDsizmbiu2aaLKBzc=; b=BugLn+KvnbnyGyeR2fEtI2pq1GAcRG3IULDWh966N3t0+ccSLW2Ah4YxgoveuzTcuC pO4B2rnVsXu3i39dVn1numrVJBa+X7dNWTeOvkXkyGMaPU9klxBEsLHAXJ6V53rYL8bO oxGYIW/EWI5KT+FU1qZx6f8o/z8724BeOtLbTGpjtZ8m381+xqDwtciMmwTUefjG/aal vfR3egwh8E4L9llZ4+CvMb4JwjejnSJcGO1AH0befR7worPXNxpiOOc3PsTs6PlKNXkA w7NezIOhA63tJ8LpSRAelkiQUyJQyTowCc0x02PJGwX7epeRxgCB6w2cJqy32qGG2e1m xXxA== X-Gm-Message-State: APjAAAXP7cNylMCWb760OKJuwhf6d4kSKp0WmJCkXg/Asf4ScKtDqdHS 75cGBlDmwM5uqMmZBlfqS/vaqFJyXcEfk0NYRpvGjg== X-Google-Smtp-Source: APXvYqx8pfrBTYMlO+MsjXwBBRNUV2vsyBr9Sht56E9K5h7BJsJLxrRpBqNGYSDrYHB5/lWP/vPiwYXAJZ3+sKb6uMI= X-Received: by 2002:a37:7643:: with SMTP id r64mr72114817qkc.467.1564431236714; Mon, 29 Jul 2019 13:13:56 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?SsO2cm4gRnJhbmtl?= Date: Mon, 29 Jul 2019 22:13:45 +0200 Message-ID: Subject: Re: Issue migrating from Zookeeper 3.4.14 to 3.5.5 To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="00000000000023d8e9058ed788b5" --00000000000023d8e9058ed788b5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable (of course i do not run them at the same time) On Mon, Jul 29, 2019 at 10:10 PM J=C3=B6rn Franke wr= ote: > thank you for the quick reply. They read from the same disk paths and hav= e > the same access rights (in fact the RHEL service executes them as the sam= e > specific user). > > On Mon, Jul 29, 2019 at 10:09 PM Enrico Olivelli > wrote: > >> Il lun 29 lug 2019, 21:50 J=C3=B6rn Franke ha scr= itto: >> >> > Hi, >> > >> > I tried to migrate a lab environment from Zookeepr 3.4.14 (used for >> Solr) >> > to 3.5.5 and encountered an issue. It is ZooKeeper in standalone mode >> > (other environments have a proper ensemble). I increased jute.maxbuffe= r >> > beyond the default (but not excessively) - this was working perfectly >> fine >> > in 3.4.14. >> > >> > Basically I reuse for the migration the same config files, except that= I >> > whitelist some commands (later I am also interested in adding SSL). >> > >> > I have the following error message when starting Zookeeper with 3.5.5 >> > (basically, I just changed the symboling link from zookeeper to point = to >> > 3.5.5 instead of the 3.4.14 directory: >> > 2019-07-29 15:16:25,217 [myid:] - DEBUG >> > [main:FileTxnLog$FileTxnIterator@655] >> > - Created new input stream /zookeeper/version-2/log.b34 >> > 2019-07-29 15:16:25,217 [myid:] - DEBUG >> > [main:FileTxnLog$FileTxnIterator@658] >> > - Created new input archive /zookeeper/version-2/log.b34 >> > 2019-07-29 15:16:25,222 [myid:] - DEBUG >> > [main:FileTxnLog$FileTxnIterator@696] >> > - EOF exception java.io.EOFException: Failed to read >> > /zookeeper/version-2/log.b34 >> > 2019-07-29 15:16:25,223 [myid:] - DEBUG >> > [main:FileTxnLog$FileTxnIterator@655] >> > - Created new input stream /zookeeper/version-2/log.b72 >> > 2019-07-29 15:16:25,223 [myid:] - DEBUG >> > [main:FileTxnLog$FileTxnIterator@658] >> > - Created new input archive /zookeeper/version-2/log.b72 >> > 2019-07-29 15:16:25,224 [myid:] - DEBUG >> > [main:FileTxnLog$FileTxnIterator@696] >> > - EOF exception java.io.EOFException: Failed to read >> > /zookeeper/version-2/log.b72 >> > 2019-07-29 15:16:25,224 [myid:] - ERROR [main:ZooKeeperServerMain@83] = - >> > Unexpected exception, exiting abnormally >> > java.io.IOException: No snapshot found, but there are log entries. >> > Something is broken! >> > at >> > >> > >> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSn= apLog.java:211) >> > at >> > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:24= 0) >> > at >> > >> > >> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.jav= a:290) >> > at >> > >> > >> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.ja= va:450) >> > at >> > >> > >> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFa= ctory.java:764) >> > at >> > >> > >> org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.= java:98) >> > at >> > >> > >> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperS= erverMain.java:144) >> > at >> > >> > >> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeep= erServerMain.java:106) >> > at >> > >> > >> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain= .java:64) >> > at >> > >> > >> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(Quoru= mPeerMain.java:128) >> > at >> > >> > >> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.ja= va:82) >> > >> > Strangely enough, if I switch back to 3.4.14 the issue is resolved and >> > Zookeeper works normally. However, I would like to leverage the new >> version >> > 3.5.5. >> > >> > There are no 0 bytes files. Disk space is plenty available. >> > >> >> >> Can you compare these logs with logs of 3.4.x ? Are they reading from t= he >> same disk paths? >> >> >> >> > Any idea beyond erasing the data dir (I would try to avoid it, I can >> > reconstruct it, but still)? I will try also in the other environments >> and >> > also with an environment with an ensemble, but i would like to know >> before >> > what the issue could be. >> > >> > Not sure if it is relevant, but: >> > Activated Kerberos Authentication and Kerberos SSL for clients and >> quorum. >> > >> >> Quorum? In standalone mode there is no 'quorum' auth >> >> Enrico >> >> > >> > --00000000000023d8e9058ed788b5--