zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kuldeep singh <kuldeep.sing...@gmail.com>
Subject Re: upgrade from 3.4.5 to 3.5.6
Date Wed, 01 Jul 2020 07:47:28 GMT
Hi,

we have done below changes in java.env file

export SERVER_JVMFLAGS="

-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
-Dzookeeper.ssl.keyStore.password=testpass
-Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
-Dzookeeper.ssl.trustStore.password=testpass"



export CLIENT_JVMFLAGS="

-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
-Dzookeeper.ssl.keyStore.password=testpass
-Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
-Dzookeeper.ssl.trustStore.password=testpass"

I have started the ZK server and it is up without any issue.

But now when I login to ZkCli then it gives the below error.

WatchedEvent state:AuthFailed type:None path:null


Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN
[nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider
for scheme: ztpasswd has x509 ip digest*

Please help me on this issue

Thanks,
---------------------
Kuldeep Singh Budania
Software Architect


On Wed, Jul 1, 2020 at 12:05 PM kuldeep singh <kuldeep.singh08@gmail.com>
wrote:

> Hi,
>
> My ZK server  is up and running in secure mode, But When I am trying to
> connect to the ZK server using ZKCli, it gives the below error.
>
> WatchedEvent state:AuthFailed type:None path:null
>
>
> Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN
> [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider
> for scheme: ztpasswd has x509 ip digest*
>
> Can someone please help me on this issue. we are using the 3.5.6 version.
>
> I appreciate if I will get a response as soon as possible, as I am stuck
> at this point.
>
> Thanks,
> ---------------------
> Kuldeep Singh Budania
> Software Architect
>
>
>
> On Thu, Jun 25, 2020 at 11:54 AM Enrico Olivelli - Diennea
> <enrico.olivelli@diennea.com.invalid> wrote:
>
>> I mean in zoo.cfg
>> Not as a system property
>>
>> Enrico
>>
>> Il giorno 25/06/20, 08:19 "Enrico Olivelli - Diennea" <
>> enrico.olivelli@diennea.com.INVALID> ha scritto:
>>
>>     Hi
>>     You have to enable Netty on the server side
>>
>>     Something like:
>>     serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>>
>>     Hope that helps
>>     Enrico
>>
>>     Il giorno 24/06/20, 19:17 "kuldeep singh" <kuldeep.singh08@gmail.com>
>> ha scritto:
>>
>>         Hi,
>>
>>         I got below error while setting SSL properties in zkEnv.sh
>>
>>
>>
>>
>> ==========================================================================================
>>
>>         2020-06-24 15:49:35,864 - INFO  [main:QuorumPeerConfig@133] -
>> Reading
>>         configuration from: /etc/zookeeper/zoo.cfg
>>
>>         2020-06-24 15:49:35,874 - INFO  [main:QuorumPeerConfig@385] -
>>         clientPortAddress is 0.0.0.0/0.0.0.0:10181
>>
>>         2020-06-24 15:49:35,874 - INFO  [main:QuorumPeerConfig@399] -
>>         secureClientPortAddress is 0.0.0.0/0.0.0.0:2281
>>
>>         2020-06-24 15:49:35,878 - INFO  [main:X509Util@79] - Setting -D
>>         jdk.tls.rejectClientInitiatedRenegotiation=true to disable
>> client-initiated
>>         TLS renegotiation
>>
>>         2020-06-24 15:49:35,897 - INFO  [main:DatadirCleanupManager@78] -
>>         autopurge.snapRetainCount set to 3
>>
>>         2020-06-24 15:49:35,897 - INFO  [main:DatadirCleanupManager@79] -
>>         autopurge.purgeInterval set to 1
>>
>>         2020-06-24 15:49:35,898 - INFO  [
>>         PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
>> started.
>>
>>         2020-06-24 15:49:35,899 - INFO  [main:ManagedUtil@46] - Log4j
>> found with
>>         jmx enabled.
>>
>>         2020-06-24 15:49:35,903 - INFO  [PurgeTask:FileTxnSnapLog@103] -
>>         zookeeper.snapshot.trust.empty : false
>>
>>         2020-06-24 15:49:35,910 - INFO  [
>>         PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
>> completed.
>>
>>         2020-06-24 15:49:35,975 - INFO  [main:QuorumPeerMain@141] -
>> Starting quorum
>>         peer
>>
>>         2020-06-24 15:49:35,983 - INFO  [main:ServerCnxnFactory@135] -
>> Using
>>         org.apache.zookeeper.server.NIOServerCnxnFactory as server
>> connection
>>         factory
>>
>>         2020-06-24 15:49:35,986 - INFO  [main:NIOServerCnxnFactory@673] -
>>         Configuring NIO connection handler with 10s sessionless
>> connection timeout,
>>         2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
>>
>>         2020-06-24 15:49:35,992 - INFO  [main:NIOServerCnxnFactory@686]
>> - binding
>>         to port 0.0.0.0/0.0.0.0:10181
>>
>>         2020-06-24 15:49:35,994 - INFO  [main:ServerCnxnFactory@135] -
>> Using
>>         org.apache.zookeeper.server.NIOServerCnxnFactory as server
>> connection
>>         factory
>>
>>         2020-06-24 15:49:35,995 - ERROR [main:QuorumPeerMain@101] -
>> Unexpected
>>         exception, exiting abnormally
>>
>>         java.lang.UnsupportedOperationException: SSL isn't supported in
>>         NIOServerCnxn
>>
>>                 at
>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:644)
>>
>>                 at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:155)
>>
>>                 at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
>>
>>                 at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>
>>
>>
>>
>>
>>         I have set the following properties in SERVER_JVMFLAGS in
>> zkEnv.sh file  :
>>
>> "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>>
>> -Dzookeeper.ssl.keyStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkkeystore.p12
>>         -Dzookeeper.ssl.keyStore.password=EvaiKiO1@123456
>>
>> -Dzookeeper.ssl.trustStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkTrustStore.jks
>>         -Dzookeeper.ssl.trustStore.password=EvaiKiO1@123456"
>>
>>         Thanks,
>>         ---------------------
>>         Kuldeep Singh Budania
>>
>>
>>
>>         On Mon, Jun 22, 2020 at 8:08 PM Jordan Zimmerman <
>> jordan@jordanzimmerman.com>
>>         wrote:
>>
>>         > It's the same as the normal ZooKeeper client:
>>         >
>> https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions
>>         > <
>>         >
>> https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions
>>         > >
>>         >
>>         > -Jordan
>>         >
>>         > > On Jun 22, 2020, at 5:50 AM, kuldeep singh <
>> kuldeep.singh08@gmail.com>
>>         > wrote:
>>         > >
>>         > > Hi Team,
>>         > > How we will do secure communication between the Curator
>> framework and
>>         > > zookeeper 3.5.6 ?
>>         > > I didn't get any solution right now.
>>         > > I appreciate it if someone could help me with the same.
>>         > >
>>         > > Thanks,
>>         > > ---------------------
>>         > > Kuldeep Singh Budania
>>         > > Software Architect
>>         > >
>>         > >
>>         > > On Fri, Apr 17, 2020 at 4:53 PM Szalay-Bekő Máté <
>>         > szalay.beko.mate@gmail.com>
>>         > > wrote:
>>         > >
>>         > >> Hello Kuldeep,
>>         > >>
>>         > >> did you download the source from the ZooKeeper webpage, or
>> checked out
>>         > from
>>         > >> git?
>>         > >> Anyway, the following commands should work:
>>         > >>
>>         > >> wget
>>         > >>
>>         > >>
>>         >
>> https://downloads.apache.org/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7.tar.gz
>>         > >> tar xzvf ./apache-zookeeper-3.5.7.tar.gz
>>         > >> cd apache-zookeeper-3.5.7
>>         > >> mvn clean install -DskipTests
>>         > >>
>>         > >> I tested it with OpenJDK 8u424 and maven 3.6.0.
>>         > >>
>>         > >> Kind regards,
>>         > >> Mate
>>         > >>
>>         > >>
>>         > >> On Fri, Apr 17, 2020 at 12:51 PM kuldeep singh <
>>         > kuldeep.singh08@gmail.com>
>>         > >> wrote:
>>         > >>
>>         > >>> Hi All,
>>         > >>>
>>         > >>> I have download the zookeeper source code and run the mvn
>> command for
>>         > >>> packaging but get problem on "zookeeper-server" directory.
>>         > >>>
>>         > >>> *command :- mvn clean package*
>>         > >>>
>>         > >>> [ERROR] Failed to execute goal
>>         > >>> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
>>         > >>> (default-compile) on project zookeeper: Fatal error
>> compiling:
>>         > >>> java.lang.NullPointerException -> [Help 1]
>>         > >>> org.apache.maven.lifecycle.LifecycleExecutionException:
>> Failed to
>>         > execute
>>         > >>> goal
>> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
>>         > >>> (default-compile) on project zookeeper: Fatal error
>> compiling
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
>>         > >>>        at
>>         > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
>>         > >>>        at
>> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
>>         > >>>        at
>> org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
>>         > >>>        at
>> org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
>>         > >>>        at
>> org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
>>         > >>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         > >>>        at java.lang.reflect.Method.invoke(Method.java:498)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>>         > >>>        at
>>         > >>>
>>         > >>
>>         >
>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>>         > >>>        at
>>         > >>>
>>         >
>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>>         > >>> Caused by: org.apache.maven.plugin.MojoExecutionException:
>> Fatal error
>>         > >>> compiling
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1140)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:188)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>>         > >>>        ... 19 more
>>         > >>> Caused by: org.codehaus.plexus.compiler.CompilerException:
>>         > >>> java.lang.NullPointerException
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1129)
>>         > >>>        ... 22 more
>>         > >>> Caused by: java.lang.RuntimeException:
>> java.lang.NullPointerException
>>         > >>>        at
>> com.sun.tools.javac.main.Main.compile(Main.java:559)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:126)
>>         > >>>        ... 24 more
>>         > >>> Caused by: java.lang.NullPointerException
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> com.sun.tools.javac.model.JavacTypes.getOverriddenMethods(JavacTypes.java:306)
>>         > >>>        at com.sun.tools.doclint.Env.setCurrent(Env.java:155)
>>         > >>>        at
>> com.sun.tools.doclint.Checker.scan(Checker.java:144)
>>         > >>>        at
>> com.sun.tools.doclint.DocLint$2.visitDecl(DocLint.java:285)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:359)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:346)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreeScanner.visitNewClass(TreeScanner.java:280)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.tree.JCTree$JCNewClass.accept(JCTree.java:1532)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreeScanner.visitVariable(TreeScanner.java:153)
>>         > >>>        at
>>         > >>>
>>         >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitVariable(DocLint.java:373)
>>         > >>>        at
>>         > >>>
>>         >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitVariable(DocLint.java:346)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.tree.JCTree$JCVariableDecl.accept(JCTree.java:864)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81)
>>         > >>>        at
>> com.sun.source.util.TreeScanner.scan(TreeScanner.java:91)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99)
>>         > >>>        at
>>         > >>
>> com.sun.source.util.TreeScanner.visitClass(TreeScanner.java:133)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:360)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:346)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>>         > >>>        at
>> com.sun.source.util.TreeScanner.scan(TreeScanner.java:91)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99)
>>         > >>>        at
>>         > >>>
>>         > >>
>>         >
>> com.sun.source.util.TreeScanner.visitCompilationUnit(TreeScanner.java:120)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:354)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:346)
>>         > >>>        at
>>         > >>>
>>         >
>> com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:550)
>>         > >>>        at
>>         > >>>
>> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68)
>>         > >>>        at
>> com.sun.tools.doclint.DocLint$3.started(DocLint.java:296)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> com.sun.tools.javac.api.ClientCodeWrapper$WrappedTaskListener.started(ClientCodeWrapper.java:668)
>>         > >>>        at
>>         > >>>
>>         > >>>
>>         > >>
>>         >
>> com.sun.tools.javac.api.MultiTaskListener.started(MultiTaskListener.java:103)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.main.JavaCompiler.attribute(JavaCompiler.java:1240)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.main.JavaCompiler.compile2(JavaCompiler.java:901)
>>         > >>>        at
>>         > >>>
>> com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:860)
>>         > >>>        at
>> com.sun.tools.javac.main.Main.compile(Main.java:523)
>>         > >>>        ... 27 more
>>         > >>> [ERROR]
>>         > >>> [ERROR] Re-run Maven using the -X switch to enable full
>> debug logging.
>>         > >>> [ERROR]
>>         > >>> [ERROR] For more information about the errors and possible
>> solutions,
>>         > >>> please read the following articles:
>>         > >>> [ERROR] [Help 1]
>>         > >>>
>>         >
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>>         > >>>
>>         > >>> Can someone help me on this issue?
>>         > >>>
>>         > >>> Thanks,
>>         > >>> ---------------------
>>         > >>> Kuldeep Singh Budania
>>         > >>>
>>         > >>>
>>         > >>>
>>         > >>> On Sat, Apr 4, 2020 at 5:57 PM Szalay-Bekő Máté <
>>         > >>> szalay.beko.mate@gmail.com>
>>         > >>> wrote:
>>         > >>>
>>         > >>>> these exceptions can mean many things... I think this can
>> be even
>>         > >> normal
>>         > >>>> duding rolling restart (as some connections get broken in
>> this case)
>>         > >>>>
>>         > >>>> However, I saw cases already when exceptions like these
>> killed
>>         > receiver
>>         > >>> or
>>         > >>>> sender threads in QuorumCnxManager / Leader Election in
>> such a way
>>         > that
>>         > >>>> they were not able to recover, so the node was unable to
>> connect to
>>         > any
>>         > >>>> quorum until restart. I remember seeing this in 3.4 too.
>>         > >>>>
>>         > >>>> Do you see these exceptions in the second server (the one
>> which you
>>         > >> just
>>         > >>>> upgraded in step 3)?
>>         > >>>> Is this issue reproducible?
>>         > >>>>
>>         > >>>> What is the tickTime and initLimit you use? Maybe the
>> server just
>>         > >> require
>>         > >>>> more time to sync?
>>         > >>>>
>>         > >>>> I would need more logs to really see what happened. Can
>> you create a
>>         > >> Jira
>>         > >>>> issue and upload the logs and also the ZooKeeper configs?
>> I am happy
>>         > to
>>         > >>>> take a closer look.
>>         > >>>> (if you need to re-run the test to collect the logs, then
>> enabling
>>         > >> DEBUG
>>         > >>>> logs would be great. The INFO level logs are usually
>> enough for these
>>         > >>>> problems, but one can never know...)
>>         > >>>>
>>         > >>>> Kind regards,
>>         > >>>> Mate
>>         > >>>>
>>         > >>>>
>>         > >>>> On Fri, Apr 3, 2020 at 10:05 AM kuldeep singh <
>>         > >> kuldeep.singh08@gmail.com
>>         > >>>>
>>         > >>>> wrote:
>>         > >>>>
>>         > >>>>> Hi Team,
>>         > >>>>>
>>         > >>>>> I have done some POC on rolling upgrade and found below
>> result.
>>         > >>>>>
>>         > >>>>>
>>         > >>>>>   1. On 1st node upgrade zookeeper . Traffic was running
>> fine
>>         > >> because
>>         > >>> 2
>>         > >>>>>   nodes are already on old zookeeper.
>>         > >>>>>   2. On 1st node upgrade our application and didn’t find
>> any issue
>>         > >>>>>   3. On 2nd node upgrade zookeeper but got below error
>> and zookeeper
>>         > >>> is
>>         > >>>>>   not taking any requests
>>         > >>>>>   4.
>>         > >>>>>
>>         > >>>>> java.io.EOFException
>>         > >>>>>
>>         > >>>>>        at
>> java.io.DataInputStream.readInt(DataInputStream.java:392)
>>         > >>>>>
>>         > >>>>>        at
>>         > >>>>>
>>         > >>>>>
>>         > >>>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
>>         > >>>>>
>>         > >>>>> 2020-03-30 14:19:55,587 - WARN
>>         > >>>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@765] -
>> Interrupting
>>         > >>> SendWorker
>>         > >>>>>
>>         > >>>>> 2020-03-30 14:19:55,588 - ERROR [LearnerHandler-/
>> 192.168.44.73:33754
>>         > >>>>> :LearnerHandler@562] - Unexpected exception causing
>> shutdown while
>>         > >>> sock
>>         > >>>>> still open
>>         > >>>>>
>>         > >>>>> java.io.EOFException
>>         > >>>>>
>>         > >>>>>        at
>> java.io.DataInputStream.readInt(DataInputStream.java:392)
>>         > >>>>>
>>         > >>>>>        at
>>         > >>>>>
>>         > >>
>> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
>>         > >>>>>
>>         > >>>>>        at
>>         > >>>>>
>>         > >>>>>
>>         > >>>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
>>         > >>>>>
>>         > >>>>>        at
>>         > >>>>>
>>         > >>>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
>>         > >>>>>
>>         > >>>>>        at
>>         > >>>>>
>>         > >>>>>
>>         > >>>>
>>         > >>>
>>         > >>
>>         >
>> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:476)
>>         > >>>>>
>>         > >>>>> 2020-03-30 14:19:55,588 - WARN
>>         > >>>>> [SendWorker:1:QuorumCnxManager$SendWorker@679] -
>> Interrupted while
>>         > >>>> waiting
>>         > >>>>> for message on queue
>>         > >>>>>
>>         > >>>>> Please let me know is this the known issue or this is
>> different issue
>>         > >>>> which
>>         > >>>>> is mention in Apache zookeeper documentation when
>> upgrading from
>>         > >> 3.4.5
>>         > >>> to
>>         > >>>>> 3.5.6
>>         > >>>>>
>>         > >>>>> Thanks,
>>         > >>>>> ---------------------
>>         > >>>>> Kuldeep Singh Budania
>>         > >>>>> Software Architect
>>         > >>>>>
>>         > >>>>>
>>         > >>>>>
>>         > >>>>> On Sun, Mar 29, 2020 at 9:06 AM Alexander Shraer <
>> shralex@gmail.com>
>>         > >>>>> wrote:
>>         > >>>>>
>>         > >>>>>> +1 to what Mate said (I wrote the quoted instructions).
>>         > >>>>>>
>>         > >>>>>>
>>         > >>>>>>
>>         > >>>>>> On Tue, Mar 24, 2020 at 7:03 AM Szalay-Bekő Máté <
>>         > >>>>>> szalay.beko.mate@gmail.com>
>>         > >>>>>> wrote:
>>         > >>>>>>
>>         > >>>>>>> Hi Kuldeep,
>>         > >>>>>>>
>>         > >>>>>>> I just want to provide you some background info about
>> our
>>         > >>>>> documentation.
>>         > >>>>>>> The reason to upgrade to 3.4.6 first is to avoid the
>> following
>>         > >>> error:
>>         > >>>>>>>
>>         > >>>>>>>> 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/
>>         > >>> 127.0.0.1:2784
>>         > >>>>>>> :QuorumCnxManager@349] - Invalid server id: -65536
>>         > >>>>>>>
>>         > >>>>>>> This error comes because of the protocol changes between
>>         > >> ZooKeeper
>>         > >>>>> server
>>         > >>>>>>> nodes during connection initiation for leader election.
>> In
>>         > >>> ZooKeeper
>>         > >>>>> 3.5
>>         > >>>>>> a
>>         > >>>>>>> protocol version was introduced (see ZOOKEEPER-107) and
>> since
>>         > >> that
>>         > >>>> time
>>         > >>>>>> the
>>         > >>>>>>> fist long value sent in the initial message is not the
>> server ID
>>         > >>> but
>>         > >>>>> the
>>         > >>>>>>> protocol version (-65536). In ZooKeeper 3.4.6 we made
>> the old 3.4
>>         > >>>>>>> ZooKeepers backward compatible, so they are able to
>> parse both
>>         > >> the
>>         > >>>> old
>>         > >>>>>> and
>>         > >>>>>>> the new protocol format (see ZOOKEEPER-1633). This
>> issue happens
>>         > >>> only
>>         > >>>>>> when
>>         > >>>>>>> you need to use old (3.4.0 - 3.4.5) and new (3.5.0+)
>> ZooKeeper
>>         > >>>> servers
>>         > >>>>>>> together in the same cluster. During a rolling upgrade,
>> this is
>>         > >>>> usually
>>         > >>>>>> the
>>         > >>>>>>> case to have old and new ZooKeepers present together.
>>         > >>>>>>>
>>         > >>>>>>> The fact that you haven't seen any issues might be
>> caused by the
>>         > >>>> order
>>         > >>>>> of
>>         > >>>>>>> the servers. In ZooKeeper the connection initiation
>> between the
>>         > >>>> servers
>>         > >>>>>>> during the leader election follows a specific rule. As
>> far as I
>>         > >>>>> remember
>>         > >>>>>>> always the server with the larger ID 'wins the
>> challenge', so it
>>         > >> is
>>         > >>>>>>> possible, that the old server didn't need to parse any
>> initial
>>         > >>>> message
>>         > >>>>>> (if
>>         > >>>>>>> it had the largest ID) and this is why you haven't seen
>> the
>>         > >> issue.
>>         > >>>> Also
>>         > >>>>>>> having 2 nodes up from the 3 nodes cluster still makes
>> the
>>         > >> cluster
>>         > >>>> work
>>         > >>>>>> (so
>>         > >>>>>>> you should also check if all the servers are part of
>> the quorum).
>>         > >>>>>>>
>>         > >>>>>>> I agree with Enrico and Norbert, the safest and most
>> stable way
>>         > >> is
>>         > >>>>>> upgrade
>>         > >>>>>>> first to 3.4.latest, then go to 3.5.latest. Still, if
>> you don't
>>         > >> see
>>         > >>>>> that
>>         > >>>>>>> you would hit this specific issue (e.g. no "Invalid
>> server id" in
>>         > >>> the
>>         > >>>>> log
>>         > >>>>>>> files), and all the three servers can handle traffic,
>> then maybe
>>         > >>> you
>>         > >>>>>> don't
>>         > >>>>>>> need to upgrade first to 3.4.latest, it is your
>> decision.
>>         > >>> Definitely
>>         > >>>>> you
>>         > >>>>>>> should test it first, as suggested by the others.
>>         > >>>>>>>
>>         > >>>>>>> Kind regards,
>>         > >>>>>>> Mate
>>         > >>>>>>>
>>         > >>>>>>> On Tue, Mar 24, 2020 at 12:29 PM Norbert Kalmar
>>         > >>>>>>> <nkalmar@cloudera.com.invalid> wrote:
>>         > >>>>>>>
>>         > >>>>>>>> Hi,
>>         > >>>>>>>>
>>         > >>>>>>>> That guide is to upgrade to 3.5.0, which was an alpha
>> version.
>>         > >> A
>>         > >>>> lot
>>         > >>>>>> has
>>         > >>>>>>>> changed for the first stable release of 3.5.5 and then
>> a few
>>         > >>> more,
>>         > >>>>> even
>>         > >>>>>>>> rolling upgrade issues have been fixed for 3.5.6.
>>         > >>>>>>>> This is a more up-to-date guide:
>>         > >>>>>>>>
>>         > >>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ
>>         > >>>>>>>>
>>         > >>>>>>>> If you have done your testing (with prod snapshot!),
>> then you
>>         > >> can
>>         > >>>>> skip
>>         > >>>>>>> 3.4
>>         > >>>>>>>> latest upgrade, but keep in mind we do our
>> recommendations for
>>         > >> a
>>         > >>>>>> reason.
>>         > >>>>>>>> There were issues reported and/or found during
>> testing. Some
>>         > >> are
>>         > >>>>> fixed
>>         > >>>>>>> with
>>         > >>>>>>>> 3.5.6, some only happens if certain conditions stand
>>         > >>> (IOException:
>>         > >>>> No
>>         > >>>>>>>> snapshot found - mentioned in the guide, fixed in
>> 3.5.6).
>>         > >>>>>>>>
>>         > >>>>>>>> So it is up to you, I would still recommend to do an
>> 3.4
>>         > >> upgrade
>>         > >>>>> first,
>>         > >>>>>>> if
>>         > >>>>>>>> it's feasible.
>>         > >>>>>>>>
>>         > >>>>>>>> Regards,
>>         > >>>>>>>> Norbert
>>         > >>>>>>>>
>>         > >>>>>>>> On Tue, Mar 24, 2020 at 11:45 AM kuldeep singh <
>>         > >>>>>>> kuldeep.singh08@gmail.com>
>>         > >>>>>>>> wrote:
>>         > >>>>>>>>
>>         > >>>>>>>>> Hi,
>>         > >>>>>>>>>
>>         > >>>>>>>>> Current Zookeeper version :- 3.4.5
>>         > >>>>>>>>> Upgraded version                :- 3.5.6
>>         > >>>>>>>>>
>>         > >>>>>>>>> We are not going with 3.5.7. Our final decision is
>> zookeeper
>>         > >>>>> version
>>         > >>>>>> is
>>         > >>>>>>>>> 3.5.6
>>         > >>>>>>>>> as per your reply first we need to move latest
>> version of
>>         > >>> 3.4.x,
>>         > >>>>> like
>>         > >>>>>>>> below
>>         > >>>>>>>>>
>>         > >>>>>>>>> 3.4.5 -> 3.4.14 -> 3.5.6 (Correct me if I am wrong
>> here)
>>         > >>>>>>>>>
>>         > >>>>>>>>> But if We are not facing any problem that i have
>> shared you
>>         > >>> that
>>         > >>>> we
>>         > >>>>>>> have
>>         > >>>>>>>>> set up of 3 node cluster where 2 node are on 3.5.6
>> version
>>         > >> and
>>         > >>> 1
>>         > >>>>> node
>>         > >>>>>>> on
>>         > >>>>>>>>> 3.4.5, Everything is running fine and didn't get any
>> issue,
>>         > >> So
>>         > >>>> what
>>         > >>>>>>> other
>>         > >>>>>>>>> problem we can face if we directly move to 3.5.6
>>         > >>>>>>>>>
>>         > >>>>>>>>> Thanks,
>>         > >>>>>>>>> ---------------------
>>         > >>>>>>>>> Kuldeep Singh Budania
>>         > >>>>>>>>> Software Architect
>>         > >>>>>>>>>
>>         > >>>>>>>>>
>>         > >>>>>>>>> On Tue, Mar 24, 2020 at 3:58 PM Enrico Olivelli <
>>         > >>>>> eolivelli@gmail.com
>>         > >>>>>>>
>>         > >>>>>>>>> wrote:
>>         > >>>>>>>>>
>>         > >>>>>>>>>> Hi
>>         > >>>>>>>>>> You have to upgrade to latest 3.4.x Zookeeper then
>> you will
>>         > >>>>> upgrade
>>         > >>>>>>> to
>>         > >>>>>>>>>> 3.5.7.
>>         > >>>>>>>>>> All should run well without issues
>>         > >>>>>>>>>>
>>         > >>>>>>>>>>
>>         > >>>>>>>>>> Enrico
>>         > >>>>>>>>>>
>>         > >>>>>>>>>> Il Mar 24 Mar 2020, 10:18 kuldeep singh <
>>         > >>>>> kuldeep.singh08@gmail.com
>>         > >>>>>>>
>>         > >>>>>>> ha
>>         > >>>>>>>>>> scritto:
>>         > >>>>>>>>>>
>>         > >>>>>>>>>>> Hi Team,
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> We are upgrading zookeeper from 3.4.5 to 3.5.6. I
>> have
>>         > >> set
>>         > >>>> up 3
>>         > >>>>>>> node
>>         > >>>>>>>>>>> cluster where 2 node are on 3.5.6 version and 1
>> node on
>>         > >>>> 3.4.5.
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> Everything is running fine and didn't get any issue
>> on my
>>         > >>>>> system.
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> but I found something on apache site  that first we
>> need
>>         > >> to
>>         > >>>>>> upgrade
>>         > >>>>>>>> on
>>         > >>>>>>>>>>> 3.4.6 than we can upgrade to 3.5.6. So is it
>> mandatory
>>         > >> to
>>         > >>> go
>>         > >>>>> on
>>         > >>>>>>>> 3.4.6
>>         > >>>>>>>>>>> first.
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> *Upgrading to 3.5.0*
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> Upgrading a running ZooKeeper ensemble to 3.5.0
>> should be
>>         > >>>> done
>>         > >>>>>> only
>>         > >>>>>>>>> after
>>         > >>>>>>>>>>> upgrading your ensemble to the 3.4.6 release. Note
>> that
>>         > >>> this
>>         > >>>> is
>>         > >>>>>>> only
>>         > >>>>>>>>>>> necessary for rolling upgrades (if you're fine with
>>         > >>> shutting
>>         > >>>>> down
>>         > >>>>>>> the
>>         > >>>>>>>>>>> system completely, you don't have to go through
>> 3.4.6).
>>         > >> If
>>         > >>>> you
>>         > >>>>>>>> attempt
>>         > >>>>>>>>> a
>>         > >>>>>>>>>>> rolling upgrade without going through 3.4.6 (for
>> example
>>         > >>> from
>>         > >>>>>>> 3.4.5),
>>         > >>>>>>>>> you
>>         > >>>>>>>>>>> may get the following error:
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> 2013-01-30 11:32:10,663 [myid:2] - INFO [localhost/
>>         > >>>>>> 127.0.0.1:2784
>>         > >>>>>>>>>>> :QuorumCnxManager$Listener@498] - Received
>> connection
>>         > >>>> request
>>         > >>>>> /
>>         > >>>>>>>>>>> 127.0.0.1:60876
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/
>>         > >>>>>> 127.0.0.1:2784
>>         > >>>>>>>>>>> :QuorumCnxManager@349] - Invalid server id: -65536
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> During a rolling upgrade, each server is taken down
>> in
>>         > >> turn
>>         > >>>> and
>>         > >>>>>>>>> rebooted
>>         > >>>>>>>>>>> with the new 3.5.0 binaries. Before starting the
>> server
>>         > >>> with
>>         > >>>>>> 3.5.0
>>         > >>>>>>>>>>> binaries, we highly recommend updating the
>> configuration
>>         > >>> file
>>         > >>>>> so
>>         > >>>>>>> that
>>         > >>>>>>>>> all
>>         > >>>>>>>>>>> server statements "server.x=..." contain client
>> ports
>>         > >> (see
>>         > >>>> the
>>         > >>>>>>>> section
>>         > >>>>>>>>>>> Specifying
>>         > >>>>>>>>>>> the client port). As explained earlier you may
>> leave the
>>         > >>>>>>>> configuration
>>         > >>>>>>>>>> in a
>>         > >>>>>>>>>>> single file, as well as leave the
>>         > >>>> clientPort/clientPortAddress
>>         > >>>>>>>>> statements
>>         > >>>>>>>>>>> (although if you specify client ports in the new
>> format,
>>         > >>>> these
>>         > >>>>>>>>> statements
>>         > >>>>>>>>>>> are now redundant).
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> Could you please let me know about this case.
>> Appreciate
>>         > >> if
>>         > >>>>>> respond
>>         > >>>>>>>>> soon.
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>> Thanks,
>>         > >>>>>>>>>>> ---------------------
>>         > >>>>>>>>>>> Kuldeep Singh Budania
>>         > >>>>>>>>>>>
>>         > >>>>>>>>>>
>>         > >>>>>>>>>
>>         > >>>>>>>>
>>         > >>>>>>>
>>         > >>>>>>
>>         > >>>>>
>>         > >>>>
>>         > >>>
>>         > >>
>>         >
>>         >
>>
>>
>>     ________________________________
>>
>>     CONFIDENTIALITY & PRIVACY NOTICE
>>     This e-mail (including any attachments) is strictly confidential and
>> may also contain privileged information. If you are not the intended
>> recipient you are not authorised to read, print, save, process or disclose
>> this message. If you have received this message by mistake, please inform
>> the sender immediately and destroy this e-mail, its attachments and any
>> copies. Any use, distribution, reproduction or disclosure by any person
>> other than the intended recipient is strictly prohibited and the person
>> responsible may incur in penalties.
>>     The use of this e-mail is only for professional purposes; there is no
>> guarantee that the correspondence towards this e-mail will be read only by
>> the recipient, because, under certain circumstances, there may be a need to
>> access this email by third subjects belonging to the Company.
>>
>>
>> ________________________________
>>
>> CONFIDENTIALITY & PRIVACY NOTICE
>> This e-mail (including any attachments) is strictly confidential and may
>> also contain privileged information. If you are not the intended recipient
>> you are not authorised to read, print, save, process or disclose this
>> message. If you have received this message by mistake, please inform the
>> sender immediately and destroy this e-mail, its attachments and any copies.
>> Any use, distribution, reproduction or disclosure by any person other than
>> the intended recipient is strictly prohibited and the person responsible
>> may incur in penalties.
>> The use of this e-mail is only for professional purposes; there is no
>> guarantee that the correspondence towards this e-mail will be read only by
>> the recipient, because, under certain circumstances, there may be a need to
>> access this email by third subjects belonging to the Company.
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message