Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: <CAP-YG6pC53XESYf0GyCNRDhJO9Dr0T4sJ=tDP9iLHv-7JP9xhA@mail.gmail.com>
References: <CAP-YG6oJhRkK8_N39=6_N++owd+ZGa4+feTX446f+_aH0MmqnA@mail.gmail.com>
 <CAP-YG6pHbLhLyMcmaL2oMVg5xFS8oZGjqAiaigv0esQTj59MwA@mail.gmail.com>
 <CA+i0x1+BbbqAV_HxGzr3JH5Mg1MuAnSaHEnb1xnTfH5ms6Du=Q@mail.gmail.com>
 <CAP-YG6p=TtijaF6sTegx60gLPOXumvi4kDhWd+TJcbTeHyWY7A@mail.gmail.com>
 <CA+i0x1+wC0x_zzROdds7q99d3cGSkfDbSstpfxX+epdz_A+cFw@mail.gmail.com> <CAP-YG6pC53XESYf0GyCNRDhJO9Dr0T4sJ=tDP9iLHv-7JP9xhA@mail.gmail.com>
From: Michael Han <hanm@cloudera.com>
Date: Thu, 21 Jul 2016 16:34:00 -0700
Message-ID: <CA+i0x1L94JsJ5GwkuC6qXWmh2_=Kb3w=jn8eOtDQD4E0vsfOtw@mail.gmail.com>
Subject: Re: Config options not being loaded?
To: UserZooKeeper <user@zookeeper.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c1256a06ce34205382dc229
archived-at: Thu, 21 Jul 2016 23:34:37 -0000

--94eb2c1256a06ce34205382dc229
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

>> using zookeeper only in memory

I don't think you can completely disable persistency - ZK will at least
write transaction logs and snap shot to disk in any case. There is
ZOOKEEPER-866 <https://issues.apache.org/jira/browse/ZOOKEEPER-866> relates
to disable persistency completely for ZK, which is probably what you are
looking for ultimately, but that work is not done yet.

>> what would you suggest?

I think your original config looks OK, in particular:
- 'forceSync=3Dno' will make transaction log write async.
- Set snapCount to a really large number to disable snap shot as it seems
you don't have durable requirement and not care about recovery time.

You might also want to clarify / provide more information regarding your
use case (e.g. work load type, read heavy or write heavy, etc), and maybe
some users here that had similar use cases can provide their insights.


On Thu, Jul 21, 2016 at 2:35 PM, Andr=C3=A9 Mantas <andremantas7@gmail.com>
wrote:

> Ok thanks I'll check it out. As for my other question of using zookeeper
> only in memory, what would you suggest?
>
> Thanks for the help so far.
>
> A 10:18PM Qui, 21 de Jul de 2016, Michael Han <hanm@cloudera.com>
> escreveu:
>
> > I think it is not guaranteed that we can find all configuration options
> in
> > server logs because ZK may not log all available configuration options.
> > Another idea to verify the configuration options are loaded with the
> value
> > you specified is to examine the Java system properties of the JVM ZK
> > process, because all the configuration options are loaded as Java syste=
m
> > properties. As an example, skipACL will also be presented as
> > zookeeper.skipACL property.
> >
> > On Thu, Jul 21, 2016 at 2:07 PM, Andr=C3=A9 Mantas <andremantas7@gmail.=
com>
> > wrote:
> >
> > > Thanks for the reply. Indeed I was able to see the skip ACL config in
> the
> > > server log but not any of the other 3 configs. Was it because I
> > specified a
> > > wrong value for these configs maybe?
> > >
> > > A 9:56PM Qui, 21 de Jul de 2016, Michael Han <hanm@cloudera.com>
> > escreveu:
> > >
> > > > The fact that 'conf' does not print out all configuration options
> > encoded
> > > > in zoo.cfg does not necessarily mean these options are not loaded i=
n
> > ZK.
> > > In
> > > > face the options 'conf' will print out is pretty limited and you ca=
n
> > > check
> > > > those options in ZooKeeperServerConf.java
> > > > <
> > > >
> > >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/z=
ookeeper/server/ZooKeeperServerConf.java#L26
> > > > >.
> > > > It'll be good to make this class up to date with regards to latest
> > > > available configuration options in ZK but most if not all
> configuration
> > > > options could also be validated in server logs (like the skipACL on=
e
> > > here).
> > > >
> > > > On Thu, Jul 21, 2016 at 11:55 AM, Andr=C3=A9 Mantas <
> andremantas7@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Follow up:
> > > > > With start-foreground I'm able to see that skipACL=3Dyes was
> effective
> > > > (full
> > > > > output in the end of the email):
> > > > >
> > > > > 2016-07-21 20:36:19,433 [myid:] - INFO
> [main:PrepRequestProcessor@87
> > ]
> > > -
> > > > > zookeeper.skipACL=3D=3D"yes", ACL checks will be skipped
> > > > >
> > > > > What is the problem with the other config values? I'm trying to
> boost
> > > > > performance by disabling the persistency of zookeeper. I'm only
> > > > interested
> > > > > in using zookeeper in memory database (no need for data persisten=
cy
> > > after
> > > > > reboot).
> > > > >
> > > > > Any advice on how to achieve this?
> > > > >
> > > > > Thanks
> > > > >
> > > > > -----
> > > > >
> > > > > ~/amantas/zookeeper-3.4.8# ./bin/zkServer.sh start-foreground
> > > > > ZooKeeper JMX enabled by default
> > > > > Using config: /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > 2016-07-21 20:36:15,060 [myid:] - INFO  [main:QuorumPeerConfig@10=
3
> ]
> > -
> > > > > Reading configuration from:
> > > > > /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > 2016-07-21 20:36:15,069 [myid:] - INFO
> > [main:DatadirCleanupManager@78
> > > ]
> > > > -
> > > > > autopurge.snapRetainCount set to 3
> > > > > 2016-07-21 20:36:15,069 [myid:] - INFO
> > [main:DatadirCleanupManager@79
> > > ]
> > > > -
> > > > > autopurge.purgeInterval set to 0
> > > > > 2016-07-21 20:36:15,069 [myid:] - INFO
> > [main:DatadirCleanupManager@101
> > > ]
> > > > -
> > > > > Purge task is not scheduled.
> > > > > 2016-07-21 20:36:15,071 [myid:] - WARN  [main:QuorumPeerMain@113]
> -
> > > > Either
> > > > > no config or no quorum defined in config, running  in standalone
> mode
> > > > > 2016-07-21 20:36:15,093 [myid:] - INFO  [main:QuorumPeerConfig@10=
3
> ]
> > -
> > > > > Reading configuration from:
> > > > > /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > 2016-07-21 20:36:15,094 [myid:] - INFO
> [main:ZooKeeperServerMain@95
> > ]
> > > -
> > > > > Starting server
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:zookeeper.version=3D3.4.8--1, built on 02/06/2016 03:=
18
> GMT
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:host.name=3Dt8.wan
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.version=3D1.8.0_91
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.vendor=3DOracle Corporation
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.home=3D/usr/lib/jvm/java-8-oracle/jre
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > >
> > > > >
> > > >
> > >
> >
> environment:java.class.path=3D/root/amantas/zookeeper-3.4.8/bin/../build/=
classes:/root/amantas/zookeeper-3.4.8/bin/../build/lib/*.jar:/root/amantas/=
zookeeper-3.4.8/bin/../lib/slf4j-log4j12-1.6.1.jar:/root/amantas/zookeeper-=
3.4.8/bin/../lib/slf4j-api-1.6.1.jar:/root/amantas/zookeeper-3.4.8/bin/../l=
ib/netty-3.7.0.Final.jar:/root/amantas/zookeeper-3.4.8/bin/../lib/log4j-1.2=
.16.jar:/root/amantas/zookeeper-3.4.8/bin/../lib/jline-0.9.94.jar:/root/ama=
ntas/zookeeper-3.4.8/bin/../zookeeper-3.4.8.jar:/root/amantas/zookeeper-3.4=
.8/bin/../src/java/lib/*.jar:/root/amantas/zookeeper-3.4.8/bin/../conf:
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > >
> > > > >
> > > >
> > >
> >
> environment:java.library.path=3D/usr/java/packages/lib/amd64:/usr/lib64:/=
lib64:/lib:/usr/lib
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.io.tmpdir=3D/tmp
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.compiler=3D<NA>
> > > > > 2016-07-21 20:36:15,108 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:os.name=3DLinux
> > > > > 2016-07-21 20:36:15,108 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:os.arch=3Damd64
> > > > > 2016-07-21 20:36:15,108 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:os.version=3D4.4.0-21-generic
> > > > > 2016-07-21 20:36:15,109 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:user.name=3Droot
> > > > > 2016-07-21 20:36:15,109 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:user.home=3D/root
> > > > > 2016-07-21 20:36:15,109 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:user.dir=3D/root/amantas/zookeeper-3.4.8
> > > > > 2016-07-21 20:36:15,118 [myid:] - INFO  [main:ZooKeeperServer@787=
]
> -
> > > > > tickTime set to 2000
> > > > > 2016-07-21 20:36:15,118 [myid:] - INFO  [main:ZooKeeperServer@796=
]
> -
> > > > > minSessionTimeout set to -1
> > > > > 2016-07-21 20:36:15,118 [myid:] - INFO  [main:ZooKeeperServer@805=
]
> -
> > > > > maxSessionTimeout set to -1
> > > > > 2016-07-21 20:36:15,132 [myid:] - INFO
> [main:NIOServerCnxnFactory@89
> > ]
> > > -
> > > > > binding to port 0.0.0.0/0.0.0.0:2181
> > > > > 2016-07-21 20:36:19,433 [myid:] - INFO
> [main:PrepRequestProcessor@87
> > ]
> > > -
> > > > > zookeeper.skipACL=3D=3D"yes", ACL checks will be skipped
> > > > >
> > > > > Andr=C3=A9 Mantas <andremantas7@gmail.com> escreveu no dia quinta=
,
> > > 21/07/2016
> > > > > =C3=A0s
> > > > > 03:28:
> > > > >
> > > > > > Hi all.
> > > > > >
> > > > > > I'm trying to experiment with some config options found in
> > > > > >
> > > > >
> > > >
> > >
> >
> https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_configurat=
ion
> > > > > but
> > > > > > they are not being loaded/used by the zookeeper server.
> > > > > >
> > > > > > I've modified my config file 'zoo.cfg' to:
> > > > > >
> > > > > > # The number of milliseconds of each tick
> > > > > > tickTime=3D2000
> > > > > > # The number of ticks that the initial
> > > > > > # synchronization phase can take
> > > > > > initLimit=3D10
> > > > > > # The number of ticks that can pass between
> > > > > > # sending a request and getting an acknowledgement
> > > > > > syncLimit=3D5
> > > > > > # the directory where the snapshot is stored.
> > > > > > # do not use /tmp for storage, /tmp here is just
> > > > > > # example sakes.
> > > > > > dataDir=3D/root/amantas/zookeeper-3.4.8/data
> > > > > > # the port at which the clients will connect
> > > > > > clientPort=3D2181
> > > > > > # Requires updates to be synced to media of the transaction log
> > > before
> > > > > > finishing processing the update.
> > > > > > # If this option is set to no, ZooKeeper will not require updat=
es
> > to
> > > be
> > > > > > synced to the media.
> > > > > > forceSync=3Dno
> > > > > > # Skips ACL checks. This results in a boost in throughput, but
> > opens
> > > up
> > > > > > full access to the data tree to everyone.
> > > > > > skipACL=3Dyes
> > > > > > # The observers now log transaction and write snapshot to disk =
by
> > > > default
> > > > > > like the participants.
> > > > > > # This reduces the recovery time of the observers on restart.
> > > > > > # Set to "false" to disable this feature. Default is "true"
> > > > > > syncEnabled=3Dfalse
> > > > > > # ZooKeeper logs transactions to a transaction log.
> > > > > > # After snapCount transactions are written to a log file a
> snapshot
> > > is
> > > > > > started and a new transaction log file is created.
> > > > > > # The default snapCount is 100,000
> > > > > > snapCount=3D1000000
> > > > > >
> > > > > > Then, restarted the zookeeper server:
> > > > > >
> > > > > > ~/amantas/zookeeper-3.4.8# ./bin/zkServer.sh stop
> > > > > > ZooKeeper JMX enabled by default
> > > > > > Using config: /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > > Stopping zookeeper ... STOPPED
> > > > > >
> > > > > > ~/amantas/zookeeper-3.4.8# ./bin/zkServer.sh start
> > > > > > ZooKeeper JMX enabled by default
> > > > > > Using config: /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > > Starting zookeeper ... STARTED
> > > > > >
> > > > > > However, when listing the server configuration I don't see some
> of
> > > the
> > > > > > config params:
> > > > > >
> > > > > > ~/amantas/zookeeper-3.4.8# nc localhost 2181
> > > > > > conf
> > > > > > clientPort=3D2181
> > > > > > dataDir=3D/root/amantas/zookeeper-3.4.8/data/version-2
> > > > > > dataLogDir=3D/root/amantas/zookeeper-3.4.8/data/version-2
> > > > > > tickTime=3D2000
> > > > > > maxClientCnxns=3D60
> > > > > > minSessionTimeout=3D4000
> > > > > > maxSessionTimeout=3D40000
> > > > > > serverId=3D0
> > > > > >
> > > > > > What am I doing wrong?
> > > > > >
> > > > > > Thanks in advance!
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >
>


--=20
Cheers
Michael.

--94eb2c1256a06ce34205382dc229--