hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishal Kher <vishalm...@gmail.com>
Subject Re: Understanding ZooKeeper data file management and LogFormatter
Date Mon, 01 Nov 2010 14:00:48 GMT
Hi Mahadev,

I had submitted some small fixes to PurgeTxnLog in
*ZOOKEEPER-872<https://issues.apache.org/jira/browse/ZOOKEEPER-872>
*. Can you or someone else take a look at it?

Thanks.
-Vishal



On Mon, Sep 13, 2010 at 5:39 PM, Mahadev Konar <mahadev@yahoo-inc.com>wrote:

> Hi Vishal,
>  Usually the default retention policy is safe enough for operations.
>
> http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperAdmin.html
>
> Gives you an overview of how to use the purging library in zookeeper.
>
> Thanks
> mahadev
>
>
> On 9/8/10 12:01 PM, "Vishal K" <vishalmlst@gmail.com> wrote:
>
> > Hi All,
> >
> > Can you please share your experience regarding ZK snapshot retention and
> > recovery policies?
> >
> > We have an application where we never need to rollback (i.e., revert back
> to
> > a previous state by using old snapshots). Given this, I am trying to
> > understand under what circumstances would we ever need to use old ZK
> > snapshots. I understand a lot of these decisions depend on the
> application
> > and amount of redundancy used at every level (e.g,. RAID level where the
> > snapshots are stored etc) in the product. To simplify the discussion, I
> > would like to rule out any application characteristics and focus mainly
> on
> > data consistency.
> >
> > - Assuming that we have a 3 node cluster I am trying to figure out when
> > would I really need to use old snapshot files. With 3 nodes we already
> have
> > at least 2 servers with consistent database. If I loose files on one of
> the
> > servers, I can use files from the other. In fact, ZK server join will
> take
> > care of this. I can remove files from a faulty node and reboot that node.
> > The faulty node will sync with the leader.
> >
> > - The old files will be useful if the current snapshot and/or log files
> are
> > lost or corrupted on all 3 servers. If  the loss is due to a disaster
> (case
> > where we loose all 3 servers), one would have to keep the snapshots on
> some
> > external storage to recover. However, if the current snapshot file is
> > corrupted on all 3 servers, then the most likely cause would be a bug in
> ZK.
> > In which case, how can I trust the consistency of the old snapshots?
> >
> > - Given a set of snapshots and log files, how can I verify the
> correctness
> > of these files? Example, if one of the intermediate snapshot file is
> > corrupt.
> >
> > - The Admin's guide says "Using older log and snapshot files, you can
> look
> > at the previous state of ZooKeeper servers and even restore that state.
> The
> > LogFormatter class allows an administrator to look at the transactions in
> a
> > log." * *Is there a tool that does this for the admin?  The LogFormatter
> > only displays the transactions in the log file.
> >
> > - Has anyone ever had to play with the snapshot files in production?
> >
> > Thanks in advance.
> >
> > Regards,
> > -Vishal
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message