couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex P <apede...@kolosy.com>
Subject Re: couch behavior for san snapshots
Date Thu, 22 Oct 2009 06:28:19 GMT
so an update on this situation. we managed to replicate the problem. took a
san snapshot, got an empty db. here's where it gets tricky. we dug around
the log files, and found error messages that matched these:

http://mail-archives.apache.org/mod_mbox/couchdb-user/200812.mbox/%3C4c69d7170812181239j6527a387oa63e129e20582b8f@mail.gmail.com%3E

on a hunch, we ran a compaction... and voila! all docs are back.

again, this is on .91 (we're moving to .10 tonight/tomorrow), but any
insight as to what could have caused this would be appreciated.

thanks,
alex

On Mon, Oct 19, 2009 at 11:10 PM, Alex P <apedenko@kolosy.com> wrote:

> we're using xfs, but thanks for the offer.
>
> looks like someone discarded the snapshot that we had, so i can't compare
> the database. the only new piece of information that i have is that ebs
> snapshots are differential, and clearing out all history seems to have
> 'resolved' the issue. not an answer i'm happy with, but it's no longer
> replicatable.
>
>
> On Mon, Oct 19, 2009 at 10:35 PM, Dusty Doris <couchdb.list@dusty.name>wrote:
>
>> On Mon, Oct 19, 2009 at 5:37 PM, Alex P <apedenko@kolosy.com> wrote:
>> > thanks! i'll definitely look into that.
>>
>> If your are using ext3 I can also help you with that.  You need to use
>> LVM2 to create volumes and then you can do something like
>>
>> $ dmsetup suspend /dev/ebs/couchvol0
>> $ ebs_snapshot
>> $ dmsetup resume /dev/ebs/couchvol0
>>
>> If you'd like more info on that let me know and I'll add some more
>> details.
>>
>>
>>
>> >
>> > On Mon, Oct 19, 2009 at 4:32 PM, Glenn Rempe <glenn@rempe.us> wrote:
>> >
>> >> Not sure if this will help at all, but I am also using EBS to store a
>> large
>> >> cdb DB.
>> >> It is recommended that the filesystem that you want to snapshot on EBS
>> is
>> >> frozen (or unmounted) if that is possible to help ensure a coherent EBS
>> >> snapshot.
>> >>
>> >> For example, on my mysql and couchdb volumes I use the XFS filesystem
>> which
>> >> has an XFS freeze command which I use to temporarily lock the FS and
>> flush
>> >> it to disk, and then request the snapshot.  You can do this in a script
>> and
>> >> it takes only a few seconds of disk downtime.  I believe writes are
>> cached
>> >> during this period.  I have been using this with mysql but have not yet
>> set
>> >> it up for my couchdb volume.
>> >>
>> >> See :
>> >>
>> >>
>> >>
>> http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=100&externalID=1663
>> >> http://alestic.com/2009/09/ec2-consistent-snapshot
>> >>
>> >> HTH
>> >>
>> >> On Mon, Oct 19, 2009 at 1:55 PM, Chris Anderson <jchris@apache.org>
>> wrote:
>> >>
>> >> > On Mon, Oct 19, 2009 at 12:53 PM, Alex P <apedenko@kolosy.com>
>> wrote:
>> >> > > a simple copy seems to work just fine. db shows up with the right
>> seq
>> >> > number
>> >> > > etc.
>> >> >
>> >> > cool. totally wanna learn about the SAN roundtrip. :)
>> >> >
>> >> > >
>> >> > > On Mon, Oct 19, 2009 at 2:37 PM, Paul Davis <
>> >> paul.joseph.davis@gmail.com
>> >> > >wrote:
>> >> > >
>> >> > >> On Mon, Oct 19, 2009 at 3:35 PM, Alex P <apedenko@kolosy.com>
>> wrote:
>> >> > >> > cdb catalogs databases purely on file name and presence
in the
>> >> > >> appropriate
>> >> > >> > directory, right? so if i copy db1 to db1_copy, i should
see
>> >> db1_copy
>> >> > >> show
>> >> > >> > up in futon?
>> >> > >>
>> >> > >> Yepper. Make sure the permissions are correct after copying
>> though.
>> >> > >>
>> >> > >> Paul Davis
>> >> > >>
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Chris Anderson
>> >> > http://jchrisa.net
>> >> > http://couch.io
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Glenn Rempe
>> >>
>> >> email                 : glenn@rempe.us
>> >> voice                 : (415) 894-5366 or (415)-89G-LENN
>> >> twitter                : @grempe
>> >> contact info        : http://www.rempe.us/contact.html
>> >> pgp                    : http://www.rempe.us/gnupg.txt
>> >>
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message