incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: couch behavior for san snapshots
Date Thu, 22 Oct 2009 15:05:56 GMT
Alex,

Are any of the beam files used by the running CouchDB node on the EBS
volume? I've seen these errors before and they're generally (always?)
a result of the Erlang VM being unable to read from its current
working directory.

Paul Davis

On Thu, Oct 22, 2009 at 2:28 AM, Alex P <apedenko@kolosy.com> wrote:
> so an update on this situation. we managed to replicate the problem. took a
> san snapshot, got an empty db. here's where it gets tricky. we dug around
> the log files, and found error messages that matched these:
>
> http://mail-archives.apache.org/mod_mbox/couchdb-user/200812.mbox/%3C4c69d7170812181239j6527a387oa63e129e20582b8f@mail.gmail.com%3E
>
> on a hunch, we ran a compaction... and voila! all docs are back.
>
> again, this is on .91 (we're moving to .10 tonight/tomorrow), but any
> insight as to what could have caused this would be appreciated.
>
> thanks,
> alex
>
> On Mon, Oct 19, 2009 at 11:10 PM, Alex P <apedenko@kolosy.com> wrote:
>
>> we're using xfs, but thanks for the offer.
>>
>> looks like someone discarded the snapshot that we had, so i can't compare
>> the database. the only new piece of information that i have is that ebs
>> snapshots are differential, and clearing out all history seems to have
>> 'resolved' the issue. not an answer i'm happy with, but it's no longer
>> replicatable.
>>
>>
>> On Mon, Oct 19, 2009 at 10:35 PM, Dusty Doris <couchdb.list@dusty.name>wrote:
>>
>>> On Mon, Oct 19, 2009 at 5:37 PM, Alex P <apedenko@kolosy.com> wrote:
>>> > thanks! i'll definitely look into that.
>>>
>>> If your are using ext3 I can also help you with that.  You need to use
>>> LVM2 to create volumes and then you can do something like
>>>
>>> $ dmsetup suspend /dev/ebs/couchvol0
>>> $ ebs_snapshot
>>> $ dmsetup resume /dev/ebs/couchvol0
>>>
>>> If you'd like more info on that let me know and I'll add some more
>>> details.
>>>
>>>
>>>
>>> >
>>> > On Mon, Oct 19, 2009 at 4:32 PM, Glenn Rempe <glenn@rempe.us> wrote:
>>> >
>>> >> Not sure if this will help at all, but I am also using EBS to store
a
>>> large
>>> >> cdb DB.
>>> >> It is recommended that the filesystem that you want to snapshot on EBS
>>> is
>>> >> frozen (or unmounted) if that is possible to help ensure a coherent
EBS
>>> >> snapshot.
>>> >>
>>> >> For example, on my mysql and couchdb volumes I use the XFS filesystem
>>> which
>>> >> has an XFS freeze command which I use to temporarily lock the FS and
>>> flush
>>> >> it to disk, and then request the snapshot.  You can do this in a script
>>> and
>>> >> it takes only a few seconds of disk downtime.  I believe writes are
>>> cached
>>> >> during this period.  I have been using this with mysql but have not
yet
>>> set
>>> >> it up for my couchdb volume.
>>> >>
>>> >> See :
>>> >>
>>> >>
>>> >>
>>> http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=100&externalID=1663
>>> >> http://alestic.com/2009/09/ec2-consistent-snapshot
>>> >>
>>> >> HTH
>>> >>
>>> >> On Mon, Oct 19, 2009 at 1:55 PM, Chris Anderson <jchris@apache.org>
>>> wrote:
>>> >>
>>> >> > On Mon, Oct 19, 2009 at 12:53 PM, Alex P <apedenko@kolosy.com>
>>> wrote:
>>> >> > > a simple copy seems to work just fine. db shows up with the
right
>>> seq
>>> >> > number
>>> >> > > etc.
>>> >> >
>>> >> > cool. totally wanna learn about the SAN roundtrip. :)
>>> >> >
>>> >> > >
>>> >> > > On Mon, Oct 19, 2009 at 2:37 PM, Paul Davis <
>>> >> paul.joseph.davis@gmail.com
>>> >> > >wrote:
>>> >> > >
>>> >> > >> On Mon, Oct 19, 2009 at 3:35 PM, Alex P <apedenko@kolosy.com>
>>> wrote:
>>> >> > >> > cdb catalogs databases purely on file name and presence
in the
>>> >> > >> appropriate
>>> >> > >> > directory, right? so if i copy db1 to db1_copy, i
should see
>>> >> db1_copy
>>> >> > >> show
>>> >> > >> > up in futon?
>>> >> > >>
>>> >> > >> Yepper. Make sure the permissions are correct after copying
>>> though.
>>> >> > >>
>>> >> > >> Paul Davis
>>> >> > >>
>>> >> > >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Chris Anderson
>>> >> > http://jchrisa.net
>>> >> > http://couch.io
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Glenn Rempe
>>> >>
>>> >> email                 : glenn@rempe.us
>>> >> voice                 : (415) 894-5366 or (415)-89G-LENN
>>> >> twitter                : @grempe
>>> >> contact info        : http://www.rempe.us/contact.html
>>> >> pgp                    : http://www.rempe.us/gnupg.txt
>>> >>
>>> >
>>>
>>
>>
>

Mime
View raw message