couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From svilen ...@svilendobrev.com>
Subject Re: huge attachments - experience?
Date Thu, 28 Mar 2013 11:23:48 GMT
yeah, checked all those. They are post-factum, file synchronizers - can
not manage directories (rename a dir => copy all the dir again).
The only thing that manages directories is bazaar but it chokes after
8-9Gb of data (out of memory). Anyway seems i need to separate
filesystem metadata (filenames, directories etc) from actual raw
datafiles. e.g. fancy hierarchy of hardlinks, pointing to flat-dir
files. And apply separate syncs to those two. But should think it out
first.

On Thu, 28 Mar 2013 10:29:30 +0100
Nils Breunese <N.Breunese@vpro.nl> wrote:

> svilen wrote:
> 
> >> Nothing couchdb, but bup (https://github.com/bup/bup/) might be 
> >> the solution for you.
> > will check it up, thanks.
> > 
> >> What did you use before? rsync? 
> > rsync , lftp, manual cp -ru ..
> > 
> >> What's the nature for the problem?
> > well, the usual computer-geek one: 3 machines with copies of same
> > huge file hierarchy (500G) somewhere. Anything changed on any one
> > machine should sooner-or-later propagate to others. "worst" kind of
> > changes are renaming of directories, splitting one dir into two,
> > etc. filesystem-level meta-changes. Nothing that i know tracks those
> > (except bazaar and maybe mercurial) although it's simple. no need to
> > be fast or immediate, it's just me changing things manualy here or
> > there.
> > setting up special ocfs2 partitions is not an option.
> 
> If it were two machines, you could use Unison, but since you have
> three and apparently setting up a network filesystem is not an
> option, you might want to take a look at csync2:
> 
> http://linuxaria.com/howto/csync2-a-filesystem-syncronization-tool-for-linux?lang=en
> http://oss.linbit.com/csync2/
> 
> You could also use csync2 with lsyncd if you need more instant
> updates.
> 
> https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/
> 
> Nils.

Mime
View raw message