incubator-wave-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pablo Ojanguren <pablo...@gmail.com>
Subject Re: Delta store
Date Fri, 03 Jan 2014 09:34:13 GMT
Ben, after seeing your draft patch I think we are taking different
approaches. First I am not going to use morphia. For example I think
DeltaStore doesn't represent an entity itself. Also I am not sure how many
collections this approach will generate (one collection per entity?)   But
I don't have any experience in mapping libraries for MongoDB/noSQL so I
can't say much more.

About WaveletDeltaRecord serialization, this class contains:

- Applied delta, already  serialized as a protobuffer. I'd store it as is.
- Transformed delta: it includes author, versions, timestamp, an a list of
WaveletOperation objects. I'd serialize this tree in mongoDB fields

So, at the end, in mongoDB, an unique document would be stored for this
delta, of course, including in the same document Wave and Wavele tIds. See
attached
MongoDbDeltaStore.java as a hint of my approach.


As far as I know, Memory and File Delta store issues are about scalabilty.
Except for WAVE-399 and WAVE-393, the file storage bug regarding the index
file for deltas.


BR,

Pablo




2014/1/3 Frank R. <renfeng.cn@gmail.com>

> I have one delta directory of 11MB at the moment, which consists of four
> files. One deltas file is 9.6MB.
>
>
> On Fri, Jan 3, 2014 at 12:45 AM, Ben Hegarty <hegsie@gmail.com> wrote:
>
> > Does anyone know if the deltas or index files are likely to ever hit
> 16MB,
> > as that is the maximum document size inside mongodb and if we are then
> the
> > persistence will need to be pushed into the mongodb GridFS?
> >
> >
> > On Thu, Jan 2, 2014 at 9:58 AM, Frank R. <renfeng.cn@gmail.com> wrote:
> >
> > > This feature is on my wish list for Wave. It's a good news. Thank you~
> > >
> > >
> > > On Thu, Jan 2, 2014 at 5:24 PM, Ben Hegarty <hegsie@gmail.com> wrote:
> > >
> > > > Hi Pablo,
> > > > Ironically I started work on this feature over the weekend as it is
> > next
> > > on
> > > > my list of must haves for my own personal project which also uses
> > > MongoDB.
> > > > I've had to upgrade the MongoDB driver and add in the morphia plugin
> > for
> > > > modelling the delta store which hopefully I'll be able to continue
> > > working
> > > > on over the weekend.  My time is limited and so if you think you'll
> be
> > > able
> > > > to finish before me let me know and I'll try and help with any
> > knowledge
> > > > I've gained around mongoDB.  It looks like getting the delta store
> > > finished
> > > > is going to be fiddly work from what I've seen so far, I started by
> > > > modelling the store on the memory version, but the first problem I've
> > > found
> > > > is that the '.' in mongoDB is a reserved character so I'm going to
> have
> > > to
> > > > start modelling the store using a mongoDB specific implementation.
> > > >
> > > > If it helps I've uploaded my current work
> > > > here<
> > > >
> > >
> >
> https://drive.google.com/file/d/0B5FF_Ld8SzsNQ2JONHdGU2xJUG8/edit?usp=sharing
> > > > >which
> > > > just keep in mind is very much just the beginning.
> > > >
> > > > HTH
> > > > Ben
> > > >
> > > >
> > > > On Thu, Jan 2, 2014 at 8:41 AM, Pablo Ojanguren <pablojan@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am a developer working for an EU research project. There, we are
> > > going
> > > > to
> > > > > use Wave (via Kune project) so we would like to help fixing bugs
> and
> > > > > improving the software along wiht the community. Currently I have
> > some
> > > > > available time for this for the next 10 months.
> > > > >
> > > > > After check out the issues, first thing I've paid attention is the
> > > > Delta's
> > > > > storage. I have reviewed server / persistence code and I think I
> can
> > > > start
> > > > > working in a mongoDB implementation.
> > > > >
> > > > > But maybe there is somebody else working on this. I would like to
> > join
> > > > them
> > > > > and to avoid duplicated efforts. Or if there is another outstanding
> > > > topics
> > > > > to suggest it would be great to know too.
> > > > >
> > > > > Thank you and Happy New Year.
> > > > >
> > > > > Pablo
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Mobile Phone: +447767 322 122
> > > > Home Phone: +4420 7737 6530
> > > >
> > >
> >
> >
> >
> > --
> > Regards
> > Ben
> >
>

Mime
View raw message