incubator-wave-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Hegarty <heg...@gmail.com>
Subject Re: Delta store
Date Sat, 11 Jan 2014 19:08:16 GMT
Just a note on the unit tests for the mongodb implementation, I noticed
that they weren't actually using the test database and so the drop database
and create db was having no effect, I fixed it and also fixed one or two of
the tests that were relying on the db existing beforehand.  so we should
now have a complete set of unit tests for mongodb that do what we expect.
Regards


On Thu, Jan 9, 2014 at 1:05 PM, Pablo Ojanguren <pablojan@gmail.com> wrote:

> Hi all,
>
> Ben, I've seen your patch for this and I had mine ready also, so I've sent
> it to review too. They both are pretty similar, so it should be good to
> open a discussion about to pros and cons of each approach.
>
> Best regards,
>
> Pablo
>
>
> 2014/1/3 Pablo Ojanguren <pablojan@gmail.com>
>
> > Ben, after seeing your draft patch I think we are taking different
> > approaches. First I am not going to use morphia. For example I think
> > DeltaStore doesn't represent an entity itself. Also I am not sure how
> many
> > collections this approach will generate (one collection per entity?)
> But
> > I don't have any experience in mapping libraries for MongoDB/noSQL so I
> > can't say much more.
> >
> > About WaveletDeltaRecord serialization, this class contains:
> >
> > - Applied delta, already  serialized as a protobuffer. I'd store it as
> is.
> > - Transformed delta: it includes author, versions, timestamp, an a list
> of
> > WaveletOperation objects. I'd serialize this tree in mongoDB fields
> >
> > So, at the end, in mongoDB, an unique document would be stored for this
> > delta, of course, including in the same document Wave and Wavele tIds.
> See
> > attached
> > MongoDbDeltaStore.java as a hint of my approach.
> >
> >
> > As far as I know, Memory and File Delta store issues are about
> scalabilty.
> > Except for WAVE-399 and WAVE-393, the file storage bug regarding the
> index
> > file for deltas.
> >
> >
> > BR,
> >
> > Pablo
> >
> >
> >
> >
> > 2014/1/3 Frank R. <renfeng.cn@gmail.com>
> >
> > I have one delta directory of 11MB at the moment, which consists of four
> >> files. One deltas file is 9.6MB.
> >>
> >>
> >> On Fri, Jan 3, 2014 at 12:45 AM, Ben Hegarty <hegsie@gmail.com> wrote:
> >>
> >> > Does anyone know if the deltas or index files are likely to ever hit
> >> 16MB,
> >> > as that is the maximum document size inside mongodb and if we are then
> >> the
> >> > persistence will need to be pushed into the mongodb GridFS?
> >> >
> >> >
> >> > On Thu, Jan 2, 2014 at 9:58 AM, Frank R. <renfeng.cn@gmail.com>
> wrote:
> >> >
> >> > > This feature is on my wish list for Wave. It's a good news. Thank
> you~
> >> > >
> >> > >
> >> > > On Thu, Jan 2, 2014 at 5:24 PM, Ben Hegarty <hegsie@gmail.com>
> wrote:
> >> > >
> >> > > > Hi Pablo,
> >> > > > Ironically I started work on this feature over the weekend as
it
> is
> >> > next
> >> > > on
> >> > > > my list of must haves for my own personal project which also
uses
> >> > > MongoDB.
> >> > > > I've had to upgrade the MongoDB driver and add in the morphia
> plugin
> >> > for
> >> > > > modelling the delta store which hopefully I'll be able to continue
> >> > > working
> >> > > > on over the weekend.  My time is limited and so if you think
> you'll
> >> be
> >> > > able
> >> > > > to finish before me let me know and I'll try and help with any
> >> > knowledge
> >> > > > I've gained around mongoDB.  It looks like getting the delta
store
> >> > > finished
> >> > > > is going to be fiddly work from what I've seen so far, I started
> by
> >> > > > modelling the store on the memory version, but the first problem
> >> I've
> >> > > found
> >> > > > is that the '.' in mongoDB is a reserved character so I'm going
to
> >> have
> >> > > to
> >> > > > start modelling the store using a mongoDB specific implementation.
> >> > > >
> >> > > > If it helps I've uploaded my current work
> >> > > > here<
> >> > > >
> >> > >
> >> >
> >>
> https://drive.google.com/file/d/0B5FF_Ld8SzsNQ2JONHdGU2xJUG8/edit?usp=sharing
> >> > > > >which
> >> > > > just keep in mind is very much just the beginning.
> >> > > >
> >> > > > HTH
> >> > > > Ben
> >> > > >
> >> > > >
> >> > > > On Thu, Jan 2, 2014 at 8:41 AM, Pablo Ojanguren <
> pablojan@gmail.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi all,
> >> > > > >
> >> > > > > I am a developer working for an EU research project. There,
we
> are
> >> > > going
> >> > > > to
> >> > > > > use Wave (via Kune project) so we would like to help fixing
bugs
> >> and
> >> > > > > improving the software along wiht the community. Currently
I
> have
> >> > some
> >> > > > > available time for this for the next 10 months.
> >> > > > >
> >> > > > > After check out the issues, first thing I've paid attention
is
> the
> >> > > > Delta's
> >> > > > > storage. I have reviewed server / persistence code and I
think I
> >> can
> >> > > > start
> >> > > > > working in a mongoDB implementation.
> >> > > > >
> >> > > > > But maybe there is somebody else working on this. I would
like
> to
> >> > join
> >> > > > them
> >> > > > > and to avoid duplicated efforts. Or if there is another
> >> outstanding
> >> > > > topics
> >> > > > > to suggest it would be great to know too.
> >> > > > >
> >> > > > > Thank you and Happy New Year.
> >> > > > >
> >> > > > > Pablo
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Mobile Phone: +447767 322 122
> >> > > > Home Phone: +4420 7737 6530
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards
> >> > Ben
> >> >
> >>
> >
> >
>



-- 
Regards
Ben

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message