incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rahul challapalli <challapallira...@gmail.com>
Subject Re: Help needed for BLUR-208
Date Wed, 09 Oct 2013 03:35:46 GMT
Thanks Saurabh for clarifying.

Looks like we have to modify the existing code to make it work with Lucene
4.4

I see 2 approaches that we can take here :

  1. We can write our own SanpshotDeletionPolicy if we want users to be
able to give names to snapshots (I can imagine people using date as part of
the name)
  2. We can use PersistentSanpshotDeletionPolicy. The only reason we did
not use it was because it uses IndexWriter for persisting which is a very
heavy object.


I guess we should create a separate branch for this and also add a subtask
for making snapshots work. Would like to hear some thoughts from others as
well. Thanks

- Rahul


On Tue, Oct 8, 2013 at 7:23 PM, Aaron McCurry <amccurry@gmail.com> wrote:

> Hmm, I see what you saying let me take a closer look at it and report back.
>
> Aaron
>
>
> On Tue, Oct 8, 2013 at 12:48 PM, saurabh gupta <saurabh.b85@gmail.com
> >wrote:
>
> > Hi,
> >
> > In BlurNRT class there is a code which loads the existing snapshots:
> >
> >  if (snapshotsDirectoryExists()) {
> >       // load existing snapshots
> >       sdp = new
> > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy(),
> > loadExistingSnapshots());
> >     } else {
> >       sdp = new
> > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy());
> >     }
> >
> > But now in 4.4 version there is no constructor with second argument. They
> > changed it corresponding to
> > LUCENE-4973<http://issues.apache.org/jira/browse/LUCENE-4973>
> > .
> >
> > Also to open a old snapshot the
> >
> > IndexCommit snapshot = snapshotter.getSnapshot(name);
> >
> > changed to
> >
> > IndexCommit snapshot = snapshotter.getIndexCommit(long gen);
> >
> > which takes generation
> >
> > I am not getting how to get the generation.
> >
> > I hope you understand what I am trying to say.
> >
> > Thanks,
> > Saurabh Gupta
> >
> >
> > On Tue, Oct 8, 2013 at 2:54 AM, Aaron McCurry <amccurry@gmail.com>
> wrote:
> >
> > > On Mon, Oct 7, 2013 at 4:54 PM, saurabh gupta <saurabh.b85@gmail.com>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > I am looking an improvement BLUR-208. I am stuck at one place in
> > BlurNRT
> > > > class where we are loading previous snapshots and set in
> > > > SnapshotDeletionPolicy. But now there is no way to load previous
> > > snapshots.
> > > >
> > >
> > > We haven't built a way to load in previous snapshots.  We have planned
> on
> > > doing so but have not actually implemented it yet.  We left the
> snapshots
> > > incomplete because at the time we were trying to write an InputFormat
> for
> > > Hadoop.
> > >
> > > It shouldn't be too hard to do if the table is offline.  Online moves
> to
> > > previous snapshots will be tricky.
> > >
> > >
> > > >
> > > > Also SnapshotDeletionPolicy is returning list of IndexCommit and
> > > otherwise
> > > > we have to use generation to get the specific IndexCommit. Now I dont
> > > know
> > > > how to get the generation.
> > > >
> > >
> > > Are you asking about how it works in Lucene or in Blur?  In Blur we
> > create
> > > a snapshot label to manage the snapshots.
> > >
> > > The Lucene basic code needed to open an old snapshot would be something
> > > like:
> > >
> > >     IndexCommit snapshot = snapshotter.getSnapshot(name);
> > >
> > >     DirectoryReader.open(snapshot);
> > > If you could let us know what / how you would expect Blur to behave
> when
> > > loading an old snapshot that would be great.  Real world use cases are
> > the
> > > best to work toward.
> > >
> > > Thanks!
> > >
> > > Aaron
> > >
> > >
> > >
> > > > Can anyone help me?
> > > >
> > > > Thanks
> > > > Saurabh
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message