incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: Help needed for BLUR-208
Date Fri, 11 Oct 2013 00:44:24 GMT
On Tue, Oct 8, 2013 at 11:35 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Thanks Saurabh for clarifying.
>
> Looks like we have to modify the existing code to make it work with Lucene
> 4.4
>
> I see 2 approaches that we can take here :
>
>   1. We can write our own SanpshotDeletionPolicy if we want users to be
> able to give names to snapshots (I can imagine people using date as part of
> the name)
>

I would prefer this option, the writer is very heavy weight as Rahul has
stated.


>   2. We can use PersistentSanpshotDeletionPolicy. The only reason we did
> not use it was because it uses IndexWriter for persisting which is a very
> heavy object.
>
>
> I guess we should create a separate branch for this and also add a subtask
> for making snapshots work. Would like to hear some thoughts from others as
> well. Thanks
>

Sounds good.


>
> - Rahul
>
>
> On Tue, Oct 8, 2013 at 7:23 PM, Aaron McCurry <amccurry@gmail.com> wrote:
>
> > Hmm, I see what you saying let me take a closer look at it and report
> back.
> >
> > Aaron
> >
> >
> > On Tue, Oct 8, 2013 at 12:48 PM, saurabh gupta <saurabh.b85@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > In BlurNRT class there is a code which loads the existing snapshots:
> > >
> > >  if (snapshotsDirectoryExists()) {
> > >       // load existing snapshots
> > >       sdp = new
> > > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy(),
> > > loadExistingSnapshots());
> > >     } else {
> > >       sdp = new
> > > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy());
> > >     }
> > >
> > > But now in 4.4 version there is no constructor with second argument.
> They
> > > changed it corresponding to
> > > LUCENE-4973<http://issues.apache.org/jira/browse/LUCENE-4973>
> > > .
> > >
> > > Also to open a old snapshot the
> > >
> > > IndexCommit snapshot = snapshotter.getSnapshot(name);
> > >
> > > changed to
> > >
> > > IndexCommit snapshot = snapshotter.getIndexCommit(long gen);
> > >
> > > which takes generation
> > >
> > > I am not getting how to get the generation.
> > >
> > > I hope you understand what I am trying to say.
> > >
> > > Thanks,
> > > Saurabh Gupta
> > >
> > >
> > > On Tue, Oct 8, 2013 at 2:54 AM, Aaron McCurry <amccurry@gmail.com>
> > wrote:
> > >
> > > > On Mon, Oct 7, 2013 at 4:54 PM, saurabh gupta <saurabh.b85@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > I am looking an improvement BLUR-208. I am stuck at one place in
> > > BlurNRT
> > > > > class where we are loading previous snapshots and set in
> > > > > SnapshotDeletionPolicy. But now there is no way to load previous
> > > > snapshots.
> > > > >
> > > >
> > > > We haven't built a way to load in previous snapshots.  We have
> planned
> > on
> > > > doing so but have not actually implemented it yet.  We left the
> > snapshots
> > > > incomplete because at the time we were trying to write an InputFormat
> > for
> > > > Hadoop.
> > > >
> > > > It shouldn't be too hard to do if the table is offline.  Online moves
> > to
> > > > previous snapshots will be tricky.
> > > >
> > > >
> > > > >
> > > > > Also SnapshotDeletionPolicy is returning list of IndexCommit and
> > > > otherwise
> > > > > we have to use generation to get the specific IndexCommit. Now I
> dont
> > > > know
> > > > > how to get the generation.
> > > > >
> > > >
> > > > Are you asking about how it works in Lucene or in Blur?  In Blur we
> > > create
> > > > a snapshot label to manage the snapshots.
> > > >
> > > > The Lucene basic code needed to open an old snapshot would be
> something
> > > > like:
> > > >
> > > >     IndexCommit snapshot = snapshotter.getSnapshot(name);
> > > >
> > > >     DirectoryReader.open(snapshot);
> > > > If you could let us know what / how you would expect Blur to behave
> > when
> > > > loading an old snapshot that would be great.  Real world use cases
> are
> > > the
> > > > best to work toward.
> > > >
> > > > Thanks!
> > > >
> > > > Aaron
> > > >
> > > >
> > > >
> > > > > Can anyone help me?
> > > > >
> > > > > Thanks
> > > > > Saurabh
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message