incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rahul challapalli <challapallira...@gmail.com>
Subject Re: Help needed for BLUR-208
Date Fri, 11 Oct 2013 20:07:40 GMT
Hi Sourabh,

I will create a branch for this over the weekend. Can you kindly create a
sub-task to make snapshots work with lucene-4.4?
We can work on this in small steps. If you have some work done on this,
provide a patch and someone will be able to review and apply it. Let me
know what you think.

You can use this link to create patches
http://ariejan.net/2009/10/26/how-to-create-and-apply-a-patch-with-git/

- Rahul


On Thu, Oct 10, 2013 at 5:44 PM, Aaron McCurry <amccurry@gmail.com> wrote:

> On Tue, Oct 8, 2013 at 11:35 PM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Thanks Saurabh for clarifying.
> >
> > Looks like we have to modify the existing code to make it work with
> Lucene
> > 4.4
> >
> > I see 2 approaches that we can take here :
> >
> >   1. We can write our own SanpshotDeletionPolicy if we want users to be
> > able to give names to snapshots (I can imagine people using date as part
> of
> > the name)
> >
>
> I would prefer this option, the writer is very heavy weight as Rahul has
> stated.
>
>
> >   2. We can use PersistentSanpshotDeletionPolicy. The only reason we did
> > not use it was because it uses IndexWriter for persisting which is a very
> > heavy object.
> >
> >
> > I guess we should create a separate branch for this and also add a
> subtask
> > for making snapshots work. Would like to hear some thoughts from others
> as
> > well. Thanks
> >
>
> Sounds good.
>
>
> >
> > - Rahul
> >
> >
> > On Tue, Oct 8, 2013 at 7:23 PM, Aaron McCurry <amccurry@gmail.com>
> wrote:
> >
> > > Hmm, I see what you saying let me take a closer look at it and report
> > back.
> > >
> > > Aaron
> > >
> > >
> > > On Tue, Oct 8, 2013 at 12:48 PM, saurabh gupta <saurabh.b85@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > In BlurNRT class there is a code which loads the existing snapshots:
> > > >
> > > >  if (snapshotsDirectoryExists()) {
> > > >       // load existing snapshots
> > > >       sdp = new
> > > > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy(),
> > > > loadExistingSnapshots());
> > > >     } else {
> > > >       sdp = new
> > > > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy());
> > > >     }
> > > >
> > > > But now in 4.4 version there is no constructor with second argument.
> > They
> > > > changed it corresponding to
> > > > LUCENE-4973<http://issues.apache.org/jira/browse/LUCENE-4973>
> > > > .
> > > >
> > > > Also to open a old snapshot the
> > > >
> > > > IndexCommit snapshot = snapshotter.getSnapshot(name);
> > > >
> > > > changed to
> > > >
> > > > IndexCommit snapshot = snapshotter.getIndexCommit(long gen);
> > > >
> > > > which takes generation
> > > >
> > > > I am not getting how to get the generation.
> > > >
> > > > I hope you understand what I am trying to say.
> > > >
> > > > Thanks,
> > > > Saurabh Gupta
> > > >
> > > >
> > > > On Tue, Oct 8, 2013 at 2:54 AM, Aaron McCurry <amccurry@gmail.com>
> > > wrote:
> > > >
> > > > > On Mon, Oct 7, 2013 at 4:54 PM, saurabh gupta <
> saurabh.b85@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > I am looking an improvement BLUR-208. I am stuck at one place
in
> > > > BlurNRT
> > > > > > class where we are loading previous snapshots and set in
> > > > > > SnapshotDeletionPolicy. But now there is no way to load previous
> > > > > snapshots.
> > > > > >
> > > > >
> > > > > We haven't built a way to load in previous snapshots.  We have
> > planned
> > > on
> > > > > doing so but have not actually implemented it yet.  We left the
> > > snapshots
> > > > > incomplete because at the time we were trying to write an
> InputFormat
> > > for
> > > > > Hadoop.
> > > > >
> > > > > It shouldn't be too hard to do if the table is offline.  Online
> moves
> > > to
> > > > > previous snapshots will be tricky.
> > > > >
> > > > >
> > > > > >
> > > > > > Also SnapshotDeletionPolicy is returning list of IndexCommit
and
> > > > > otherwise
> > > > > > we have to use generation to get the specific IndexCommit. Now
I
> > dont
> > > > > know
> > > > > > how to get the generation.
> > > > > >
> > > > >
> > > > > Are you asking about how it works in Lucene or in Blur?  In Blur
we
> > > > create
> > > > > a snapshot label to manage the snapshots.
> > > > >
> > > > > The Lucene basic code needed to open an old snapshot would be
> > something
> > > > > like:
> > > > >
> > > > >     IndexCommit snapshot = snapshotter.getSnapshot(name);
> > > > >
> > > > >     DirectoryReader.open(snapshot);
> > > > > If you could let us know what / how you would expect Blur to behave
> > > when
> > > > > loading an old snapshot that would be great.  Real world use cases
> > are
> > > > the
> > > > > best to work toward.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Aaron
> > > > >
> > > > >
> > > > >
> > > > > > Can anyone help me?
> > > > > >
> > > > > > Thanks
> > > > > > Saurabh
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message