incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From saurabh gupta <saurabh....@gmail.com>
Subject Re: Help needed for BLUR-208
Date Sat, 12 Oct 2013 17:32:04 GMT
Hi Rahul,

This is a good idea that you will create a branch. I will create sub task
in story. Yes I have changed code. But BlurNRT has compilation error due to
snapshotdeletionpolicy so suggest me whether I should create a patch or not.

Thanks,
Saurabh


On Fri, Oct 11, 2013 at 10:07 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Hi Sourabh,
>
> I will create a branch for this over the weekend. Can you kindly create a
> sub-task to make snapshots work with lucene-4.4?
> We can work on this in small steps. If you have some work done on this,
> provide a patch and someone will be able to review and apply it. Let me
> know what you think.
>
> You can use this link to create patches
> http://ariejan.net/2009/10/26/how-to-create-and-apply-a-patch-with-git/
>
> - Rahul
>
>
> On Thu, Oct 10, 2013 at 5:44 PM, Aaron McCurry <amccurry@gmail.com> wrote:
>
> > On Tue, Oct 8, 2013 at 11:35 PM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Thanks Saurabh for clarifying.
> > >
> > > Looks like we have to modify the existing code to make it work with
> > Lucene
> > > 4.4
> > >
> > > I see 2 approaches that we can take here :
> > >
> > >   1. We can write our own SanpshotDeletionPolicy if we want users to be
> > > able to give names to snapshots (I can imagine people using date as
> part
> > of
> > > the name)
> > >
> >
> > I would prefer this option, the writer is very heavy weight as Rahul has
> > stated.
> >
> >
> > >   2. We can use PersistentSanpshotDeletionPolicy. The only reason we
> did
> > > not use it was because it uses IndexWriter for persisting which is a
> very
> > > heavy object.
> > >
> > >
> > > I guess we should create a separate branch for this and also add a
> > subtask
> > > for making snapshots work. Would like to hear some thoughts from others
> > as
> > > well. Thanks
> > >
> >
> > Sounds good.
> >
> >
> > >
> > > - Rahul
> > >
> > >
> > > On Tue, Oct 8, 2013 at 7:23 PM, Aaron McCurry <amccurry@gmail.com>
> > wrote:
> > >
> > > > Hmm, I see what you saying let me take a closer look at it and report
> > > back.
> > > >
> > > > Aaron
> > > >
> > > >
> > > > On Tue, Oct 8, 2013 at 12:48 PM, saurabh gupta <
> saurabh.b85@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > In BlurNRT class there is a code which loads the existing
> snapshots:
> > > > >
> > > > >  if (snapshotsDirectoryExists()) {
> > > > >       // load existing snapshots
> > > > >       sdp = new
> > > > > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy(),
> > > > > loadExistingSnapshots());
> > > > >     } else {
> > > > >       sdp = new
> > > > > SnapshotDeletionPolicy(_tableContext.getIndexDeletionPolicy());
> > > > >     }
> > > > >
> > > > > But now in 4.4 version there is no constructor with second
> argument.
> > > They
> > > > > changed it corresponding to
> > > > > LUCENE-4973<http://issues.apache.org/jira/browse/LUCENE-4973>
> > > > > .
> > > > >
> > > > > Also to open a old snapshot the
> > > > >
> > > > > IndexCommit snapshot = snapshotter.getSnapshot(name);
> > > > >
> > > > > changed to
> > > > >
> > > > > IndexCommit snapshot = snapshotter.getIndexCommit(long gen);
> > > > >
> > > > > which takes generation
> > > > >
> > > > > I am not getting how to get the generation.
> > > > >
> > > > > I hope you understand what I am trying to say.
> > > > >
> > > > > Thanks,
> > > > > Saurabh Gupta
> > > > >
> > > > >
> > > > > On Tue, Oct 8, 2013 at 2:54 AM, Aaron McCurry <amccurry@gmail.com>
> > > > wrote:
> > > > >
> > > > > > On Mon, Oct 7, 2013 at 4:54 PM, saurabh gupta <
> > saurabh.b85@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I am looking an improvement BLUR-208. I am stuck at one
place
> in
> > > > > BlurNRT
> > > > > > > class where we are loading previous snapshots and set in
> > > > > > > SnapshotDeletionPolicy. But now there is no way to load
> previous
> > > > > > snapshots.
> > > > > > >
> > > > > >
> > > > > > We haven't built a way to load in previous snapshots.  We have
> > > planned
> > > > on
> > > > > > doing so but have not actually implemented it yet.  We left
the
> > > > snapshots
> > > > > > incomplete because at the time we were trying to write an
> > InputFormat
> > > > for
> > > > > > Hadoop.
> > > > > >
> > > > > > It shouldn't be too hard to do if the table is offline.  Online
> > moves
> > > > to
> > > > > > previous snapshots will be tricky.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Also SnapshotDeletionPolicy is returning list of IndexCommit
> and
> > > > > > otherwise
> > > > > > > we have to use generation to get the specific IndexCommit.
Now
> I
> > > dont
> > > > > > know
> > > > > > > how to get the generation.
> > > > > > >
> > > > > >
> > > > > > Are you asking about how it works in Lucene or in Blur?  In
Blur
> we
> > > > > create
> > > > > > a snapshot label to manage the snapshots.
> > > > > >
> > > > > > The Lucene basic code needed to open an old snapshot would be
> > > something
> > > > > > like:
> > > > > >
> > > > > >     IndexCommit snapshot = snapshotter.getSnapshot(name);
> > > > > >
> > > > > >     DirectoryReader.open(snapshot);
> > > > > > If you could let us know what / how you would expect Blur to
> behave
> > > > when
> > > > > > loading an old snapshot that would be great.  Real world use
> cases
> > > are
> > > > > the
> > > > > > best to work toward.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Aaron
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Can anyone help me?
> > > > > > >
> > > > > > > Thanks
> > > > > > > Saurabh
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message