Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAMpYv7ArTwYfM5U1EgvN9=C=GzSaSimcRbp1K2qgNJsSCvMJiQ@mail.gmail.com>
References: 
 <CADKBNfd66=mH46hXB2dZbOy9o_rTVcgdeBmwu8_ViJjMGh0W=w@mail.gmail.com>
 <CACAwhFmM7nSqkD1-33gUjEP2cdMxx2unOa=H5biQS0gv3rDeLQ@mail.gmail.com>
 <CAMpYv7ArTwYfM5U1EgvN9=C=GzSaSimcRbp1K2qgNJsSCvMJiQ@mail.gmail.com>
From: rahul challapalli <challapallirahul@gmail.com>
Date: Mon, 3 Aug 2015 12:21:36 -0700
Message-ID: 
 <CADKBNfcq3hvLGZhhQu2USXzW3MSPAyBWHskH2eSmaeZagPwWjQ@mail.gmail.com>
Subject: Re: Lucene Format Plugin
To: dev@drill.apache.org
Content-Type: multipart/alternative; boundary=001a11c34c782d8686051c6d153b

--001a11c34c782d8686051c6d153b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks Jason.

I want to look at the solr plugin and see where we can collaborate or if we
already duplicated part of the effort.

I still need to push a few commits. I will share the code once I get these
changes pushed.

- Rahul


On Mon, Aug 3, 2015 at 11:31 AM, Jason Altekruse <altekrusejason@gmail.com>
wrote:

> Hey Rahul,
>
> This is really cool! Thanks for all of the time you put into writing this=
,
> I think we have a lot of available opportunities to reach new communities
> with efforts like this.
>
> I noticed last week another contributor opened a JIRA for a solr plugin,
> there might be a good opportunity for the two of you to join efforts, as =
I
> believe he likely stated working on a lucene reader as part of his solr
> work.
>
> Would you like to post a link to your work on Github or another public ho=
st
> of your code?
>
> https://issues.apache.org/jira/browse/DRILL-3585
>
> On Mon, Aug 3, 2015 at 2:29 AM, Stef=C3=A1n Baxter <stefan@activitystream=
.com>
> wrote:
>
> > Hi,
> >
> > I'm pretty new around here but I just wanted to tell you how much your
> work
> > can benefit us. This is great!.
> >
> > Look forward to trying it out.
> >
> > Regards,
> >  -Stef=C3=A1n
> >
> > On Mon, Aug 3, 2015 at 8:38 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Hello Drillers,
> > >
> > > I have been working on a lucene format plugin. In its current state,
> the
> > > below sample query successfully searches a lucene index and returns t=
he
> > > results.
> > >
> > > select path from dfs_test.`/search-index` where
> > contents=3D'maxItemsPerBlock'
> > > and contents =3D 'BlockTreeTermsIndex'
> > >
> > >
> > >
> > > *High Level Overview of Current Implementation:*
> > >
> > > *Parallelization:* A lucene segment is the lowest level of
> > > parrallelization.
> > > *Filter Pushdown:* Currently the format plugin is designed to push th=
e
> > > complete filter into the scan.
> > > *Filter Evaluation:* Each condition in the filter is treated as a
> lucene
> > > TermQuery
> > > <
> > >
> >
> http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/TermQue=
ry.html
> > > >
> > > and multiple conditions are joined using a BooleanQuery
> > > <
> > >
> >
> http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/Boolean=
Query.html
> > > >.
> > > If we *do not* use a TermQuery, then we have to know the exact type o=
f
> > > Analyzer
> > > <
> > >
> >
> https://lucene.apache.org/core/5_2_1/core/org/apache/lucene/analysis/Anal=
yzer.html
> > > >
> > > to use with each field in the query.
> > >     Ex: 'contents' field might have been analyzed using a
> > StandardAnalyzer
> > > <
> > >
> >
> https://lucene.apache.org/core/5_2_1/analyzers-common/org/apache/lucene/a=
nalysis/standard/StandardAnalyzer.html
> > > >
> > > and the 'path' field might not have been analyzed at all.
> > > If desired, support for raw lucene queries with a reserved word shoul=
d
> be
> > > easy to add.
> > >     Ex: select * from dfs.`search-index` where searchQuery =3D
> > > "+contents:maxItemsPerBlock
> > > +path:/home/file.txt";
> > > *Converting SqlFilter to Lucene Query:* Currently only "=3D" and "!=
=3D"
> > > operators are handled while converting a sql filter into a lucene
> query.
> > > For indexed fields this might be sufficient to handle a good number o=
f
> > > cases. For non-indexed fields operators like ">,<, like etc" need to =
be
> > > handled.
> > > *FileSystems:* Currently the format plugin only works on a local
> > > filesystem.
> > >
> > >
> > > Though far from complete, I want to work with the community to get so=
me
> > > feedback and avoid any chance of duplication of work. Kindly let me
> know
> > > your thoughts
> > >
> > > - Rahul
> > >
> >
>

--001a11c34c782d8686051c6d153b--