Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
In-Reply-To: <b24c466954ac4cc79e084b242e4616b7@POST-2.gp.cv.commvault.com>
References: 
 <CADKBNfd66=mH46hXB2dZbOy9o_rTVcgdeBmwu8_ViJjMGh0W=w@mail.gmail.com>
	<CACAwhFmM7nSqkD1-33gUjEP2cdMxx2unOa=H5biQS0gv3rDeLQ@mail.gmail.com>
	<CAMpYv7ArTwYfM5U1EgvN9=C=GzSaSimcRbp1K2qgNJsSCvMJiQ@mail.gmail.com>
	<CADKBNfcq3hvLGZhhQu2USXzW3MSPAyBWHskH2eSmaeZagPwWjQ@mail.gmail.com>
	<CADKBNfdYfsL1fQPvLLRqNieDj0Au-XH=UZKkmstXM-Gupk704A@mail.gmail.com>
	<b24c466954ac4cc79e084b242e4616b7@POST-2.gp.cv.commvault.com>
Date: Sat, 22 Aug 2015 07:00:12 +0000
Message-ID: 
 <CACAwhFk++FeQC66o5_jc7ipwm1P7wUKcKg-3YegwMsq3t6cMvg@mail.gmail.com>
Subject: Re: Lucene Format Plugin
From: =?UTF-8?Q?Stef=C3=A1n_Baxter?= <stefan@activitystream.com>
To: dev <dev@drill.apache.org>
Cc: challapallirahul@gmail.com
Content-Type: multipart/alternative; boundary=001a11431d4a88afe2051de0ef8a

--001a11431d4a88afe2051de0ef8a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Rahul,

Can you elaborate a bit on the status of the Lucene plugin and what needs
to be done before using it?

Also let me know if there are specific things that need improving. We want
to try to using it in our project and perhaps we can contribute something
meaningful.

Regards,
 -Stefan


On Mon, Aug 10, 2015 at 5:01 AM, Sudip Mukherjee <smukherjee@commvault.com>
wrote:

> Hi Rahul,
>
> Thanks for sharing your code. I was trying to get plugin for solr engine.
> But I thought of using solr's rest api to do the queries ,get schema
> metadata info etc.
> The goal for me is to expose a solr engine to tools like Tableau or  MS
> Excel and user can do stuff there.
>
> I am still very new to this and there is a learning curve. It would be
> great if you can comment/review whatever I've done so far.
>
> https://github.com/sudipmukherjee/drill/tree/master/contrib/storage-solr
>
> Thanks,
> Sudip
>
> -----Original Message-----
> From: rahul challapalli [mailto:challapallirahul@gmail.com]
> Sent: 10 August 2015 AM 05:21
> To: dev@drill.apache.org
> Subject: Re: Lucene Format Plugin
>
> Below is the link to my branch which contains the changes related to the
> format plugin.
>
> https://github.com/rchallapalli/drill/tree/lucene/contrib/format-lucene
>
> Any thoughts on how to handle contributions like this which still have
> some work to be done?
>
> - Rahul
>
>
> On Mon, Aug 3, 2015 at 12:21 PM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Thanks Jason.
> >
> > I want to look at the solr plugin and see where we can collaborate or
> > if we already duplicated part of the effort.
> >
> > I still need to push a few commits. I will share the code once I get
> > these changes pushed.
> >
> > - Rahul
> >
> >
> >
> > On Mon, Aug 3, 2015 at 11:31 AM, Jason Altekruse
> > <altekrusejason@gmail.com
> > > wrote:
> >
> >> Hey Rahul,
> >>
> >> This is really cool! Thanks for all of the time you put into writing
> >> this, I think we have a lot of available opportunities to reach new
> >> communities with efforts like this.
> >>
> >> I noticed last week another contributor opened a JIRA for a solr
> >> plugin, there might be a good opportunity for the two of you to join
> >> efforts, as I believe he likely stated working on a lucene reader as
> >> part of his solr work.
> >>
> >> Would you like to post a link to your work on Github or another
> >> public host of your code?
> >>
> >> https://issues.apache.org/jira/browse/DRILL-3585
> >>
> >> On Mon, Aug 3, 2015 at 2:29 AM, Stef=C3=A1n Baxter
> >> <stefan@activitystream.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm pretty new around here but I just wanted to tell you how much
> >> > your
> >> work
> >> > can benefit us. This is great!.
> >> >
> >> > Look forward to trying it out.
> >> >
> >> > Regards,
> >> >  -Stef=C3=A1n
> >> >
> >> > On Mon, Aug 3, 2015 at 8:38 AM, rahul challapalli <
> >> > challapallirahul@gmail.com> wrote:
> >> >
> >> > > Hello Drillers,
> >> > >
> >> > > I have been working on a lucene format plugin. In its current
> >> > > state,
> >> the
> >> > > below sample query successfully searches a lucene index and
> >> > > returns
> >> the
> >> > > results.
> >> > >
> >> > > select path from dfs_test.`/search-index` where
> >> > contents=3D'maxItemsPerBlock'
> >> > > and contents =3D 'BlockTreeTermsIndex'
> >> > >
> >> > >
> >> > >
> >> > > *High Level Overview of Current Implementation:*
> >> > >
> >> > > *Parallelization:* A lucene segment is the lowest level of
> >> > > parrallelization.
> >> > > *Filter Pushdown:* Currently the format plugin is designed to
> >> > > push the complete filter into the scan.
> >> > > *Filter Evaluation:* Each condition in the filter is treated as a
> >> lucene
> >> > > TermQuery
> >> > > <
> >> > >
> >> >
> >> http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/Ter
> >> mQuery.html
> >> > > >
> >> > > and multiple conditions are joined using a BooleanQuery <
> >> > >
> >> >
> >> http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/Boo
> >> leanQuery.html
> >> > > >.
> >> > > If we *do not* use a TermQuery, then we have to know the exact
> >> > > type of Analyzer <
> >> > >
> >> >
> >> https://lucene.apache.org/core/5_2_1/core/org/apache/lucene/analysis/
> >> Analyzer.html
> >> > > >
> >> > > to use with each field in the query.
> >> > >     Ex: 'contents' field might have been analyzed using a
> >> > StandardAnalyzer
> >> > > <
> >> > >
> >> >
> >> https://lucene.apache.org/core/5_2_1/analyzers-common/org/apache/luce
> >> ne/analysis/standard/StandardAnalyzer.html
> >> > > >
> >> > > and the 'path' field might not have been analyzed at all.
> >> > > If desired, support for raw lucene queries with a reserved word
> >> should be
> >> > > easy to add.
> >> > >     Ex: select * from dfs.`search-index` where searchQuery =3D
> >> > > "+contents:maxItemsPerBlock
> >> > > +path:/home/file.txt";
> >> > > *Converting SqlFilter to Lucene Query:* Currently only "=3D" and "=
!=3D"
> >> > > operators are handled while converting a sql filter into a lucene
> >> query.
> >> > > For indexed fields this might be sufficient to handle a good
> >> > > number of cases. For non-indexed fields operators like ">,<, like
> >> > > etc" need to
> >> be
> >> > > handled.
> >> > > *FileSystems:* Currently the format plugin only works on a local
> >> > > filesystem.
> >> > >
> >> > >
> >> > > Though far from complete, I want to work with the community to
> >> > > get
> >> some
> >> > > feedback and avoid any chance of duplication of work. Kindly let
> >> > > me
> >> know
> >> > > your thoughts
> >> > >
> >> > > - Rahul
> >> > >
> >> >
> >>
> >
> >
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you=
."
> **********************************************************************

--001a11431d4a88afe2051de0ef8a--