Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of johan.tibell@gmail.com
 designates 209.85.215.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAE9W93NJPb8Z=wJwhCtMW0+xGo5AmN=-qz+ZvwMVd7GcDOc6AA@mail.gmail.com>
References: 
 <CAK-tuPY6KjNJRsQc71fTwwxt7F6f+cHKUNJQabr_uhhhSR-cgg@mail.gmail.com>
 <CAE9W93NJPb8Z=wJwhCtMW0+xGo5AmN=-qz+ZvwMVd7GcDOc6AA@mail.gmail.com>
From: Johan Tibell <johan.tibell@gmail.com>
Date: Wed, 4 Jun 2014 11:59:36 +0200
Message-ID: 
 <CAK-tuPaJh-Sb_ngK0KjHFA-0JqY9wiy1SRNdKFD+PM4qFFa32Q@mail.gmail.com>
Subject: Re: How to approach indexing source code?
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=001a11c38be2cc597004faffb022

--001a11c38be2cc597004faffb022
Content-Type: text/plain; charset=UTF-8

The the majority of queries will be look-ups of functions/types by fully
qualified name. For example, the query [Data.Map.insert] will find the
definition and all uses of the `insert` function defined in the `Data.Map`
module. The corpus is all Haskell open source code on hackage.haskell.org.

Being able to support qualified name queries is the main benefit of
indexing the output of the compiler (which has resolved unqualified names
to qualified names) rather than using a simple text-based indexing.

There are three levels of name qualification I want to support in queries:

 * Unqualified: myFunction
 * Module qualified: MyModule.myFunction
 * Package and module qualified: mypackage-MyModule.myFunction

I expect the middle one to be used the most. The last form is sometimes
needed for disambiguation and the first is nice to support as a shorthand
when the function name is unlikely to be ambiguous.

For scoring I'd like to have a couple of attributes available. The most
important one is whether a term represents a use site or a definition site.
This would allow the definition of a function to appear as the first search
result.

Is this precise enough? Naturally the scope will grow over time, but this
is the core of what I'm trying to do.

-- Johan


On Wed, Jun 4, 2014 at 8:02 AM, Aditya <findbestopensource@gmail.com> wrote:

> Hi Johan,
>
> How you want to search, What is your search requirement and according to
> that you need to index. You could check duckduckgo or github code search.
>
> The easiest approach would be to have a parser which will read each source
> file and indexes as a single document. When you search, you will have a
> single search field which will search the index and retrieves the result.
> The search field accepts any text in the source file. It could be function
> name, class name, comments or variables etc.
>
> Another approach is to have different search fields for Functions, Classes,
> Package etc.  You need to parse the file, identify comments, function name,
> class name etc and index it in a separate field.
>
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
>
>
> On Wed, Jun 4, 2014 at 7:02 AM, Johan Tibell <johan.tibell@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'd like to index (Haskell) source code. I've run the source code
> through a
> > compiler (GHC) to get rich information about each token (its type, fully
> > qualified name, etc) that I want to index (and later use when ranking).
> >
> > I'm wondering how to approach indexing source code. I can see two
> possible
> > approaches:
> >
> >  * Create a file containing all the metadata and write a custom
> > tokenizer/analyzer that processes the file. The file could use a simple
> > line-based format:
> >
> > myFunction,1:12-1:22,my-package,defined-here,more-metadata
> > myFunction,5:11-5:21,my-package,used-here,more-metadata
> > ...
> >
> > The tokenizer would use CharTermAttribute to write the function name,
> > OffsetAttribute to write the source span, etc.
> >
> >  * Use and IndexWriter to create a Document directly, as done here:
> >
> >
> http://www.onjava.com/pub/a/onjava/2006/01/18/using-lucene-to-search-java-source.html?page=3
> >
> > I'm new to Lucene so I can't quite tell which approach is more likely to
> > work well. Which way would you recommend?
> >
> > Other things I'd like to do that might influence the answer:
> >
> >  - Index several tokens at the same position, so I can index both the
> fully
> > qualified name (e.g. module.myFunction) and unqualified name (e.g.
> > myFunction) for a term.
> >
> > -- Johan
> >
>

--001a11c38be2cc597004faffb022--