Mailing-List: contact lucy-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of rcmuir@gmail.com designates
 209.85.214.47 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=t7coqpkFeX7xhlGCqUlAasJz678WFpfy4BZ1e6S9P+vzIF+90v4B8eTNOHxL4GU9IB
         26C9KpAYRI1zVRIh0Ve0kaIb9y1ybPopvbCSPZH3xUvGO1w3AfmiVa3+GXfzHX7Iwj04
         uQdSsBuBPaPYgGIyyo10gqAfDhm2F4Hb0O4AE=
MIME-Version: 1.0
In-Reply-To: <20101109030159.GA1688@rectangular.com>
References: <20101030005221.GB21848@rectangular.com>
 <4CCB7AE5.50701@peknet.com>
 <20101109030159.GA1688@rectangular.com>
From: Robert Muir <rcmuir@gmail.com>
Date: Tue, 9 Nov 2010 04:51:33 -0500
Message-ID: <AANLkTimt7HBDvjt2yhXdJKocRyFwDFd-dr1o7O44SF8c@mail.gmail.com>
To: lucy-dev@incubator.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [lucy-dev] Bundling Snowball

On Mon, Nov 8, 2010 at 10:01 PM, Marvin Humphrey <marvin@rectangular.com> w=
rote:
> The "semiclean" build target has been added. =C2=A0I opened
> <https://issues.apache.org/jira/browse/LUCY-125> for bundling the Snowbal=
l
> stemming library. =C2=A0A separate issue will follow for bundling the sto=
plists.
>

Some quick notes, from lucene-java:
* are you going to do svn checkouts for bundling snowball? I don't
think they are really releasing anymore, but there are in fact new
languages, etc in svn.
* every so often snowball makes changes to the rules for the
languages.. this can be tricky depending on how you handle backwards
compatibility. In lucene java we have a checkout of revision 502, but
then with the newer languages added (Armenian, Catalan, Basque)... if
we fully 'svn updated' to the latest rev it would change things about
german stemming from our previous release, for example, and be a
hassle for people who created indexes with those older versions.
* when bundling the stoplists: there are some languages, even
"released" ones (Turkish, Romanian, etc) that don't have
snowball-included stoplists. if you want, you could use the ones we
have in lucene to provide stoplists for these languages...

http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/sr=
c/resources/org/apache/lucene/analysis/tr/stopwords.txt
http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/sr=
c/resources/org/apache/lucene/analysis/ro/stopwords.txt
http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/sr=
c/resources/org/apache/lucene/analysis/hy/stopwords.txt
http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/sr=
c/resources/org/apache/lucene/analysis/eu/stopwords.txt
http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/sr=
c/resources/org/apache/lucene/analysis/ca/stopwords.txt

these are of variable quality: the ones with source information in the
header means that I found one clearly marked with BSD or Apache.
If they have no header, it means i made them myself... it might seem
absurd to worry about "licensing" for stopwords, but you never know :)