Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of hefest@gmail.com designates
 64.233.166.181 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=k4jNqlNZqnNYnbDtsERyI5tw0y2H3KT3J5N8jLtGKV0+5mH0KcL+sMAQ4r8UgsMOGOoAcb2FcRbQ59+56MaelgX3UlzapD2bfV3WeCVjmkbdzI3OUg75sbhMdEXlWhKSNaXckZY7kxNPL7whuIVzMMKm16rUfJC9rsIlgrsdZ9g=
Message-ID: <d487eb8e0607140913h25652cf8ka8b481e0efc6283c@mail.gmail.com>
Date: Fri, 14 Jul 2006 18:13:28 +0200
From: "Tomi NA" <hefest@gmail.com>
To: java-user@lucene.apache.org,
 "Otis Gospodnetic" <otis_gospodnetic@yahoo.com>
Subject: Re: accented characters, wildcards and other problems
In-Reply-To: <20060713165358.5879.qmail@web50315.mail.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <d487eb8e0607130519m3b60471dnc24abcfee849ad1a@mail.gmail.com>
	 <20060713165358.5879.qmail@web50315.mail.yahoo.com>

On 7/13/06, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
> Bok Tomi,
>
> What do you mean by "terms are misrepresented"?  What should they be, and what are you seeing?

I mean 3/5 accented characters appear in the index with accents
correctly displayed, but the remaining 2 accented characters appear as
characters I don't know how to pronounce or what they're called -
somewhere along the line some kind of encoding/decoding process
mistakenly assumes the data is encoded in a certain way.
Update: I've managed to solve the problem localy (when I index a test
directory with accented characters on my ext3 partition), but when I
try indexing a directory I access via a samba mount, I'm stuck with
the old problem again. Could be the iocharset, although there are 2
other encoding related settings which might cause the problem.

> > What I'm not clear on is how can I see the problematic *terms* in the list of terms, but not the documents they're stored in?
>
> Are you saying that the content got indexed, but the file names did not?

I'm saying that I expect to see a list of indexed documents in the
"documents" list, and I don't see the documents containing the
problematic accented characters. However, I see the terms with the
problematic accented characters, although they are missrepresented.

> Out of curiosity (note my last name), I'm curious about what analyzer/tokenizer you're using.  Is there an equivallent of Porter stemmer for Croatian?  I could use that. :)

I'm very new to the technology, so I'm using whatever nutch is using
by default. As far as the stemmer's concerned, I'd say that wildcards
go a long way in providing the necessairy functionality, probably even
better than automatic stemming. However, I apreciate the fact that
most users' minds don't come with an inbuilt regexp constructor. :)
As far as Croatian is concerned, a stemming database was developed
just recently (by "completed" I mean "a usable language coverage") at
the department of Croatian language studies (for want of a better
word)...the problem is, however, it's not publicly available. You see,
when I pay my taxes out of which their salaries are paid, it doesn't
seem to obligate them produce value to me as their indirect invester.
But that's something I'd like to say to their faces, with just a tad
more feeling. ;)

t.n.a.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org