lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mohammad Norouzi" <mnr...@gmail.com>
Subject Re: question about field equality in query
Date Sun, 15 Apr 2007 15:21:57 GMT
well, let me tell you what we're going to do.
our information are too much. actually, we have a huge database for
hospitals, and their services and their patients and also pharmacy infos.

just for test, we first joined some important tables and as you said
de-normalized them and then tried to create an index. it was about 3
millions documents and surprisingly, we saw that using this system we can
achieve our goal in a few seconds, however, by using SQL statements it took
more than half an hour !!! so what did we want more? we just want to reach
our result and Lucene fulfils our needs.

but we are not going to leave the database and throw it out. we need Lucene
in some special cases. in our case, reporting from whole informatin.

but the problem is, because there are more tables (actually there are more
than 10 database views) so we decided to categorize information and  create
a view for each category. and then index them individually.

but we need to find some information on say, index2, where they have
relation with say index1

Erick, your second way doesnt fulfil our needs because the information are
too complex. and the third one, yes we finally do that, but before that we
need to do some searching (with a complex query) on indexes so we need many
information on our indexes.

I was thinking of building another table (table, I dont mean database table,
it can be every structure) that keeps lucene's internal ids and their
relations with other indexes. and I tested it, now it works fine but you
know this internal ids will change once we call optimize() method. so we
should re-create them each time we update the index. (we do it every other
week)

so if you had provided a feature for such problem to find the relations
between various indexes every thing was nice.

I know this is not an aspect for Lucene, but in such situation it would help
and save the developers and users, specially when we were able to disable or
enable that feature depend on our needs.

lots of thanks

On 4/15/07, Erick Erickson <erickerickson@gmail.com> wrote:
>
> well, how much data are we talking about here? If your index
> is already 100G, your options are much more limited than if
> it's 1M...
>
> Essentially, you have several choices:
> 1> de-normalize your data to whatever extent is "sufficient"
>     and index it.
> 2> Try to do what amount to joins. If your needs aren't too complex,
>      you can probably do some interesting things with
>      TermEnum/TermDocs, but I should emphasize the word *simple*
>      in the relationships or else you'll be re-inventing an RDBMS <G>.
> 3> You can mix-n-match. That is, use Lucene to index searchable
>      data and use the database for some portion of your joins.
>
> Best
> Erick
>
>
> On 4/15/07, Mohammad Norouzi <mnrz57@gmail.com> wrote:
> >
> > Thanks Karl,
> > sorry I was on business vacation and could not reply soon.
> >
> > Ok, I want to index my database and search through the indexes, and the
> > tables in database are more than 50 and it's hard to joining them and
> then
> > indexing. so I was thinking of indexing each table in a document but in
> > one
> > index file and then make relations between them
> >
> > I know that Lucene is not a relational database but it really helps us
> to
> > improve accessing our information and even generating reports.
> >
> > Even if this feature decline the performance, still it is better than
> > generating reports from SQL statements and the speed up accessing
> > information. now our users are very satisfied by this. but we need to
> > create
> > some relationship between documents.
> >
> >
> > On 4/11/07, karl wettin <karl.wettin@gmail.com> wrote:
> > >
> > >
> > > 11 apr 2007 kl. 11.19 skrev Mohammad Norouzi:
> > >
> > > > is it possible (or a trickery way)  to search with a given query in
> > > > which we
> > > > can set an equality for two fields
> > > > for example:
> > > >
> > > > Document:
> > > > field1             field2                field3          field4
> > > >
> > > > Query:
> > > >  field1:"test phrase" AND field2:"test" AND field3:field4
> > > >
> > > > in this query we said that do search where field3 is equal to
> > > > field4 ...
> > > > (and rest of the query)
> > > >
> > > > if it is not possible, can we ask it as a new feature from Lucene
> > > > developers?
> > >
> > > It could probably be done, but would be quite an expensive query to
> > > place.
> > >
> > > However, a request like this usually tiggers questions about the
> > > design of an index with such needs. There is probably a better
> > > solution. Start by telling us about the data you have inserted to the
> > > index, and what it is you try to achieve.
> > >
> > > In essence, Lucene is not a relational database.
> > >
> > > --
> > > karl
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> >
> > --
> > Regards,
> > Mohammad
> >
>



-- 
Regards,
M. Norouzi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message