lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanyi <need4...@yahoo.com>
Subject Re: What is the best file system for Lucene?
Date Tue, 30 Nov 2004 13:50:15 GMT
> simply load your index into a
> RAMDirectory instead of using FSDirectory. 

I have 3GByte RAM and my index is 3GByte big currently. (it'll be soon about 4GByte)
So, I have to find out this another way.

> First off, 1.8GHz Pentium-M machines are supposed to run at about the
> speed of a 2.4GHz machine.  The clock speeds on the mobile chips are
> lower, but they tend to perform much better than rated.   I recommend
> you take a general benchmark of both machines testing both disk speed
> and cpu speed to get a baseline performance comparision.

I think that it a good general benchmark that almost everything runs at least twice as fast
on the
3.0GHz P4 except lucene search.

I can tell one more interesting info:
I have a MySQL table with ~20million records.
I throw a DROP INDEX on that table, MySQL rebuilds the whole huge table into a tempfile.
It completes in 30 minutes on both systems.
It doesn't matter again that the 15kRPM U320 HDD is 2x-3x as fast.
Very surprising again.
Hmm... reiserfs must be very-very slow, or I'm completly lost :)

> I also suggest turning of HT for your benchmarks and performance testing.

I'll try this later and I really hope it won't be the reason.

> Secondly, while the second machine appears to be twice as fast, the
> disk could actually perform slower on the Linux box, especially if the
> notebook drive has a big (8M) cache like most 7200RPM ata disk drives
> do. 

Both drives have 8M cache.

> I imagine that if you hit the index with lots of simultaneous
> searches, that the Linux box would hold its own for much longer than
> the XP box simply due to the random seek performance of the scsi disk
> combined with scsi command queueing.

Are you saying that SCSI command queuing wastes more time than a 15kRPM 3.9ms HDD can gain
over a
7.2kRPM 8-9ms HDD?
It sounds terrible and I hope it isn't true.

> RAM speed is a factor too.  Is the p4 a xeon processor?  The older HT
> xeons have a much slower bus than the newer p4-m processors.  Memory
> speed will be affected accordingly.

It is not a Xeon, just a P4 3.0GHz HT.

> I haven't heard of a hard disk referred to as a winchester disk in a
> very long time :)

;)

> Once you have an idea of how the two machines actually compare
> performance-wise, you can then judge how they perform index
> operations.

Lucene indexing completes in 13-15 hours on the desktop system while it completes in about
29-33
hours on the notebook.

Now, combine it with the DROP INDEX tests completing in the same amount of time on both and
find
out why is the search only slightly faster :)

> Until then, all your measurements are subjective and you
> don't gain much by comparing the two indexing processes.

I'm worried about searching. Indexing is a lot faster on the desktop config.

Regards,
Sanyi



		
__________________________________ 
Do you Yahoo!? 
All your favorites on one personal page  Try My Yahoo!
http://my.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message