lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Glen Newton <glen.new...@gmail.com>
Subject Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Date Thu, 19 Feb 2009 13:54:29 GMT
I will look a little deeper into the information you supplied and
comment, but will suggest this on my initial cursory review:

1 - You have 32GB of memory. Using the 64bit VM, try using a 16GB or 24GB heap;
2 - Turn-on huge pages:
      -XX:+UseLargePages
      -XX:LargePageSizeInBytes=256m
3 - Turn on aggressive opts:
      -XX:+AggressiveOpts
4 - Try using bound threads:
      -XX:+UseBoundThreads
5 - Try using thread-local object allocation:
      -XX:+UseTLAB
6 -  Try Biased Locking:
      -XX:+UseBiasedLocking

I consider #1-3 most likely to help you. I would first try them
separately then in combination. #4-6 I am not sure given your hardware
(of which I am very envious!), but would try nonetheless.

 I am more used to working with quad dual core ia64 machines with 8GB
memory running Linux. And large indexes, with tests up to 1B documents
and ~700GB in size, in a single Lucene index.

Please let me know how these things help, if at all.

thanks,

Glen





2009/2/19 Varun Dhussa <varun@mapmyindia.com>:
> Hi,
>
> The details are as follows:
> Solaris version: Solaris 10 U5 and U6
> For the Java Setup, I have tried with:
> Sun JDK 1.5 (32 & 64)
> Sun JDK 1.6 (32 & 64)
> Heap Space: 2G from 32 bit and 4G for 64 bit (Set the same values for both
> XMS and XMX)
>
> Disk: Tried with ZFS (U6) and UFS (U5)
> I reduced the Inode density on the partition since page faults were
> happening initially. Also, the block size was 8K.
> The following reports are on Solaris 10 U5 with UFS (8K block size).
>
>
> The vmstat report is below:
>
> # vmstat 3
> kthr      memory            page            disk          faults      cpu
> r b w   swap  free  re  mf pi po fr de sr s0 s2 -- --   in   sy   cs us sy
> id
> 0 0 0 43408368 29603112 13 107 4 7 6 0  0  2 -0  0  0  458  589  321  0  0
> 99
> 0 0 0 22517072 30241872 830 7709 0 0 0 0 0 0  0  0  0 2537 30699 3899 36 3
> 61
> 0 0 0 24666416 30483248 860 7942 0 0 0 0 0 0  0  0  0 2235 31886 3362 28 3
> 69
> 0 0 0 24184008 30450728 791 7813 0 0 0 0 0 0  0  0  0 2335 31426 3740 33 3
> 65
> 0 0 0 23472024 30442168 813 7822 0 0 0 0 0 0  0  0  0 2494 31312 3923 35 3
> 62
> 0 0 0 22751768 30297240 787 7654 0 0 0 0 0 0  0  0  0 2498 31820 3892 35 3
> 62
> 0 0 0 25385568 30527600 835 7963 0 0 0 0 0 0  0  0  0 2360 32144 3691 29 3
> 68
> 0 0 0 22741056 30281376 787 7643 0 0 0 0 0 0  0  0  0 2249 30008 3484 37 3
> 60
> 0 0 0 22753800 30373072 847 7923 0 0 0 0 0 0  0  0  0 2501 31375 3854 33 3
> 64
> 0 0 0 23725600 30431312 841 8049 0 0 0 0 0 0  0  0  0 2362 32514 3752 32 3
> 65
> 0 0 0 22743008 30256368 826 7719 0 0 0 0 0 0  0  0  0 2427 32101 3773 34 3
> 63
> 0 0 0 23950336 30405312 818 7925 0 0 0 0 0 0  0  0  0 2337 32095 3606 31 3
> 66
> 0 0 0 22752872 30319312 778 7486 0 0 0 0 0 0  0  0  0 2435 30277 3769 37 3
> 60
> 0 0 0 24657848 30401408 836 7797 0 0 0 0 0 0  0  0  0 2469 31879 3770 32 3
> 65
> 0 0 0 23460408 30331056 776 7560 0 0 0 0 0 0  0  0  0 2461 31344 3883 37 3
> 60
> 0 0 0 22748352 30302248 792 7534 0 0 0 0 0 0  0  0  0 2424 30683 3733 36 3
> 61
> 0 0 0 23953848 30389280 837 7868 0 0 0 0 0 0  0  0  0 2280 30710 3511 34 3
> 63
> 0 0 0 22736592 30249672 810 7716 0 0 0 0 0 0  0  0  0 2345 30980 3593 34 3
> 63
> kthr      memory            page            disk          faults      cpu
> r b w   swap  free  re  mf pi po fr de sr s0 s2 -- --   in   sy   cs us sy
> id
> 0 0 0 23951544 30400016 834 7992 0 0 0 0 0 0  0  0  0 2431 32647 3776 31 3
> 66
> 0 0 0 21797992 30245536 812 7635 0 0 0 0 0 0  0  0  0 2402 30213 3693 39 3
> 59
> 0 0 0 22745264 30307728 781 7539 0 0 0 0 0 0  0  0  0 2437 30744 3860 38 3
> 59
> 0 0 0 22748536 30286456 851 7987 0 0 0 0 0 0  0  0  0 2433 32842 3678 29 3
> 68
> 0 0 0 22747680 30297544 791 7514 0 0 0 0 0 0  0  0  0 2369 30625 3655 38 3
> 59
> 0 0 0 23104848 30309488 793 7524 0 0 0 0 0 0  0  0  0 2566 31190 3942 36 3
> 61
> 0 0 0 23460368 30340688 810 7784 0 0 0 0 0 0  0  0  0 2302 31430 3688 33 3
> 64
> 0 0 0 22031344 30215504 792 7397 0 0 0 0 0 0  0  0  0 2477 30733 3850 39 3
> 58
> 0 0 0 22744168 30277312 834 7829 0 0 0 0 0 0  0  0  0 2370 29926 3592 35 3
> 63
> 0 0 0 24177728 30432600 806 7789 0 0 0 0 0 0  0  0  0 2379 31870 3731 32 3
> 65
> 0 0 0 23467896 30382728 802 7810 0 0 0 0 0 0  0  0  0 2527 31866 3980 33 3
> 64
> 0 0 0 22757912 30331080 823 7847 0 0 0 0 0 0  0  0  0 2496 31580 3871 33 3
> 64
> 0 0 0 22043312 30215904 819 7558 0 0 0 0 0 0  0  0  0 2253 30252 3455 36 3
> 61
> 0 0 0 22744952 30275008 793 7657 0 0 0 0 0 0  0  0  0 2499 31840 3937 36 3
> 61
> 0 0 0 23715200 30419008 843 7906 0 0 0 0 0 0  0  0  0 2404 31741 3707 33 3
> 64
> 0 0 0 21307184 30148472 788 7515 0 0 0 0 0 0  0  0  0 2430 31340 3819 39 3
> 58
> 0 0 0 23459544 30332160 817 7687 0 0 0 0 0 0  0  0  0 2353 31011 3526 34 3
> 63
> 0 0 0 23467304 30376392 819 7846 0 0 0 0 0 0  0  0  0 2522 31918 3933 32 3
> 65
> 0 0 0 23473464 30421992 830 7854 0 0 0 0 0 0  0  0  0 2407 31714 3827 33 3
> 65
> kthr      memory            page            disk          faults      cpu
> r b w   swap  free  re  mf pi po fr de sr s0 s2 -- --   in   sy   cs us sy
> id
> 0 0 0 22508648 30240256 799 7447 0 0 0 0 0 0  0  0  0 2524 30483 3911 39 3
> 58
> 0 0 0 22737496 30276864 837 7731 0 0 0 0 0 0  0  0  0 2310 30289 3473 36 3
> 61
> 0 0 0 22515488 30312728 792 7646 0 0 0 0 0 0  0  0  0 2472 31610 3867 35 3
> 62
> 0 0 0 22756248 30251584 823 7669 0 0 0 0 0 0  0  0  0 2637 31413 4119 36 3
> 60
> 0 0 0 23940168 30379336 833 7973 0 0 0 0 0 0  0  0  0 2240 32325 3472 31 3
> 66
> 0 0 0 24184736 30455928 846 7838 0 0 0 0 0 0  0  0  0 2288 31508 3538 32 3
> 65
> 0 0 0 22751808 30309984 776 7614 0 0 0 0 0 0  0  0  0 2480 31486 3906 36 3
> 61
> 0 0 0 22989624 30309536 813 7626 0 0 0 0 0 0  0  0  0 2537 30347 3988 38 3
> 59
> 0 0 0 22739224 30276936 814 7692 0 0 0 0 0 0  0  0  0 2261 31528 3409 34 3
> 63
> 0 0 0 23236200 30350216 853 7968 0 0 0 0 0 0  0  0  0 2676 32636 4167 32 3
> 65
> 0 0 0 24194016 30506960 836 8005 0 0 0 0 0 0  0  0  0 2222 31503 3382 30 3
> 67
> 0 0 0 23480504 30424536 834 7848 0 0 0 0 0 0  0  0  0 2390 32039 3736 32 3
> 65
> 0 0 0 22739048 30233176 812 7742 0 0 0 0 0 0  0  0  0 2604 31693 4029 35 3
> 62
> 0 0 0 23461664 30358528 805 7700 0 0 0 0 0 0  0  0  0 2463 30936 3922 36 3
> 61
> 0 0 0 22031616 30257808 796 7533 0 0 0 0 0 0  0  0  0 2226 29829 3377 37 3
> 60
> 0 0 0 23458168 30327688 813 7773 0 0 0 0 0 0  0  0  0 2388 31519 3763 33 3
> 64
> 0 0 0 23465184 30371904 834 7796 0 0 0 0 0 0  0  0  0 2510 32101 3933 34 3
> 63
> 0 0 0 22513648 30312240 800 7596 0 0 0 0 0 0  0  0  0 2416 30538 3734 38 3
> 59
> 0 0 0 22272824 30228912 802 7640 0 0 0 0 0 0  0  0  0 2292 31395 3469 34 3
> 63
> kthr      memory            page            disk          faults      cpu
> r b w   swap  free  re  mf pi po fr de sr s0 s2 -- --   in   sy   cs us sy
> id
> 0 0 0 22744032 30253480 830 7758 0 0 0 0 0 0  0  0  0 2568 31896 3935 35 3
> 62
> 0 0 0 23287696 30368432 808 7736 0 0 0 0 0 0  0  0  0 2439 30929 3844 35 3
> 62
> 0 0 0 22038504 30253112 794 7480 0 0 0 0 0 0  0  0  0 2213 29774 3433 37 3
> 60
> 0 0 0 24660144 30480240 846 8103 0 0 0 0 0 0  0  0  0 2548 32825 3944 30 3
> 66
> 0 0 0 24183648 30445120 823 7895 0 0 0 0 0 0  0  0  0 2441 32699 3822 31 3
> 66
> 0 0 0 23469136 30391744 830 7916 0 0 0 0 0 0  0  0  0 2441 31731 3813 32 3
> 65
> 0 0 0 22516312 30289688 796 7592 0 0 0 0 0 0  0  0  0 2318 30310 3616 36 3
> 61
> 0 0 0 23943448 30363248 821 7821 0 0 0 0 0 0  0  0  0 2355 31789 3641 31 3
> 66
> 0 0 0 22742768 30284896 795 7602 0 0 0 0 0 0  0  0  0 2387 30608 3675 37 3
> 60
> 0 0 0 23475456 30416136 852 7991 0 0 0 0 0 0  0  0  0 2489 32066 3871 31 3
> 66
> 0 0 0 22756848 30233968 798 7588 0 0 0 0 0 0  0  0  0 2464 31106 3836 37 3
> 60
>
> Cooltest report said virtually nothing.
>
>                 System Configuration
>
> Host name                       mmiAS02
> System name                     SUNW,SPARC-Enterprise-T5120
> Effective UID                   0
> Cooltst version                 3.0.1
> OS                              Solaris
> OS release                      5.10
> OS version                      Generic_127127-11
> Distro                          Solaris
> BIOS/PROM                       OBP 4.28.0 2008/01/22 21:10
> Memory                          32640 MB
> Chip                            UltraSPARC-T2
> MHz                             1165
> Architecture                    SPARC
> # of Virtual CPUs               64
>  P0: 1165 MHz UltraSPARC-T2
>  P1: 1165 MHz UltraSPARC-T2
>  P2: 1165 MHz UltraSPARC-T2
>  P3: 1165 MHz UltraSPARC-T2
>  P4: 1165 MHz UltraSPARC-T2
>  P5: 1165 MHz UltraSPARC-T2
>  P6: 1165 MHz UltraSPARC-T2
>  P7: 1165 MHz UltraSPARC-T2
>  P8: 1165 MHz UltraSPARC-T2
>  P9: 1165 MHz UltraSPARC-T2
>  P10: 1165 MHz UltraSPARC-T2
>  P11: 1165 MHz UltraSPARC-T2
>  P12: 1165 MHz UltraSPARC-T2
>  P13: 1165 MHz UltraSPARC-T2
>  P14: 1165 MHz UltraSPARC-T2
>  P15: 1165 MHz UltraSPARC-T2
>  P16: 1165 MHz UltraSPARC-T2
>  P17: 1165 MHz UltraSPARC-T2
>  P18: 1165 MHz UltraSPARC-T2
>  P19: 1165 MHz UltraSPARC-T2
>  P20: 1165 MHz UltraSPARC-T2
>  P21: 1165 MHz UltraSPARC-T2
>  P22: 1165 MHz UltraSPARC-T2
>  P23: 1165 MHz UltraSPARC-T2
>  P24: 1165 MHz UltraSPARC-T2
>  P25: 1165 MHz UltraSPARC-T2
>  P26: 1165 MHz UltraSPARC-T2
>  P27: 1165 MHz UltraSPARC-T2
>  P28: 1165 MHz UltraSPARC-T2
>  P29: 1165 MHz UltraSPARC-T2
>  P30: 1165 MHz UltraSPARC-T2
>  P31: 1165 MHz UltraSPARC-T2
>  P32: 1165 MHz UltraSPARC-T2
>  P33: 1165 MHz UltraSPARC-T2
>  P34: 1165 MHz UltraSPARC-T2
>  P35: 1165 MHz UltraSPARC-T2
>  P36: 1165 MHz UltraSPARC-T2
>  P37: 1165 MHz UltraSPARC-T2
>  P38: 1165 MHz UltraSPARC-T2
>  P39: 1165 MHz UltraSPARC-T2
>  P40: 1165 MHz UltraSPARC-T2
>  P41: 1165 MHz UltraSPARC-T2
>  P42: 1165 MHz UltraSPARC-T2
>  P43: 1165 MHz UltraSPARC-T2
>  P44: 1165 MHz UltraSPARC-T2
>  P45: 1165 MHz UltraSPARC-T2
>  P46: 1165 MHz UltraSPARC-T2
>  P47: 1165 MHz UltraSPARC-T2
>  P48: 1165 MHz UltraSPARC-T2
>  P49: 1165 MHz UltraSPARC-T2
>  P50: 1165 MHz UltraSPARC-T2
>  P51: 1165 MHz UltraSPARC-T2
>  P52: 1165 MHz UltraSPARC-T2
>  P53: 1165 MHz UltraSPARC-T2
>  P54: 1165 MHz UltraSPARC-T2
>  P55: 1165 MHz UltraSPARC-T2
>  P56: 1165 MHz UltraSPARC-T2
>  P57: 1165 MHz UltraSPARC-T2
>  P58: 1165 MHz UltraSPARC-T2
>  P59: 1165 MHz UltraSPARC-T2
>  P60: 1165 MHz UltraSPARC-T2
>  P61: 1165 MHz UltraSPARC-T2
>  P62: 1165 MHz UltraSPARC-T2
>  P63: 1165 MHz UltraSPARC-T2
> OS release detail:
> Solaris 10 5/08 s10s_u5wos_10 SPARC Copyright 2008 Sun Microsystems, Inc.
>  All Rights Reserved. Use is subject to license terms. Assembled 24 March
> 2008
>
>                    Workload Measurements
>
> Observed system for             10 min
>       in intervals of        10 sec
> Cycles                          44768051692942
> Instructions                    3980806371547
> CPI                             11.25 **
> FP instructions                 5821938521
> Emulated FP instructions        0
> FP Percentage                    0.1%
> The following applies to the measurement interval with the
> busiest single thread or process:
> Peak thread utilization at      2009-02-13 14:36:16
>  Corresponding file name      1234515976
>  CPU utilization               39.4%
>  Command                      java
>  PID/LWPID                    16396/1
>  Thread utilization           0.6%
> More detail on processes and threads is in data/process.out
>
> **Cycles per Instruction (CPI) is not comparable between UltraSPARC
> T1 and T2 processors and conventional processors. Conventional
> processors execute an idle loop when there is no work to do, so
> CPI may be artificially low, especially when the system is
> somewhat idle. The UltraSPARC T1 and T2 "park" idle threads,
> consuming no energy, when there is no work to do, so CPI may
> be artificially high, especially when the system is somewhat idle.
>
>                    Advice
>
> Floating Point                 GREEN
>  Observed floating point content was not excessive for
>  an UltraSPARC T1 processor. Floating point content is not
>  a limitation for UltraSPARC T2.
>
> Parallelism                    GREEN
>  The observed workload has sufficient threads of execution to
>  efficiently utilize the multiple cores and threads of an
>  UltraSPARC T1 or UltraSPARC T2 processor.
>
>
>
> Varun Dhussa
> Product Architect
> CE InfoSystems (P) Ltd
> http://www.mapmyindia.com
>
>
>
> Glen Newton wrote:
>>
>> Could you give some configuration details:
>> - Solaris version
>> - Java VM version, heap size, and any other flags
>> - disk setup
>>
>> You should also consider using huge pages (see
>>
>> http://zzzoot.blogspot.com/2009/02/java-mysql-increased-performance-with.html)
>>
>> I will also be posting performance gains using huge pages for Java
>> Lucene Linux large scale indexing in the next week or so...
>>
>> -glen
>>
>>
>>
>> 2009/2/18 Varun Dhussa <varun@mapmyindia.com>:
>>
>>>
>>> Hi,
>>>
>>> I have had a bad experience when migrating my application from Intel Xeon
>>> based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search
>>> just
>>> does not perform. A search which took approximately 500 ms takes more
>>> than 6
>>> seconds to execute.
>>>
>>> The index has about 100,000,000 records. So, I tried to split it into 10
>>> indices and used the ParallelSearcher on it, but still got similar
>>> results.
>>>
>>> I am guessing that this is because the distance implementation used by
>>> Lucene requires higher clock speed and can't be parallelized much.
>>>
>>> Please advice
>>>
>>> --
>>> Varun Dhussa
>>> Product Architect
>>> CE InfoSystems (P) Ltd
>>> http://www.mapmyindia.com
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>



-- 

-

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message