lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Moen (Commented) (JIRA)" <>
Subject [jira] [Commented] (SOLR-3282) Perform Kuromoji/Japanese stability test before 3.6 freeze
Date Tue, 27 Mar 2012 16:00:27 GMT


Christian Moen commented on SOLR-3282:

h3. Test 1: Indexing Japanese Wikipedia

I've extracted text pretty accurately from Japanese Wikipedia and removed all the gory markup
so the content is clean.  There are 1,443,764 documents in total and this is mix of short
and very long documents.

These have been converted this to files in Solr XML format and there is 1,000 documents per

I'm posting using 

curl -s http://localhost:8983/solr/update -H 'Content-type:text/xml; charset=UTF-8' --data-binary

and committing after all the files have been posted with

curl -s http://localhost:8983/solr/update -F 'stream.body= <commit />'

Posting the entire Wikipedia in one file is perhaps a lot faster.

Posting took

real	18m39.206s
user	0m12.682s
sys	0m11.065s

The GC log looks fine.  There wasn't even a full GC probably like to the large heap size.

I'm attaching these files

|| Filename || Description ||
|jawiki-index-gc.log| GC log |
|jawiki-index-gcviewer.png| Screenshot from GCViewer |
|jawiki-index-visualvm.png| Screenshot from VisualVM | 

> Perform Kuromoji/Japanese stability test before 3.6 freeze
> ----------------------------------------------------------
>                 Key: SOLR-3282
>                 URL:
>             Project: Solr
>          Issue Type: Task
>          Components: Schema and Analysis
>    Affects Versions: 3.6, 4.0
>            Reporter: Christian Moen
>            Assignee: Christian Moen
> Kuromoji might be used by many and also in mission critical systems.  I'd like to run
a stability test before we freeze 3.6.
> My thinking is to test the out-of-the-box configuration using fieldtype {{text_ja}} as
> # Index all of Japanese Wikipedia documents (approx. 1.4M documents) in a never ending
> # Simultaneously run many tens of thousands typical Japanese queries against the index
at 3-5 queries per second with highlighting turned on
> While Solr is indexing and searching, I'd like to verify that:
> * Indexing and queries are working as expected
> * Memory and heap usage looks stable over time
> * Garbage collection is overall low over time -- no Full-GC issues
> I'll post findings and results to this JIRA.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message