Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: local policy)
From: "Rob Staveley (Tom)" <rstaveley@seseit.com>
To: <java-user@lucene.apache.org>
Subject: RE: Compound / non-compound index files and SIGKILL
Date: Thu, 8 Jun 2006 08:56:36 +0100
MIME-Version: 1.0
Content-Type: multipart/signed;
	protocol="application/x-pkcs7-signature";
	micalg=SHA1;
	boundary="----=_NextPart_000_000F_01C68AD9.727B0900"
In-Reply-To: <20060607233346.6270D187B1@mail.seseit.com>
Thread-Index: AcaKZxVIA/VO24gqQa+CowBWMMEQuAAG4VlgABJdfRA=
Message-Id: <20060608075627.BA9C4187B1@mail.seseit.com>

------=_NextPart_000_000F_01C68AD9.727B0900
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

OK, I see it now. It all happens in DocumentWriter.invertDocument. I should
have looked at the Lucene source.

IndexWriter.addDocument -> DocumentWriter.addDocument ->
DocumentWriter.invertDocument

Many thanks, Chris. 

Whom should I post to suggest adding a note to
http://lucene.apache.org/java/docs/api/org/apache/lucene/document/Field.html
#Field(java.lang.String,%20java.io.Reader) to say that the Reader is only
actually read from when the document is added? Without internal knowledge,
it isn't obvious. I can see others making the same mistake I did assuming
that the Reader would have fulfilled its purpose when the call to the
constructor returns.

-----Original Message-----
From: Rob Staveley (Tom) [mailto:rstaveley@seseit.com] 
Sent: 08 June 2006 00:34
To: java-user@lucene.apache.org
Subject: RE: Compound / non-compound index files and SIGKILL

> I'm not sure what exactly your process method is doing

In essence it gets text from the content's input stream and writes it to the
PipedWriter and hence to the PipedReader passed to the Field constructor.
The process method for a plain text content handler simply copies from the
input stream and writes to the PipedWriter, but the HTML handler goes
through a few more hoops, involving 3rd party libraries. I favoured the
Reader constructor because it avoided having to load large documents into
RAM. 

> It isn't until you add that document using IndexWriter.addDocument 
> that
anything is read from that Reader

>>From what you are saying, my design is flawed, because the PipedWriter 
>is
necessarily closed before the document is added ("// Close the output stream
to get the PipedReader to see EOF"). If reading only occurs during the
document add, it can only works if the entire content is able to be held in
the PipedReader's input buffer, which presumably would mean effectively
having it all loaded in RAM. Bah! Perhaps that's my problem. If reading
occurs during the document addition and not during the Field construction, I
guess I need to use a temporary file, which I should delete after the
document addition to avoid RAM overhead. The Pipe looked natty, but it was
ill-conceived.

-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: 07 June 2006 20:17
To: java-user@lucene.apache.org
Subject: RE: Compound / non-compound index files and SIGKILL

: However, I'm not sure what to make of:
: --------8<--------
: Thread 3740: (state = BLOCKED)
: - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
: - java.lang.Object.wait() @bci=2, line=474 (Compiled frame)
: Error occurred during stack walking:
: java.lang.NullPointerException
: at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)

crap.  i don't know what that means, but it certainly looks bad ... if the
JVM can't make sense of your stack, i can't imagein how there would be any
hope of your program recovering.

: Does that mean that the PipedReader in the following might persist beyond
: the scope of this code, and be read from only when the Lucene document is
: added to the index?

I'm not sure what exactly your process method is doing, but it is certainly
true that Lucene won't do anything to read from that PipedReader in this
code... all the new Field constructor willdo is hand on to a refrence to
that reader, and all the Document.add call will do is hang on to a refrence
to that Field.  It isn't until you add that document using
IndexWriter.addDocument that anything is read from that Reader.

That's acctually teh whole point of makinga Field from a Reader -- if you're
slurping a big chunk of data off disk, or a network connectin or something,
giving lucene a Reader allows you to pipeline the data all the way to the
Analyzer without ever needinga fully copy of it in memory.

Which means if the thing producing your data and streaming it to that
PipedReader chokes and causes your app to crash, it will crash in the middle
of of writing out your Document.

try buffering all the data for each doc in memory as a String and building a
Field with that -- it may not prevent your app from crashing (that soudns
like a problem somewhere else in your code) but it may prevent your index
from getting corrupt when it does crash.

I say it "may" prevent it -- because if you are multithreading your app then
there's no garuntee that one thread won't cause acrash while another thread
is in the middle of writting data to your index.

:
: --------8<--------
: 	final PipedWriter pw = new PipedWriter();
: 	Thread t = new Thread() {
: 		public void run() {
: 			try {
: 				// Index the body text in the Lucene
: document,
: 				// but do not store it
: 				doc.add(
: 					new Field(
: 						"body"
: 						,new PipedReader(pw)
: 					)
: 				);
: 			}
: 			catch (IOException e) {
: 				e.printStackTrace();
: 			}
: 		}
: 	};
: 	t.start();
:
: 	// Process an input stream for the content handler.
: 	// Tokens extracted from the stream are written to
: 	// the piped writer and hence made available to the
: 	// PipedReader used in the "body" field constructor.
: 	process(is,pw);
:
: 	// Close the output stream to get the PipedReader to see EOF
: 	pw.close();
:
: 	// Join the thread to wait for the field to be added to the
: 	// document
: 	t.join();
:
: 	// Now go on to add other fields (metadata), and then add
: 	// the document to the index...
: --------8<--------
:
:
: -----Original Message-----
: From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
: Sent: 06 June 2006 20:13
: To: java-user@lucene.apache.org
: Subject: RE: Compound / non-compound index files and SIGKILL
:
:
: 1) have you tried forcing a threaddump of the JVM when it hangs to see
what
: it's doing?  (i don't remember which signal it is off the top of my head,
: but even if it's not responding to SIGTERM it might respond to that)
:
: : SIGTERM. I guess I'd feel more confident about using SIGKILL, if I knew
: that
: : the uninterruptible hanged thread was creating a Document, which I could
: : interrupt without corrupting the index, rather than adding the document
to
: : the index, which is liable to result in orphaned files and/or a
corrupted
: : index, if killed.
:
: 2) It's possible that the thread is doing both (creating and adding) at
the
: same time ... if you are Constructing documents using Fields that contain
: Readers you get back from convertors which stream data from complex
: documents as needed, then DocumentWriter may have started to write your
: document, gotten to a Field with a Reader, and then your convertor may be
: choking on something within the source document while it tries to stream
: data to that Reader.
:
:
: 	...just a theory.
:
:
: : -----Original Message-----
: : From: Volodymyr Bychkoviak [mailto:vbychkoviak@i-hypergrid.com]
: : Sent: 06 June 2006 10:54
: : To: java-user@lucene.apache.org
: : Subject: Re: Compound / non-compound index files and SIGKILL
: :
: : If your content handlers should respond quickly then you should move
: : indexing process to separate thread and maintain items in queue.
: :
: : Rob Staveley (Tom) wrote:
: : > This is a real eye-opener, Volodymyr. Many thanks. I guess that means
: : > that my orphan-producing hangs must be addDocument() calls, and not in
: : > the content handlers, as I'd previously assumed. I'll put some debug
: : > before and after my addDocument() calls to confirm (and point my
: : > writer's infoStream to System.out).
: : >
: : > -----Original Message-----
: : > From: Volodymyr Bychkoviak [mailto:vbychkoviak@i-hypergrid.com]
: : > Sent: 05 June 2006 18:33
: : > To: java-user@lucene.apache.org
: : > Subject: Re: Compound / non-compound index files and SIGKILL
: : >
: : > Hi.
: : > My five cents :)
: : >
: : > It might be helpful to know how lucene is working with compound files.
: : > When segment is flushed to disk it is written uncompound and after
: : > that is merged into single .cfs file. If you don't change default
: : > setting for using compound files (which is on) this is only place (I
: : > guess) for these files to appear.
: : >
: : > If you're working with large indexes, than merging segments can take a
: : > while (Maybe here is your problem? :) ) (merging happens on
: : > addDocument() call).  If you will kill indexing process during such
: : > merge you'll get many orphaned files...
: : >
: : > You can just run optimize on this index. You'll get three files:
: : > segments, deletable, <segment>.cfs; you can look name of segment in
: : > 'segments' file. Everything else is 'garbage' - you can delete it.
: : >
: : >
: : > Rob Staveley (Tom) wrote:
: : >
: : >> I've been indexing live data into a compound index from an MTA. I'm
: : >> resolving a bunch of problems unrelated to Lucene (disparate hangs in
: : >> my content handlers). When I get a hang, I typically need to kill my
: : >> daemon, alas more often than not using kill -9 (SIGKILL).
: : >>
: : >> However, these SIGKILLs are leaving large temporary(?) files, which I
: : >>
: : > guess
: : >
: : >> are non-compound index files transiently extracted from the working
: : >> .cfs
: : >> files:
: : >>
: : >> -rw-r--r--    1  373138432 Jun  2 13:42 _18hup.fdt
: : >> -rw-r--r--    1      5054464 Jun  2 13:42 _18hup.fdx
: : >> -rw-r--r--    1              426 Jun  2 13:42 _18hup.fnm
: : >>
: : >> -rw-r--r--    1  457253888 Jun  2 09:22 _15djq.fdt
: : >> -rw-r--r--    1      6205440 Jun  2 09:22 _15djq.fdx
: : >> -rw-r--r--    1              426 Jun  2 09:21 _15djq.fnm
: : >>
: : >> They are left intact after restarting my daemon. Presumably they are
: : >> not treated as being part of the compound index. I see no
: : >> corresponding .cfs file for them.
: : >>
: : >> As a consequence of these - I suspect - I am getting a very large
: : >> overall disk requirement for my index, presumably because of
: : >> replicated field
: : >>
: : > data.
: : >
: : >> My guess is that the field data in the orphaned .fdt files needs to
: : >> be regenerated.
: : >>
: : >> In another index directory from a previous test run (again with
: : >> SIGKILLs),
: : >>
: : > I
: : >
: : >> have 98 GB of index files, with only 12 BG devoted to compound files
: : >> for
: : >>
: : > the
: : >
: : >> field index (.cfs). The rest of the disk space is used by orphaned
: : >> uncompounded index files; I see 51 GB devoted to uncompounded field
: : >> data (.fdt), 13 BG devoted to term positions (.prx) and 13 BG devoted
: : >> to term frequencies (.frq).
: : >>
: : >> Here's my question:
: : >>
: : >> How can I attempt to merge these orphaned into the compound index,
: : >> using IndexWriter.addIndexes(), or would I be foolish attempting
this?
: : >>
: : >>
: : >
: : >
: :
: : --
: : regards,
: : Volodymyr Bychkoviak
: :
: :
:
:
:
: -Hoss
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

------=_NextPart_000_000F_01C68AD9.727B0900
Content-Type: application/x-pkcs7-signature;
	name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKaDCCAj0w
ggGmAhEAzbp/VvDf5LxU/iKss3KqVTANBgkqhkiG9w0BAQIFADBfMQswCQYDVQQGEwJVUzEXMBUG
A1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVibGljIFByaW1hcnkgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNOTYwMTI5MDAwMDAwWhcNMjgwODAxMjM1OTU5WjBfMQsw
CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVi
bGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwgZ8wDQYJKoZIhvcNAQEBBQADgY0A
MIGJAoGBAOUZv22jVmEtmUhx9mfeuY3rt56GgAqRDvo4Ja9GiILlc6igmyRdDR/MZW4MsNBWhBiH
mgabEKFz37RYOWtuwfYV1aioP6oSBo0xrH+wNNePNGeICc0UEeJORVZpH3gCgNrcR5EpuzbJY1zF
4Ncth3uhtzKwezC6Ki8xqu6jZ9rbAgMBAAEwDQYJKoZIhvcNAQECBQADgYEATD+4i8Zo3+5DMw5d
6abLB4RNejP/khv0Nq3YlSI2aBFsfELM85wuxAc/FLAPT/+Qknb54rxK6Y/NoIAK98Up8YIiXbix
3YEjo3slFUYweRb46gVLlH8dwhzI47f0EEA8E8NfH1PoSOSGtHuhNbB7Jbq4046rPzidADQAmPPR
cZQwggNiMIICy6ADAgECAhAL2gsXwT+JjqsJdHq0zi4zMA0GCSqGSIb3DQEBAgUAMF8xCzAJBgNV
BAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE3MDUGA1UECxMuQ2xhc3MgMSBQdWJsaWMg
UHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw05ODA1MTIwMDAwMDBaFw0wODA1MTIy
MzU5NTlaMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1
c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNv
cnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJ
bmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMIGfMA0GCSqGSIb3DQEB
AQUAA4GNADCBiQKBgQC7WkSKBBa7Vf0DeootlE8VeDa4DUqyb5xUv7zodyqdufBou5XZMUFweoFL
uUgTVi3HCOGEQqvAopKrRFyqQvCCDgLpL/vCO7u+yScKXbawNkIztW5UiE+HSr8Z2vkV6A+Hthzj
zMaajn9qJJLj/OBluqexfu/J2zdqyErICQbkmQIDAQABo4GwMIGtMA8GA1UdEwQIMAYBAf8CAQAw
RwYDVR0gBEAwPjA8BgtghkgBhvhFAQcBATAtMCsGCCsGAQUFBwIBFh93d3cudmVyaXNpZ24uY29t
L3JlcG9zaXRvcnkvUlBBMDEGA1UdHwQqMCgwJqAkoCKGIGh0dHA6Ly9jcmwudmVyaXNpZ24uY29t
L3BjYTEuY3JsMAsGA1UdDwQEAwIBBjARBglghkgBhvhCAQEEBAMCAQYwDQYJKoZIhvcNAQECBQAD
gYEAAn2eb0VLOKC43ulTZCG85Ewrjx7+kkCs2Ao5aqEyISwHm6tZ/tJiGn1VOLA3c9z0B2ZjYr3h
U3BSh+eo2FLpWy2q4d7PrDFU1IsZyNgjqO8EKzJ9LBgcyHyJqC538kTRZQpNdLXu0xuSc3QuiTs1
E3LnQDGa07LEq+dWvovj+xUwggS9MIIEJqADAgECAhB1uB0L6Y5EVCCaaM0jkP2tMA0GCSqGSIb3
DQEBBQUAMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1
c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNv
cnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJ
bmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMB4XDTA1MTAwNDAwMDAw
MFoXDTA2MTAxODIzNTk1OVowggEZMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMW
VmVyaVNpZ24gVHJ1c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0
b3J5L1JQQSBJbmNvcnAuIGJ5IFJlZi4sTElBQi5MVEQoYyk5ODEeMBwGA1UECxMVUGVyc29uYSBO
b3QgVmFsaWRhdGVkMTMwMQYDVQQLEypEaWdpdGFsIElEIENsYXNzIDEgLSBOZXRzY2FwZSBGdWxs
IFNlcnZpY2UxGzAZBgNVBAMUElJvYiBTdGF2ZWxleSAoVG9tKTEjMCEGCSqGSIb3DQEJARYUcnN0
YXZlbGV5QHNlc2VpdC5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCm4C+s/DP/
NIW0DrHFueJPsZAZxTzi8Aw2bXOUQbHRDNlSO7nWoDPlkTs5RQl0tsJUDlB5TubjJGWKGBdRePXq
ueK/8yj23cBTzKRTU2gNG6cmmc/f/HwHcuc4MCpW9+okvFCPeaFNzuBPPvegjewz3BL3ewfjeOKr
ly//fr/8JBVW19EgK070C2COZgegYNQ5NeV4Y4oDjqETADQDDnYuTe7EMBuSMysmOSgkCfVzEG4h
8N+1ImW6hWe+bH6Rify+q9eZAQint7sSFGZv3ZbBgmE654FKrvE7O8BOOP807/yaJUBlpDkF/X8G
UwvYNwhH/zbHN1BSCD3tstcSIUGHAgMBAAGjgcswgcgwCQYDVR0TBAIwADBEBgNVHSAEPTA7MDkG
C2CGSAGG+EUBBxcDMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9ycGEw
CwYDVR0PBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMEBggrBgEFBQcDAjAUBgpghkgBhvhFAQYH
BAYWBE5vbmUwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC52ZXJpc2lnbi5jb20vY2xhc3Mx
LmNybDANBgkqhkiG9w0BAQUFAAOBgQCtsGmCX3v78Qn3I5DAo9mJDenolmgmBjmlaSJr1e5nSBLv
CEflpx3FaZuirDQNJ/sTxtUCJhlv/kTLuczeJkSDj1ALhqeYH1GX9T5zFY+3GCQqTJcTIarHSqZC
PQzwv3dwZCRE0fuELb6oobi4kGA5PXbu3+P7upHSBAmN81O00TGCBL8wggS7AgEBMIHhMIHMMRcw
FQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazFG
MEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNvcnAuIEJ5IFJlZi4s
TElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJbmRpdmlkdWFsIFN1
YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkAhB1uB0L6Y5EVCCaaM0jkP2tMAkGBSsOAwIa
BQCgggKyMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA2MDYwODA3
NTYzNVowIwYJKoZIhvcNAQkEMRYEFDU9hLXjWTN4Mmx/eQLECP+bHdKqMGcGCSqGSIb3DQEJDzFa
MFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0G
CCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqGSIb3DQIFMIHyBgkrBgEEAYI3EAQxgeQwgeEwgcwx
FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3Jr
MUYwRAYDVQQLEz13d3cudmVyaXNpZ24uY29tL3JlcG9zaXRvcnkvUlBBIEluY29ycC4gQnkgUmVm
LixMSUFCLkxURChjKTk4MUgwRgYDVQQDEz9WZXJpU2lnbiBDbGFzcyAxIENBIEluZGl2aWR1YWwg
U3Vic2NyaWJlci1QZXJzb25hIE5vdCBWYWxpZGF0ZWQCEHW4HQvpjkRUIJpozSOQ/a0wgfQGCyqG
SIb3DQEJEAILMYHkoIHhMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVy
aVNpZ24gVHJ1c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5
L1JQQSBJbmNvcnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xh
c3MgMSBDQSBJbmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkAhB1uB0L
6Y5EVCCaaM0jkP2tMA0GCSqGSIb3DQEBAQUABIIBAEfFunP0xMg3HRrowzHX9NXm3Wx4A9u9xUGq
AhV9DGqbrIFRARE44oYgWSDEvSavilXPVWyGc7Rj7IjSJLZgR4HR1mAgO5qpoT74SxK5mvU8Q8zz
Az0YqKE9gAtltaTf+Oabd00FyGUrM4yyeiEqABybjemrgqCdeWBNwmBwX//FsqPuvJZeBI85IFeD
Ic+GTxMvyBveFOO7gb2KxpZrGV82+bWljCqQKU/xQqyDdP2PETaOgvNM/HDDp4MmF1yEqEuofiP/
aMLQy/RrajqvHDi1R+HhtUcRM8VIWuVBMajOfzIgnrHxJ8zQhDoHKUAdBoRTV4w6Y2qaZhbYk3Ld
1hQAAAAAAAA=

------=_NextPart_000_000F_01C68AD9.727B0900--