Return-Path: 
 <lucene-dev-return-3126-qmlist-jakarta-archive-lucene-dev=nagoya.apache.org@jakarta.apache.org>
Delivered-To: apmail-jakarta-lucene-dev-archive@apache.org
Received: (qmail 88560 invoked from network); 14 Feb 2003 17:14:43 -0000
Received: from exchange.sun.com (192.18.33.10)
  by daedalus.apache.org with SMTP; 14 Feb 2003 17:14:43 -0000
Received: (qmail 757 invoked by uid 97); 14 Feb 2003 17:16:15 -0000
Delivered-To: qmlist-jakarta-archive-lucene-dev@nagoya.betaversion.org
Received: (qmail 750 invoked from network); 14 Feb 2003 17:16:15 -0000
Received: from daedalus.apache.org (HELO apache.org) (208.185.179.12)
  by nagoya.betaversion.org with SMTP; 14 Feb 2003 17:16:15 -0000
Received: (qmail 88304 invoked by uid 500); 14 Feb 2003 17:14:39 -0000
Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
List-Unsubscribe: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
List-Subscribe: <mailto:lucene-dev-subscribe@jakarta.apache.org>
List-Help: <mailto:lucene-dev-help@jakarta.apache.org>
List-Post: <mailto:lucene-dev@jakarta.apache.org>
List-Id: "Lucene Developers List" <lucene-dev.jakarta.apache.org>
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Delivered-To: mailing list lucene-dev@jakarta.apache.org
Received: (qmail 88289 invoked from network); 14 Feb 2003 17:14:39 -0000
Received: from exchange.sun.com (192.18.33.10)
  by daedalus.apache.org with SMTP; 14 Feb 2003 17:14:39 -0000
Received: (qmail 743 invoked by uid 50); 14 Feb 2003 17:16:11 -0000
Date: 14 Feb 2003 17:16:11 -0000
Message-ID: <20030214171611.742.qmail@nagoya.betaversion.org>
From: bugzilla@apache.org
To: lucene-dev@jakarta.apache.org
Cc: 
Subject: DO NOT REPLY [Bug 16719]  -
    java.io.IOException: Pipe closed
X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N
X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://nagoya.apache.org/bugzilla/show_bug.cgi?id=16719>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=16719

java.io.IOException: Pipe closed





------- Additional Comments From Eric.Isakson@sas.com  2003-02-14 17:16 -------
I started seeing this also, it is not being raised as an exception just dumped 
to System.err.

In DocumentWriter.invertDocument(Document doc) when you exceed maxFieldLength 
it does stream.close which closes the reader used for input to HTMLParser. The 
stack trace is generated by ParserThread.run() when the IOException occurs as 
the HTMLParser attempts to write to pipeOut which was closed when the reader 
was closed from the other thread.

I suppose you could have other IO problems that might raise IOException and you 
would want to see those end up in your log so you wouldn't want to not report 
these.

I don't know enough about threading to know if there is some solution to this 
that would allow IndexWriter to wait for the parser to finish or interrupt it 
before closing the reader or for the HTMLParser to notice that its input was 
closed, so I just wanted to share what I found.

maxFieldLength is set in IndexWriter and passed in when you construct the 
DocumentWriter:
  /** The maximum number of terms that will be indexed for a single field in a
    document.  This limits the amount of memory required for indexing, so that
    collections with very large files will not crash the indexing process by
    running out of memory.

    <p>By default, no more than 10,000 terms will be indexed for a field. */
  public int maxFieldLength = 10000;

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org