lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Trejkaz <>
Subject Bug? "TokenStream contract violation: close() call missing", but close() call was not actually missing
Date Tue, 26 May 2015 01:44:53 GMT
Hi all.

I found a bug where Tokenizer seems to be complaining about close()
not being called, but on further investigation it looks like we're not
the ones who are opening it, but rather it's being opened by Lucene.

Firstly, the tests rely on this method, which may or may not be
correct, but I tried to follow the docs:

    private List<String> consume(TokenStream stream) throws Exception {
        ImmutableList.Builder<String> tokens = ImmutableList.builder();

        //The consumer calls reset().
        try {
            //The consumer retrieves attributes from the stream and
stores local references to all attributes it wants to access.
            CharTermAttribute termAttribute =
            //The consumer calls incrementToken() until it returns
false consuming the attributes after each call.
            while (stream.incrementToken()) {
            //The consumer calls end() so that any end-of-stream
operations can be performed.
        } finally {
            //The consumer calls close() to release any resource when
finished using the TokenStream.


I also tested reusing the stream for multiple readers, which seems to work:

    public void testStreamReuse() throws Exception {
        Tokenizer stream = new StandardTokenizer(new StringReader("reader #1"));
        assertThat(consume(stream), contains("reader", "1"));

        stream.setReader(new StringReader("reader 2"));
        assertThat(consume(stream), contains("reader", "2"));

But if the reader throws an exception:

    public void testStreamReuseAfterFailure() throws Exception {
        class FailingReader extends Reader {
            public int read(@NotNull char[] buffer, int off, int len)
throws IOException {
                throw new IOException("Synthetic exception");

            public void close() throws IOException {
                throw new IOException("Synthetic exception");

        Tokenizer stream = new StandardTokenizer(new FailingReader());
        try {
            fail("Expected IOException");
        } catch (IOException e) {
            // Expected

        stream.setReader(new StringReader("working reader"));
        assertThat(consume(stream), contains("working", "reader"));

This fails:

    java.lang.IllegalStateException: TokenStream contract violation:
close() call missing
            at org.apache.lucene.analysis.Tokenizer.setReader(
            at TestStandardTokenizerStandalone.testStreamReuseAfterFailure(

As far as I can see from the code, I am calling close() correctly.
Maybe I'm not, though, so is this a bug in my code or a bug in Lucene?


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message