lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: test buffering causes OOM errors
Date Sat, 21 Jul 2012 09:49:14 GMT
On Sat, Jul 21, 2012 at 5:36 AM, Dawid Weiss <dawid.weiss@gmail.com> wrote:
> Full stack trace would me more helpful

Full stack trace below.

> but I think the problem is that solr
> tests emit megabytes of data to the console and this has to be buffered and
> passed to report listeners.

I've not looked into the test framework architecture and don't
understand why things can't just be streamed - but from my perspective
the most important thing is just getting the logs when a test fails.
Logs are most useful in big stress tests where things are not easily
reproducible or debuggable with a debugger - and these also tend to
produce a lot of logging.

A month or so ago, I also hit issues with tests.iters (I think I got
OOM errors when I cranked it too high?)
so I went back to looping the test myself in a shell script.
I don't know what the exact behavior here is, but what would be most
useful is if the output were reset for each run so that when things
fail you are left with the full log for just the run that failed.

> It is possible to spill to disk of course but not a trivial refactoring.
>
> An easy workaround for now would be to redirect java logging to a file and
> keep the console output smaller.
>
> There is an issue for those Ooms on github but I admit it isn't of super
> high priority for me given the workaround above?

-Yonik
http://lucidimagination.com


[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] 	at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4] 	at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4] 	at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
[junit4:junit4] 	at java.lang.StringBuffer.append(StringBuffer.java:224)
[junit4:junit4] 	at java.io.StringWriter.write(StringWriter.java:84)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.LineBufferWriter.write(LineBufferWriter.java:26)
[junit4:junit4] 	at java.io.Writer.write(Writer.java:96)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.PrefixedWriter.write(PrefixedWriter.java:32)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.PrefixedWriter.write(PrefixedWriter.java:41)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:219)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.SlaveInfo.decodeStreams(SlaveInfo.java:129)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.decodeStreamEvents(TextReport.java:480)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.format(TextReport.java:443)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(TextReport.java:249)
[junit4:junit4] 	at
sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
[junit4:junit4] 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveSuiteEnd(AggregatingListener.java:150)
[junit4:junit4] 	at
sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
[junit4:junit4] 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] 	at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4] 	at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4] 	at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4] 	at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4] 	at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:219)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.SlaveInfo.decodeStreams(SlaveInfo.java:129)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.buildModel(AntXmlReport.java:149)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.onSuiteResult(AntXmlReport.java:95)
[junit4:junit4] 	at
sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
[junit4:junit4] 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveTestEnd(AggregatingListener.java:145)
[junit4:junit4] 	at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
[junit4:junit4] 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.LocalSlaveStreamHandler.pumpEvents(LocalSlaveStreamHandler.java:174)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.LocalSlaveStreamHandler$1.run(LocalSlaveStreamHandler.java:94)
[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] 	at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4] 	at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4] 	at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4] 	at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4] 	at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.processInput(WriterOutputStream.java:283)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:214)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serializeEvents(JsonAggregatedSuiteResultEventAdapter.java:75)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serialize(JsonAggregatedSuiteResultEventAdapter.java:34)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serialize(JsonAggregatedSuiteResultEventAdapter.java:20)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.GsonToMiniGsonTypeAdapterFactory$3.write(GsonToMiniGsonTypeAdapterFactory.java:97)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:512)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonReport.onSuiteResult(JsonReport.java:198)
[junit4:junit4] 	at
sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
[junit4:junit4] 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveTestEnd(AggregatingListener.java:145)
[junit4:junit4] 	at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
[junit4:junit4] 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] 	at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message