lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: java.io.IOException: Map failed
Date Sat, 03 Oct 2015 16:35:33 GMT
bq: Unfortunately I am using a company server and the system admin
refuses to change those settings. For now my only option is to
explicitly use either SimpleFSDirectory or NIOFSDirectory. But at
least it is working!

Then find another employer ;). Really, if the system admin is
unwilling to listen/learn from Uwe's blog, do you really want to
continue working in a place that says, in effect, "I'm going to ignore
your concerns, hobble your ability to work effectively, and still hold
you responsible for the outcome all because we have a prima donna of a
system administrator who won't listen"? I wouldn't be quite that
blunt, but that's sure what I'd be thinking.

Practically, though, you _never_ want to turn this sort of thing into
a confrontation, at least as a first step. So I'd propose a test.
Something on the order of "give me a test machine that we can
experiment with these settings on".  Obviously not prod. From there,
build a test harness (very easy, just something like jMeter). Copy the
exact same index from prod to the test system. Run your stress tests
changing various settings on the test machine. Tune the heck out of
it.

In the end, you'll either
1> demonstrate that these settings are hampering your performance.
or
2> discover that there's something _else_ that's the problem


No matter what, you now have some hard _data_ to feed back to the
people who run things and they can make a rational decision about
whether or not to change whatever you find the root cause is. And I've
rarely found a system admin who really refuses to change things when
presented with a quantifiable cost. After all, having things run
fast/well is in their interests too. The key is presenting _data_
rather than "Some guy on the internet said..." [1]

Let's claim that <1> is the problem. The system administrator now
needs to justify why these settings are necessary to the his boss. And
boss's boss. and.... Perhaps they all think that the performance
slowdown is acceptable for whatever reason. Perhaps they override the
system administrator. Or perhaps the system administrator sees the
results and volunteers to change the settings, even "just for the Solr
machines" voluntarily. Whatever, but you've shifted the argument from
vague generalities (on both sides) to quantifiable costs/benefits that
can be used to make a rational decision.

And if <2> is the case, you haven't made any waves, caused anyone any
embarrassment etc. And you need a test system anyway, 'cause if your
organization expects to change/improve version N+1 of your app without
a test system to do stress tests on, you're just asking for trouble.

[1] "some guy on the internet...". Unless they have worked with Uwe,
they have no clue how very knowledgable he is.

On Sat, Oct 3, 2015 at 4:03 AM, Ziqi Zhang <ziqi.zhang@sheffield.ac.uk> wrote:
> Thanks Uwe
>
> Unfortunately I am using a company server and the system admin refuses to
> change those settings. For now my only option is to explicitly use either
> SimpleFSDirectory or NIOFSDirectory. But at least it is working!
>
>
>
>
> On 01/10/2015 20:53, Uwe Schindler wrote:
>>
>> Hi,
>>
>> You must ask the system administrator to raise those limits. Or use sudo
>> or get root yourself if its your own machine. Those settings cannot be done
>> as normal user because they affect whole system. In general, those settings
>> don't survive reboots so its better to modify corresponding config files in
>> /etc so its applied on system startup. This depends on your Linux
>> distribution, we cannot give any help on this.
>>
>> I would also recommend to review my blog post as stated with URL in the
>> exception message!
>>
>> Uwe
>>
>> Am 1. Oktober 2015 21:25:30 MESZ, schrieb Ziqi Zhang
>> <ziqi.zhang@sheffield.ac.uk>:
>>
>>     Hi,
>>
>>     I have a problem which I think is the same as that described here:
>>
>>
>> http://stackoverflow.com/questions/8892143/error-when-opening-a-lucene-index-map-failed
>>
>>     However the solution does not apply in this case so I am providing
>> more
>>     details and asking again.
>>
>>     The index is created using Solr 5.3
>>
>>     The line of code causing the exception is:
>>
>> ------------------------------------------------------------------------
>>
>>           IndexReader indexReader =
>>     DirectoryReader.open(FSDirectory.open(Paths.get("the_path")));
>>
>>
>>     The exception stacktrace is:
>>
>> ------------------------------------------------------------------------
>>
>>           Exception in thread "main"java.io <http://java.io>.IOException:
>> Map failed:
>>
>> MMapIndexInput(path="/mnt/fastdata/ac1zz/JATE/solr-5.3.0/server/solr/jate/data_aclrd/index/_5t.tvd")
>>     [this may be caused by lack of enough unfragmented virtual address
>> space
>>     or too
>>     restrictive virtual memory limits enforced by the operating
>>     system, preventing us to map a chunk of 434505698 bytes. Please review
>>     'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl
>>     vm.max_map_count'. More information:
>>
>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
>>           atsun.nio.ch
>> <http://sun.nio.ch>.FileChannelImpl.map(FileChannelImpl.java:907)
>>           at
>> org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:265)
>>           at
>>
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:239)
>>           at
>>
>> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:144)
>>           at
>>
>> org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
>>           at
>>
>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
>>           at
>> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
>>           at
>>
>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
>>           at
>>
>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
>>           at
>>
>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
>>           at
>>
>> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
>>           at
>>     org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
>>           atuk.ac
>> <http://uk.ac>.shef.dcs.jate.app.AppATTF.extract(AppATTF.java:39)
>>           atuk.ac
>> <http://uk.ac>.shef.dcs.jate.app.AppATTF.main(AppATTF.java:33)
>>
>>
>>     The suggested solutions as in the exception message do not work in
>> this
>>     case because I am running
>>     the application on a server and I do not have
>>     permissions to change those.
>>
>>     Namely,
>>     -----------
>>           ulimit -v unlimited
>>
>>     prints: "-bash: ulimit: virtual memory: cannot modify limit: Operation
>>     not permitted"
>>
>>     and
>>     -----
>>           sysctl -w vm.max_map_count=10000000
>>
>>     gives:"error: permission denied on key 'vm.max_map_count'"
>>
>>
>>     Is there any other way I can solve this?
>>
>>     Thanks
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>>     To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>     For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> -- Uwe Schindler H.-H.-Meier-Allee 63, 28213 Bremen http://www.thetaphi.de
>
> --
> Ziqi Zhang
> Research Associate
> Department of Computer Science
> University of Sheffield

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message