jena-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikael Pesonen <mikael.peso...@lingsoft.fi>
Subject Re: Limit memory usage of Fuseki server?
Date Tue, 10 Apr 2018 13:26:29 GMT

We still have this issue:

Apr 10 15:54:29 -- kernel: Out of memory: Kill process 17136 (java) 
score 770 or sacrifice child
Apr 10 15:54:29 -- kernel: Killed process 17136 (java) 
total-vm:9760620kB, anon-rss:3536204kB, file-rss:0kB
Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Main process 
exited, code=killed, status=9/KILL
Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Unit entered 
failed state.
Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Failed with 
result 'signal'.

Any idea how to solve please?

On 6.2.2018 19:34, Andy Seaborne wrote:
> In this case, the details are on the command line:
>
>> ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar
>> /opt/insight/jena/fuseki-server.jar --update --port 3030
>> --loc=/opt/insight/jena_data /ds
>
>     Andy
>
> On 06/02/18 16:11, Laura Morales wrote:
>> You should have another file for the dataset configuration too, 
>> inside "run/configuration/".
>>
>> Sent: Tuesday, February 06, 2018 at 4:13 PM
>> From: "Mikael Pesonen" <mikael.pesonen@lingsoft.fi>
>> To: users@jena.apache.org
>> Subject: Re: Limit memory usage of Fuseki server?
>> # Licensed under the terms of http://www.apache.org/licenses/LICENSE-2.0
>>
>> ## Fuseki Server configuration file.
>>
>> @prefix :        <#> .
>> @prefix fuseki: 
>> <http://jena.apache.org/fuseki#[http://jena.apache.org/fuseki#]> .
>> @prefix rdf: 
>> <http://www.w3.org/1999/02/22-rdf-syntax-ns#[http://www.w3.org/1999/02/22-rdf-syntax-ns#]>

>> .
>> @prefix rdfs: 
>> <http://www.w3.org/2000/01/rdf-schema#[http://www.w3.org/2000/01/rdf-schema#]>

>> .
>> @prefix ja: 
>> <http://jena.hpl.hp.com/2005/11/Assembler#[http://jena.hpl.hp.com/2005/11/Assembler#]>

>> .
>>
>> [] rdf:type fuseki:Server ;
>>     # Example::
>>     # Server-wide query timeout.
>>     #
>>     # Timeout - server-wide default: milliseconds.
>>     # Format 1: "1000" -- 1 second timeout
>>     # Format 2: "10000,60000" -- 10s timeout to first result,
>>     #                            then 60s timeout for
the rest of query.
>>     #
>>     # See javadoc for ARQ.queryTimeout for details.
>>     # This can also be set on a per dataset basis in the dataset 
>> assembler.
>>     #
>>     # ja:context [ ja:cxtName "arq:queryTimeout" ;  ja:cxtValue 
>> "30000" ] ;
>>
>>     # Add any custom classes you want to load.
>>     # Must have a "public static void init()" method.
>>     # ja:loadClass "your.code.Class" ;
>>
>>     # End triples.
>>     .
>>
>>
>>
>> On 6.2.2018 16:58, Lorenz Buehmann wrote:
>>> I guess Andy was hoping to see the Fuseki config file(s)
>>>
>>>
>>> On 06.02.2018 15:56, Mikael Pesonen wrote:
>>>> Is it this one?
>>>>
>>>> File: /lib/systemd/system/apache-jena-fuseki.service:
>>>>
>>>> [Unit]
>>>>
>>>> Description=Apache Jena Fuseki
>>>>
>>>>
>>>> [Service]
>>>>
>>>> Type=simple
>>>>
>>>> User=fuseki
>>>>
>>>> Environment=JAVA_HOME=/usr/lib/jvm/java-8-oracle/
>>>>
>>>> Environment=FUSEKI_HOME=/opt/insight/jena/
>>>>
>>>> Environment=FUSEKI_BASE=/opt/insight/jena/run
>>>>
>>>> ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar
>>>> /opt/insight/jena/fuseki-server.jar --update --port 3030
>>>> --loc=/opt/insight/jena_data /ds
>>>>
>>>>
>>>> [Install]
>>>>
>>>> WantedBy=multi-user.target
>>>>
>>>>
>>>> File: /lib/systemd/system/apache-jena-fuseki.path
>>>>
>>>> [Path]
>>>>
>>>> PathExistsGlob=/var/crash/*.crash
>>>>
>>>> Unit=apache-jena-fuseki.service
>>>>
>>>>
>>>>
>>>> On 6.2.2018 16:46, Andy Seaborne wrote:
>>>>> What is the dataset/service setup?
>>>>>
>>>>> On 06/02/18 14:33, Mikael Pesonen wrote:
>>>>>> I'm not good with Linux so don't know what setup means. We have
>>>>>> Ubuntu 14.04, Jena Fuseki 3.6.0 with default config.ttl and running
>>>>>> as a service.
>>>>>>
>>>>>> Test server which is having both issues is running on Upcloud:
>>>>>>
>>>>>> $ cat /proc/meminfo
>>>>>> MemTotal:        8692992 kB
>>>>>> MemFree:         6716504 kB
>>>>>> Buffers:           50052 kB
>>>>>> Cached:           508684 kB
>>>>>> SwapCached:            0 kB
>>>>>> Active:          1385768 kB
>>>>>> Inactive:         469640 kB
>>>>>> Active(anon):    1300740 kB
>>>>>> Inactive(anon):     9076 kB
>>>>>> Active(file):      85028 kB
>>>>>> Inactive(file):   460564 kB
>>>>>> Unevictable:           0 kB
>>>>>> Mlocked:               0 kB
>>>>>> SwapTotal:             0 kB
>>>>>> SwapFree:              0 kB
>>>>>> Dirty:                48 kB
>>>>>> Writeback:             0 kB
>>>>>> AnonPages:       1296688 kB
>>>>>> Mapped:           125308 kB
>>>>>> Shmem:             13128 kB
>>>>>> Slab:              30824 kB
>>>>>> SReclaimable:      16120 kB
>>>>>> SUnreclaim:        14704 kB
>>>>>> KernelStack:        1520 kB
>>>>>> PageTables:        12920 kB
>>>>>> NFS_Unstable:          0 kB
>>>>>> Bounce:                0 kB
>>>>>> WritebackTmp:          0 kB
>>>>>> CommitLimit:     4346496 kB
>>>>>> Committed_AS:    1669692 kB
>>>>>> VmallocTotal:   34359738367 kB
>>>>>> VmallocUsed:       34708 kB
>>>>>> VmallocChunk:   34359694588 kB
>>>>>> HardwareCorrupted:     0 kB
>>>>>> AnonHugePages:   1193984 kB
>>>>>> HugePages_Total:       0
>>>>>> HugePages_Free:        0
>>>>>> HugePages_Rsvd:        0
>>>>>> HugePages_Surp:        0
>>>>>> Hugepagesize:       2048 kB
>>>>>> DirectMap4k:       46960 kB
>>>>>> DirectMap2M:     2574336 kB
>>>>>> DirectMap1G:     6291456 kB
>>>>>>
>>>>>> Anything else?
>>>>>>
>>>>>> The setup in which I have to concurrency error every time, is apache
>>>>>> web server running php, which is calling Fuseki GSP with curl. With
>>>>>> 4 concurrent tests i can't reproduce the error with some trying,
>>>>>> with 5 concurrent tests error occurs every time.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 6.2.2018 16:14, Andy Seaborne wrote:
>>>>>>> I don't know what's going on.  I can't start to try to reproduce
it
>>>>>>> because I don't know what the trigger is.  It might be a previously
>>>>>>> corrupted database or something messing with the DB behind the
>>>>>>> server the effect is right but it might well be something else.
>>>>>>>
>>>>>>> ajs6f asked for details of the setup - could you provide those
>>>>>>> please? and a complete minimal example.  The fact you can't
create
>>>>>>> one points towards an environment/config/hardware issue, not
a code
>>>>>>> bug.
>>>>>>>
>>>>>>>      Andy
>>>>>>>
>>>>>>> On 06/02/18 13:14, Mikael Pesonen wrote:
>>>>>>>> Do you have now enough info for figuring out what's going
on? It's
>>>>>>>> a bug that can be fixed?
>>>>>>>>
>>>>>>>> I guess this is not related to concurrency issue I sent on
another
>>>>>>>> thread? Both issues are happening on same setup.
>>>>>>>>
>>>>>>>> Br
>>>>>>>>
>>>>>>>> On 6.2.2018 14:18, Andy Seaborne wrote:
>>>>>>>>> Expections in alloc-write seem to be a cause of you rother
issues
>>>>>>>>> so, maybe, this is the same rot cause with a different
>>>>>>>>> manifestation.  It would have been useful to know this
at the
>>>>>>>>> start of the thread. It could be the broken alloc-write
is simply
>>>>>>>>> asking for a huge amount of RAM becaue it says:
>>>>>>>>>
>>>>>>>>> report_vm_out_of_memory
>>>>>>>>>
>>>>>>>>> so it is not the system killing Fuseki, its the JVM exiting.
>>>>>>>>>
>>>>>>>>> With a single TDB, 12G is way too much heap (try 2G)
but the
>>>>>>>>> figure may include mapped files.
>>>>>>>>>
>>>>>>>>>      Andy
>>>>>>>>>
>>>>>>>>> On 06/02/18 09:56, Mikael Pesonen wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> ok I'll try that. I had out of memory error on other
server now.
>>>>>>>>>> Is there a place I could post log files, if they
help?
>>>>>>>>>>
>>>>>>>>>> [2018-02-06 11:48:51] BindingTDB ERROR get1(?o)
>>>>>>>>>> org.apache.jena.tdb.base.file.FileException: In the
middle of an
>>>>>>>>>> alloc-write
>>>>>>>>>>       at
>>>>>>>>>> org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:311)

>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>> [2018-02-06 11:48:52] BindingTDB ERROR get1(?o)
>>>>>>>>>> java.nio.BufferOverflowException
>>>>>>>>>>       at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:214)
>>>>>>>>>>       at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>>>>>>>>>>       at
>>>>>>>>>> sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741)

>>>>>>>>>>
>>>>>>>>>>       at 
>>>>>>>>>> sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
>>>>>>>>>>       at
>>>>>>>>>> org.apache.jena.tdb.base.file.BufferChannelFile.read(BufferChannelFile.java:112)

>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Stack: [0x00007f1956dda000,0x00007f1956edb000],
>>>>>>>>>> sp=0x00007f1956ed9720, free space=1021k
>>>>>>>>>> Native frames: (J=compiled Java code, j=interpreted,
Vv=VM code,
>>>>>>>>>> C=native code)
>>>>>>>>>> V  [libjvm.so+0xaba7ea] VMError::report_and_die()+0x2ba
>>>>>>>>>> V  [libjvm.so+0x4f9e3b] report_vm_out_of_memory(char
const*,
>>>>>>>>>> int, unsigned long, VMErrorType, char const*)+0x8b
>>>>>>>>>> V  [libjvm.so+0x91b613] os::Linux::commit_memory_impl(char*,
>>>>>>>>>> unsigned long, bool)+0x103
>>>>>>>>>> V  [libjvm.so+0x91b6dc] os::pd_commit_memory(char*,
unsigned
>>>>>>>>>> long, bool)+0xc
>>>>>>>>>> V  [libjvm.so+0x91518a] os::commit_memory(char*,
unsigned long,
>>>>>>>>>> bool)+0x2a
>>>>>>>>>> V  [libjvm.so+0x9198af] os::pd_create_stack_guard_pages(char*,
>>>>>>>>>> unsigned long)+0x7f
>>>>>>>>>> V  [libjvm.so+0xa6049e] 
>>>>>>>>>> JavaThread::create_stack_guard_pages()+0x5e
>>>>>>>>>> V  [libjvm.so+0xa69e14]  JavaThread::run()+0x34
>>>>>>>>>> V  [libjvm.so+0x91d9d8] java_start(Thread*)+0x108
>>>>>>>>>> C  [libpthread.so.0+0x8182]  start_thread+0xc2
>>>>>>>>>>
>>>>>>>>>> On 6.2.2018 10:57, Laura Morales wrote:
>>>>>>>>>>>> kernel: Killed process 29037 (java) total-vm:12745796kB
>>>>>>>>>>> 12.7GB? Used by Fuseki? Or does this account
other processes as
>>>>>>>>>>> well? There is no way Fuseki is using 12GB of
memory for a few
>>>>>>>>>>> thousands of triples, unless maybe memory is
leaked during
>>>>>>>>>>> update operations. You should try try:
>>>>>>>>>>>
>>>>>>>>>>> 1. start Fuseki with your database
>>>>>>>>>>> 2. run a lot of live update queries (insert/delete/update
data)
>>>>>>>>>>> and monitor how your VM responds
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sent: Tuesday, February 06, 2018 at 9:22 AM
>>>>>>>>>>> From: "Mikael Pesonen" <mikael.pesonen@lingsoft.fi>
>>>>>>>>>>> To: users@jena.apache.org
>>>>>>>>>>> Subject: Re: Limit memory usage of Fuseki server?
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> config.ttl is defaut, Fuseki is ran as service,
here is the log
>>>>>>>>>>> from
>>>>>>>>>>> around the kill (notice fairly high VM)
>>>>>>>>>>>
>>>>>>>>>>>    kernel: Out of memory: Kill process 29037
(java) score 
>>>>>>>>>>> 807 or
>>>>>>>>>>> sacrifice child
>>>>>>>>>>>    kernel: Killed process 29037 (java) total-vm:12745796kB,
>>>>>>>>>>> anon-rss:3458140kB, file-rss:0kB
>>>>>>>>>>>
>>>>>>>>>>> systemd[1]: apache-jena-fuseki.service: Main
process exited,
>>>>>>>>>>> code=killed, status=9/KILL
>>>>>>>>>>>    systemd[1]: apache-jena-fuseki.service:
Unit entered failed
>>>>>>>>>>> state.
>>>>>>>>>>>    systemd[1]: apache-jena-fuseki.service:
Failed with result
>>>>>>>>>>> 'signal'.
>>>>>>>>>>>
>>>>>>>>>>> Usage is not much, around few thousands of triplets,
post, 
>>>>>>>>>>> get and
>>>>>>>>>>> delete, using Fuseki's GSP over HTTP. Uptime
was around one 
>>>>>>>>>>> week.
>>>>>>>>>>>
>>>>>>>>>>> Sorry can't think of anything else to tell about
the system.
>>>>>>>>>>>
>>>>>>>>>>> Br
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 5.2.2018 18:08, ajs6f wrote:
>>>>>>>>>>>> Can you tell us more about your configuration,
data, workload,
>>>>>>>>>>>> and the timing of the problem?
>>>>>>>>>>>>
>>>>>>>>>>>> ajs6f
>>>>>>>>>>>>
>>>>>>>>>>>>> On Feb 5, 2018, at 7:08 AM, Laura Morales
<lauretas@mail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can tell the Kernel to limit memory
usage per process or
>>>>>>>>>>>>> per user. If you set the limit too low
your process will be
>>>>>>>>>>>>> killed because it can't allocate more
memory, but you can
>>>>>>>>>>>>> tell the Kernel to only limit resident
memory while using up
>>>>>>>>>>>>> all the virtual memory it wants. You
can do this with 
>>>>>>>>>>>>> cgroups.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sent: Monday, February 05, 2018 at 12:56
PM
>>>>>>>>>>>>> From: "Mikael Pesonen" <mikael.pesonen@lingsoft.fi>
>>>>>>>>>>>>> To: users@jena.apache.org
>>>>>>>>>>>>> Subject: Limit memory usage of Fuseki
server?
>>>>>>>>>>>>> We have a cloud server with 4gb of memory
and after a while
>>>>>>>>>>>>> system is
>>>>>>>>>>>>> killing Fuseki server java process for
using up all memory.
>>>>>>>>>>>>> Is it
>>>>>>>>>>>>> possible to limit memory usage, and what
would be the
>>>>>>>>>>>>> performance
>>>>>>>>>>>>> penalty for doing that?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Lingsoft - 30 years of Leading Language
Management
>>>>>>>>>>>>>
>>>>>>>>>>>>> www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]][http://www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]]]

>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Speech Applications - Language Management
- Translation -
>>>>>>>>>>>>> Reader's and Writer's Tools - Text Tools
- E-books and 
>>>>>>>>>>>>> M-books
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mikael Pesonen
>>>>>>>>>>>>> System Engineer
>>>>>>>>>>>>>
>>>>>>>>>>>>> e-mail: mikael.pesonen@lingsoft.fi
>>>>>>>>>>>>> Tel. +358 2 279 3300
>>>>>>>>>>>>>
>>>>>>>>>>>>> Time zone: GMT+2
>>>>>>>>>>>>>
>>>>>>>>>>>>> Helsinki Office
>>>>>>>>>>>>> Eteläranta 10
>>>>>>>>>>>>> FI-00130 Helsinki
>>>>>>>>>>>>> FINLAND
>>>>>>>>>>>>>
>>>>>>>>>>>>> Turku Office
>>>>>>>>>>>>> Kauppiaskatu 5 A
>>>>>>>>>>>>> FI-20100 Turku
>>>>>>>>>>>>> FINLAND
>>>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Lingsoft - 30 years of Leading Language Management
>>>>>>>>>>>
>>>>>>>>>>> www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]]

>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Speech Applications - Language Management - Translation
-
>>>>>>>>>>> Reader's and Writer's Tools - Text Tools - E-books
and M-books
>>>>>>>>>>>
>>>>>>>>>>> Mikael Pesonen
>>>>>>>>>>> System Engineer
>>>>>>>>>>>
>>>>>>>>>>> e-mail: mikael.pesonen@lingsoft.fi
>>>>>>>>>>> Tel. +358 2 279 3300
>>>>>>>>>>>
>>>>>>>>>>> Time zone: GMT+2
>>>>>>>>>>>
>>>>>>>>>>> Helsinki Office
>>>>>>>>>>> Eteläranta 10
>>>>>>>>>>> FI-00130 Helsinki
>>>>>>>>>>> FINLAND
>>>>>>>>>>>
>>>>>>>>>>> Turku Office
>>>>>>>>>>> Kauppiaskatu 5 A
>>>>>>>>>>> FI-20100 Turku
>>>>>>>>>>> FINLAND
>>
>> -- 
>> Lingsoft - 30 years of Leading Language Management
>>
>> www.lingsoft.fi[http://www.lingsoft.fi]
>>
>> Speech Applications - Language Management - Translation - Reader's 
>> and Writer's Tools - Text Tools - E-books and M-books
>>
>> Mikael Pesonen
>> System Engineer
>>
>> e-mail: mikael.pesonen@lingsoft.fi
>> Tel. +358 2 279 3300
>>
>> Time zone: GMT+2
>>
>> Helsinki Office
>> Eteläranta 10
>> FI-00130 Helsinki
>> FINLAND
>>
>> Turku Office
>> Kauppiaskatu 5 A
>> FI-20100 Turku
>> FINLAND
>>

-- 
Lingsoft - 30 years of Leading Language Management

www.lingsoft.fi

Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text
Tools - E-books and M-books

Mikael Pesonen
System Engineer

e-mail: mikael.pesonen@lingsoft.fi
Tel. +358 2 279 3300

Time zone: GMT+2

Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND

Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND


Mime
View raw message