predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <>
Subject Re: How can I limit RAM usage?
Date Wed, 13 Dec 2017 16:09:09 GMT
Spark uses a minimum of 2g per executor with no data or work. There is always one executor
and one driver with Spark. Welcome to Big Data.

A 4g server is not sufficient to run Spark let alone the rest of the PIO stack. 16g minimum
is my recommendation—and I do mean minimum. Machine learning is not an application you can
squeeze into small RAM machines.

On Dec 13, 2017, at 4:14 AM, Noelia Osés Fernández <> wrote:

I should mention that when I reboot the server and before I start PIO 3GB of RAM are already
used, so there is only 1GB free RAM for PIO. 

On 13 December 2017 at 13:08, Noelia Osés Fernández < <>>

Hi all,

I have a cloud server with 4GB RAM in which I have installed PIO 0.12.0-incubating and a few
other things. However, I'm having trouble running even the smallest examples as it runs out
of RAM (plus the swap is also full).

I have set a limit for ES in jvm.options as follows:

Before training and deploying I limit java's RAM usage as follows:

export JAVA_OPTS="-Xmx1g -Xms1g -Dfile.encoding=UTF-8"
pio train  -- --driver-memory 1G --executor-memory 1G
nohup pio deploy > deploy.out &

Are there any more measures I can take to limit RAM usage?

I would like to know I've done absolutely everything possible before paying to get more RAM
on the server.

Thank you for your help!



Noelia Osés Fernández, PhD
Senior Researcher |
Investigadora Senior <>
+[34] 943 30 92 30
Data Intelligence for Energy and
Industrial Processes | Inteligencia
de Datos para Energía y Procesos

 <>  <>

member of:  <>     <>

Legal Notice - Privacy policy <>

View raw message