predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Re: How can I limit RAM usage?
Date Wed, 13 Dec 2017 16:09:09 GMT
Spark uses a minimum of 2g per executor with no data or work. There is always one executor
and one driver with Spark. Welcome to Big Data.

A 4g server is not sufficient to run Spark let alone the rest of the PIO stack. 16g minimum
is my recommendation—and I do mean minimum. Machine learning is not an application you can
squeeze into small RAM machines.
 

On Dec 13, 2017, at 4:14 AM, Noelia Osés Fernández <noses@vicomtech.org> wrote:

I should mention that when I reboot the server and before I start PIO 3GB of RAM are already
used, so there is only 1GB free RAM for PIO. 
 

On 13 December 2017 at 13:08, Noelia Osés Fernández <noses@vicomtech.org <mailto:noses@vicomtech.org>>
wrote:

Hi all,

I have a cloud server with 4GB RAM in which I have installed PIO 0.12.0-incubating and a few
other things. However, I'm having trouble running even the smallest examples as it runs out
of RAM (plus the swap is also full).

I have set a limit for ES in jvm.options as follows:
-Xms512m
-Xmx512m

Before training and deploying I limit java's RAM usage as follows:

export JAVA_OPTS="-Xmx1g -Xms1g -Dfile.encoding=UTF-8"
pio train  -- --driver-memory 1G --executor-memory 1G
nohup pio deploy > deploy.out &

Are there any more measures I can take to limit RAM usage?

I would like to know I've done absolutely everything possible before paying to get more RAM
on the server.

Thank you for your help!
Noelia


 



-- 
 <http://www.vicomtech.org/>

Noelia Osés Fernández, PhD
Senior Researcher |
Investigadora Senior

noses@vicomtech.org <mailto:noses@vicomtech.org>
+[34] 943 30 92 30
Data Intelligence for Energy and
Industrial Processes | Inteligencia
de Datos para Energía y Procesos
Industriales

 <https://www.linkedin.com/company/vicomtech>  <https://www.youtube.com/user/VICOMTech>
 <https://twitter.com/@Vicomtech_IK4>

member of:  <http://www.graphicsmedia.net/>     <http://www.ik4.es/>

Legal Notice - Privacy policy <http://www.vicomtech.org/en/proteccion-datos>

Mime
View raw message