arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tanveer Ahmad - EWI <>
Subject Running plasma_store_server (in background) on each Spark worker node
Date Wed, 10 Jun 2020 21:22:51 GMT
Hi all,

I want to run an external command (plasma_store_server -m 3000000000 -s /tmp/store0 &)
in the background on each worker node of my Spark cluster<>.
So that that external process should be running during the whole Spark job.

The plasma_store_server process is used for storing and retrieving Apache Arrow data in Apache

I am using PySpark for Spark programming and SLURM for Spark cluster<>

Any help will be highly appreciated!


Tanveer Ahmad

View raw message