Hi!
I am trying to use plasma store to reduce the memory usage of a pytorch dataset/dataloader combination, and had 4  questions. I don’t think any of them require pytorch knowledge. If you prefer to comment inline there is a quip with identical content and prettier formatting here https://quip.com/3mwGAJ9KR2HT

*1)* My script starts the plasma-store from python with 200 GB:

nbytes = (1024 ** 3) * 200
_server = subprocess.Popen(["plasma_store", "-m", str(nbytes), "-s", path])
where nbytes is chosen arbitrarily. >From my experiments it seems that one should start the store as large as possible within the limits of dev/shm . I wanted to verify whether this is actually the best practice (it would be hard for my app to know the storage needs up front) and also whether there is an automated way to figure out how much storage to allocate.

*2)* Does plasma store support simultaneous reads? My code, which has multiple clients all asking for the 6 arrays from the plasma-store thousands of times, was segfaulting with different errors, e.g.
Check failed: RemoveFromClientObjectIds(object_id, entry, client) == 1
until I added a lock around my client.get

if self.use_lock: # Fix segfault
    with FileLock("/tmp/plasma_lock"):
        ret = self.client.get(self.object_id)
else:
    ret = self.client.get(self.object_id)

which fixes.

Here is a full traceback of the failure without the lock https://gist.github.com/sshleifer/75145ba828fcb4e998d5e34c46ce13fc
Is this expected behavior?

*3)* Is there a simple way to add many objects to the plasma store at once? Right now, we are considering changing,

oid = client.put(array)
to
oids = [client.put(x) for x in array]

so that we can fetch one entry at a time. but the writes are much slower.

* 3a) Is there a lower level interface for bulk writes?
* 3b) Or is it recommended to chunk the array and have different python processes write simultaneously to make this faster?

*4)* Is there a way to save/load the contents of the plasma-store to disk without loading everything into memory and then saving it to some other format?

Replication

Setup instructions for fairseq+replicating the segfault: https://gist.github.com/sshleifer/bd6982b3f632f1d4bcefc9feceb30b1a
My code is here: https://github.com/pytorch/fairseq/pull/3287

Thanks!
Sam