airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bandaru, Vivek Shresta" <vivb...@iu.edu>
Subject Validating user storage quota
Date Fri, 31 Jul 2020 01:31:20 GMT
Hi All,

I’ve been working on various approaches to validate the storage quota for a gateway user.
Though the storage for experiments is taken care by the gateway, the storage quota validation
needs to be done on Airavata. This way, the gateways need not develop their own mechanism
to track the quotas for every user.

The only way as far as I know for a user to add his files to the storage are through Storage
page and the Create a New experiment page where they can add input files. Utilizing the existing
datastore.size(‘directory_path’) api in the Django-portal, the amount of space used by
the user can be tracked, and when the Gateway sends this data to Airavata through a new API,
Airavata validates the space utilized with the Storage Quota specified in the Storage Preference.
Currently, whenever a user tries to create a new Experiment, even before the input files are
added to the Experiment’s directory, I’ve added the validation and when the storage limit
exceeds, this is how the new experiment page is rendered:

[A screenshot of a cell phone  Description automatically generated]

Questions:


  1.  Since UserStoragePreference feature isn’t being used, I’m not really sure if I need
to use the StorageResource mentioned in UserStoragePreference. Once the gateways start using
this (not sure if it’s only Django-Portal which isn’t using UserStoragePreference), then
the necessary code can be added.
I would like to know the teams thoughts on this.
  2.  I feel that maybe a better way to approach this problem would be if Airavata tracks
the size of each and every file going into the user’s directory instead of the gateway telling
the size of the user’s directory. The amount of space used by an individual user can be
tracked in the USERS table through a new column. This is where I’m currenty stuck.
Airavata currently uses SCPFileTransferWrapper.java for uploading the input files onto the
compute resources. There is no means of knowing the file size being transferred.
Using the current implementation, one possible approach that I could think of is to download
the file from the gateway to a temporary directory where Airavata is deployed, and through
that URI, the file size can be retrieved. But this would involve downloading every input file
on all the gateways of all the users onto a temporary location on Airavata and deleting them.
I’m currently looking for a better alternative to track the file size given a URI.
  3.  Once MFT is integrated into Airavata, Airavata will not use the existing file transfer
protocols. So, does it make sense to develop this tracking mechanism on MFT and for now, use
the above mentioned validation mechanism(Gateway gets the size of the user directory)?

Any pointers are appreciated.
Thanks for reading.

Regards,
Vivek.
Mime
View raw message