airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miller, Mark" <>
Subject RE: Saving the content of the STDOUT and STDERR to database
Date Sun, 11 Jan 2015 17:59:53 GMT
this technique has worked well for cipres. We refer all of our files to pointers.
We also use a hash key so there are no stored duplicate files.

Both steps save costly database space. Paul Hoover has done this for us, and he is available
should you need advice.


From: Pamidighantam, Sudhakar V []
Sent: Friday, January 09, 2015 7:15 PM
Subject: Re: Saving the content of the STDOUT and STDERR to database

If the STDOUT becomes large it is not good to store in a database table. Perhaps the data
base can contain a pointer(URI)  to the file.

On Jan 9, 2015, at 5:01 PM, Chathuri Wimalasena <<>>

Hi All,

At the moment, we are saving the content of the STDOUT and STDERR in to DATA_TRANSFER_DETAIL
table in the database and when retrieving the whole experiment object, DataTransferDetail
object is also included. Since we no longer using wrapper scripts for different applications
(instead using the module itself), now the most of the applications write the output to STDOUT.
This output might contain special characters which might ruin the json response at the client
side. We face this issue with Gamess.

Now we treat STDOUT and STDERR as normal outputs which will be available to users at the end
of the experiment. Due to that IMO, we no longer need to save the content of those files to
database. If we still want to save them for some other reason, we should save them as files,
not the string content. This change will need some database table data type modifications.

Feel free to provide your input.


View raw message