systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Dusenberry <dusenberr...@gmail.com>
Subject Remove "Scratch Space" In Favor Of Temp Folder
Date Thu, 31 Mar 2016 17:58:32 GMT
Hi all,

Currently, SystemML makes use of a "scratch space" folder for temporary
files during execution.  This is currently set to a relative
"scratch_space" directory that will be placed relative to the execution
path (local mode) or in the user's directory on HDFS.  This works okay in
some cases, although it can cause confusion as to why the folder exists.
In other cases, such as on Databricks Cloud, a relative path for HDFS is
not allowed, and thus the user must change this "scratch space" folder to
an absolute path, or else a strange error message will occur.

Since this "scratch space" folder is just for temporary files during
execution, might it be better to simply query HDFS (which falls back to
local FS if need) for a temporary folder, and just use that?  If so, this
would remove the need to adjust this setting, thus making it easier to use
SystemML.

Thoughts?


- Mike

--

Michael W. Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message