reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julia (JIRA)" <>
Subject [jira] [Commented] (REEF-1080) Make local temp file configurable in FileSystemInputPartition
Date Wed, 16 Dec 2015 19:55:46 GMT


Julia commented on REEF-1080:

*Java TempFileCreator mainly wraps tempFolderAsFile as File object and provide methods to
create directory. In C#, File class is to provide static methods only. We could do a similar
TempFileCreator in C#, but it might only wrap strings as file path. 
*In Java,WorkingDirectoryTempFileCreator is bound to TempFileCreator as part of Evaluator
configuration and hard coded. What if user want to specify his own TempFileCreator? 
*I agree we can make a generic one for temp folder, either as a named parameter of string,
or an interface/class, We can provide default value of the string, we can also provide default
implementation of the interface, however, if it is hard coded in the Evaluator configuration,
that doesn't  meet what we want, right? 
*If we remove hard coded one (currently it is for Java Evaluator only), we need to have a
way to allow the user to specify and pass it to Evaluator configuration. May be defining a
provider is a way to do it. 
*I don't see any usage in Java for TempFileCreator  except tests. Do I miss anything? 

> Make local temp file configurable in FileSystemInputPartition
> -------------------------------------------------------------
>                 Key: REEF-1080
>                 URL:
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF.NET
>            Reporter: Julia
>            Assignee: Julia
> Currently in FileSystemInputPartition, when download remote file to a local file before
processing the data. The local file is based on Path.GetTempPath() which usually is a folder
under C driver. There are multiple issues:
> When file size is big, c: driver may not have enough space 
> We would like the temp file to be deleted in the end, although we already implemented
disposable to delete the temp file, and client should make sure it is called, there is still
possibility the file is not cleared if the system suddenly crash. 
> To resolve above issues, we would like to make the temp file folder for FileSystemInputPartition
configurable, and default to the current working folder so that in case the system is crashed,
the working folder will still be cleared. 

This message was sent by Atlassian JIRA

View raw message