reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1206) Update Deserializer API to allow deserialize from remote files directly
Date Tue, 16 Feb 2016 23:01:18 GMT

    [ https://issues.apache.org/jira/browse/REEF-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149483#comment-15149483
] 

Markus Weimer commented on REEF-1206:
-------------------------------------

{quote}
For deserializer, it also needs to know the file paths are local or remote. For remote files,
deserializer must use IFileSystem to access the files. For local files, normal .Net file system
can be simply used. 
{quote}

Can't we use IFileSystem in both cases? We have local and remote implementations of that.

Also, the deserializer shouldn't have to know any of this, right? It could be based on {{Stream}}
instances alone.

> Update Deserializer API to allow deserialize from remote files directly
> -----------------------------------------------------------------------
>
>                 Key: REEF-1206
>                 URL: https://issues.apache.org/jira/browse/REEF-1206
>             Project: REEF
>          Issue Type: Task
>            Reporter: Julia
>            Assignee: Julia
>
> Currently Deserialize() API accept a folder as input parameter. It only works for local
files as only in local case, we can guarantee all the files are copied to the same folder
and the folder only contains the files that need to be deserialized.  
> For remote file case, the file paths are passed from upper stream, the file folder may
contain some other irrelevant files. In this case, passing a set of individual file oaths
would be more more sense. 
> For deserializer, it also needs to know the file paths are local or remote. For remote
files, deserializer must use IFileSystem to access the files. For local files, normal .Net
file system can be simply used. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message