reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Chung (JIRA)" <>
Subject [jira] [Commented] (REEF-1339) Adding IInputPartition.Cache() for data download and cache
Date Mon, 25 Apr 2016 20:30:12 GMT


Andrew Chung commented on REEF-1339:

What I envision is a complete separation of Caching and Data loading -- there will only be
one implementation of {{IInputPartition}}, with a parameterless {{Cache}} function. Data loading
will be handled by a provided/custom implementation of {{IDataLoader}} and data caching will
be handled by a collection of provided/custom implementations of {{IDataCache}}. Both will
be injected by the user. On start, the default implementation of {{IInputPartition}} will
fetch the data and cache in all injected {{IDataCache}}'s, should the user choose to cache.

> Adding IInputPartition.Cache() for data download and cache
> ----------------------------------------------------------
>                 Key: REEF-1339
>                 URL:
>             Project: REEF
>          Issue Type: Task
>            Reporter: Julia
>            Assignee: Andrew Chung
>              Labels: FT
> Currently, in FileSystemInputPartition, data downloading is implemented in Initilaize()
and called from GetPartitionHandle. It doesn't give client a flexibility to decide when to
download data. Besides, if client wants to cache data in advance, they need to call GetPartitionHandle()
and iterate the data. 
> We would like to expose a new API Cache() in IInputPartition which performs data download
to RAM, SSD, HDD, etc based on client's configuration. 
> The method should be called in ContextStartHandler  in IMRU scenarios. 

This message was sent by Atlassian JIRA

View raw message