reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Chung (JIRA)" <>
Subject [jira] [Commented] (REEF-1357) Allow different caching levels for caching in IInputPartition
Date Fri, 24 Jun 2016 23:40:16 GMT


Andrew Chung commented on REEF-1357:

Sorry, there seems to be multiple threads going on within this thread and I am a bit lost.
Have we reached a consensus? When we mention caching, should we do it automatically in a wrapper
of {{IInputPartition}} or allow the implementers of {{IInputPartition}} to perform their own
caching? Are we only caching to memory, or are we also providing the ability to cache to some
other storage? [~markus.weimer] [~dkm2110] [~shravanmn] provided that this is mainly your
use case and I'm unfamiliar with the workload characteristics, it would be great if one of
you could sum up the discussion. Thanks!

> Allow different caching levels for caching in IInputPartition
> -------------------------------------------------------------
>                 Key: REEF-1357
>                 URL:
>             Project: REEF
>          Issue Type: Improvement
>          Components: IMRU, REEF.NET
>            Reporter: Andrew Chung
>            Assignee: Andrew Chung
>              Labels: FT
> Caching in {{IInputPartition}} is currently entirely dependent on the implementation
of the {{IInputPartition}}. We should modify/expand {{IInputPartition}} such that the data
loading component is fully separate from the caching component. An idea is to instead provide
only a single concrete {{IInputPartition}} implementation (while deprecating the old implementations)
and use the composition model, complete with an {{IDataLoader}} which fetches data back in
a {{Stream}}, and an {{IDataCache}}, which allows caching at granular levels.

This message was sent by Atlassian JIRA

View raw message