apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Re: [malhar-users] Caching in Datatorrent
Date Tue, 01 Sep 2015 22:23:39 GMT
 The cache manager
https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java
is a cache used only for DTFile. DTFile is a TFileReader(Writer) with cache
which provides much faster IO compared to native hadoop TFile but keeps the
format



On Tue, Sep 1, 2015 at 2:58 PM, Thomas Weise <thomas.weise@gmail.com> wrote:

> An operator can use any technology to implement the cache. Here is an
> example how to structure a cache from the Malhar library:
>
>
> https://github.com/apache/incubator-apex-malhar/blob/v3.0.0/library/src/main/java/com/datatorrent/lib/db/cache/AbstractDBLookupCacheBackedOperator.java
>
>
> Please note that the resulting caching is local to each operator instance
> (every partition has its own cache). The idea is that access to shared
> state is minimized for more efficient processing.
>
>
> On Mon, Aug 3, 2015 at 8:21 AM, Amol Kekre <amol@datatorrent.com> wrote:
>
> >
> > Chiranjeevi,
> > We do have guava (
> > https://code.google.com/p/guava-libraries/wiki/CachesExplained)
> available
> > as a Malhar operator
> >
> >
> >
> https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java
> >
> > Do take a look.
> >
> > Thks,
> > Amol
> >
> >
> > On Mon, Aug 3, 2015 at 1:51 AM, ravi <chiru.vcj@gmail.com> wrote:
> >
> >> Hi Team,
> >>
> >> Do we have caching available in Datatorrent? If yes, Please let me know
> >> how to implement.
> >>
> >> -many thanks
> >> Chiranjeevi
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Malhar" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to malhar-users+unsubscribe@googlegroups.com.
> >> To post to this group, send email to malhar-users@googlegroups.com.
> >> Visit this group at http://groups.google.com/group/malhar-users.
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Malhar" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to malhar-users+unsubscribe@googlegroups.com.
> > To post to this group, send email to malhar-users@googlegroups.com.
> > Visit this group at http://groups.google.com/group/malhar-users.
> > For more options, visit https://groups.google.com/d/optout.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message