Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CA+5xAo3s1TDnMG64UDYF0t0H47sVMg3xidOP-2BJPM0Hz5S9ow@mail.gmail.com>
References: <3b293890-d0f1-47db-9aca-9e6a8d0008ff@googlegroups.com>
	<CAMaZnT7rD8+hqadf3t80=iwy7PfOVf-iMoLtP8OUkeWojNhiKg@mail.gmail.com>
	<CA+5xAo3s1TDnMG64UDYF0t0H47sVMg3xidOP-2BJPM0Hz5S9ow@mail.gmail.com>
Date: Tue, 1 Sep 2015 15:23:39 -0700
Message-ID: 
 <CAOi1F6VXBS+DjouRK63iq6HJJY87Dn+xCyf8FtyxidCwXuKU5Q@mail.gmail.com>
Subject: Re: [malhar-users] Caching in Datatorrent
From: Siyuan Hua <siyuan@datatorrent.com>
To: dev@apex.incubator.apache.org
Cc: malhar-users@googlegroups.com
Content-Type: multipart/alternative; boundary=089e0111b8667552cf051eb7007b

--089e0111b8667552cf051eb7007b
Content-Type: text/plain; charset=UTF-8

 The cache manager
https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java
is a cache used only for DTFile. DTFile is a TFileReader(Writer) with cache
which provides much faster IO compared to native hadoop TFile but keeps the
format


On Tue, Sep 1, 2015 at 2:58 PM, Thomas Weise <thomas.weise@gmail.com> wrote:

> An operator can use any technology to implement the cache. Here is an
> example how to structure a cache from the Malhar library:
>
>
> https://github.com/apache/incubator-apex-malhar/blob/v3.0.0/library/src/main/java/com/datatorrent/lib/db/cache/AbstractDBLookupCacheBackedOperator.java
>
>
> Please note that the resulting caching is local to each operator instance
> (every partition has its own cache). The idea is that access to shared
> state is minimized for more efficient processing.
>
>
> On Mon, Aug 3, 2015 at 8:21 AM, Amol Kekre <amol@datatorrent.com> wrote:
>
> >
> > Chiranjeevi,
> > We do have guava (
> > https://code.google.com/p/guava-libraries/wiki/CachesExplained)
> available
> > as a Malhar operator
> >
> >
> >
> https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java
> >
> > Do take a look.
> >
> > Thks,
> > Amol
> >
> >
> > On Mon, Aug 3, 2015 at 1:51 AM, ravi <chiru.vcj@gmail.com> wrote:
> >
> >> Hi Team,
> >>
> >> Do we have caching available in Datatorrent? If yes, Please let me know
> >> how to implement.
> >>
> >> -many thanks
> >> Chiranjeevi
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Malhar" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to malhar-users+unsubscribe@googlegroups.com.
> >> To post to this group, send email to malhar-users@googlegroups.com.
> >> Visit this group at http://groups.google.com/group/malhar-users.
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Malhar" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to malhar-users+unsubscribe@googlegroups.com.
> > To post to this group, send email to malhar-users@googlegroups.com.
> > Visit this group at http://groups.google.com/group/malhar-users.
> > For more options, visit https://groups.google.com/d/optout.
> >
>

--089e0111b8667552cf051eb7007b--