drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Phillips <sphill...@maprtech.com>
Subject Re: what is the correct behavior when an upstream operator returns STOP
Date Mon, 04 May 2015 17:20:20 GMT
The *current* correct behavior is to return STOP downstream. That is
because the ownership of batches created by that operator's allocator may
have been transferred downstream, and if we try to cleanup right away, it
will result in apparent memory leaks.

Like Jacques mentioned, I am working on a patch that will change this
behavior, and the reason we will be able to clean up right away is because
there is work in progress to move to a model where ownership of buffers is
transferred as well.

On Mon, May 4, 2015 at 10:11 AM, Jacques Nadeau <jacques@apache.org> wrote:

> In general, an operator should release its resources as soon as it can.  I
> believe Steven is working on a patch to make all resource intensive
> operators do just this.
>
> On Mon, May 4, 2015 at 9:47 AM, Abdel Hakim Deneche <adeneche@maprtech.com
> >
> wrote:
>
> > What is the correct behavior for an operator when it receives an
> > IterOutcome.STOP from upstream:
> > 1. should it just stop right away and return STOP downstream ?, or
> > 2. should it cleanup it's resources before returning STOP downstream ?
> >
> > Some operators follow 1 and others (like OrderedPartitionRecordBatch)
> > follow 2.
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>



-- 
 Steven Phillips
 Software Engineer

 mapr.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message