drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: Out Of Memory Error (Possible Regression)
Date Wed, 30 Dec 2015 19:01:25 GMT
Since the accounting changed (more accurate), the termination condition for
the sort operator will be different than before. In fact, this likely will
be sooner since our accounting is much larger than previously (since we
correctly consider the entire allocation rather than simply the used
allocation).

Hakim,
Steven and I were discussing the need to update the ExternalSort operator
to use the new allocator functionality to better manage its memory
envelope. Would you be interested in working on this since you seem to be
working with that code the most? Basically, it used to be that there was no
way the sort operator would be able to correctly detect a memory condition
and so it jumped through a bunch of hoops to try to figure out the
termination condition.With the transfer accounting in place, this code can
be greatly simplified to just use the current operator memory allocation.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> I installed the latest master and ran this query. So
> planner.memory.max_query_memory_per_node should have been the default
> value. I switched back to 1.4.0 branch and this query completed
> successfully.
>
> On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com
> > wrote:
>
> > Rahul,
> >
> > How much memory was assigned to the sort operator (
> > planner.memory.max_query_memory_per_node) ?
> >
> > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > I am seeing an OOM error while executing a simple CTAS query. I raised
> > > DRILL-4324 for this. The query mentioned in the JIRA used to complete
> > > successfully without any issue prior to 1.5. Any idea what could have
> > > caused the regression?
> > >
> > > - Rahul
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message