drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Phillips <ste...@dremio.com>
Subject Re: Out Of Memory Error (Possible Regression)
Date Wed, 30 Dec 2015 20:42:02 GMT
I didn't see any tests running out of memory. Which tests are you seeing
with this?

On Wed, Dec 30, 2015 at 1:37 PM, Abdel Hakim Deneche <adeneche@maprtech.com>
wrote:

> Steven,
>
> were you able to successfully run the regression tests on the transfer
> patch ? I just tried and saw several queries running out of memory !
>
> On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com
> > wrote:
>
> > Created DRILL-4236 <https://issues.apache.org/jira/browse/DRILL-4236> to
> > keep track of this improvement.
> >
> > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau <jacques@dremio.com>
> > wrote:
> >
> >> Since the accounting changed (more accurate), the termination condition
> >> for
> >> the sort operator will be different than before. In fact, this likely
> will
> >> be sooner since our accounting is much larger than previously (since we
> >> correctly consider the entire allocation rather than simply the used
> >> allocation).
> >>
> >> Hakim,
> >> Steven and I were discussing the need to update the ExternalSort
> operator
> >> to use the new allocator functionality to better manage its memory
> >> envelope. Would you be interested in working on this since you seem to
> be
> >> working with that code the most? Basically, it used to be that there was
> >> no
> >> way the sort operator would be able to correctly detect a memory
> condition
> >> and so it jumped through a bunch of hoops to try to figure out the
> >> termination condition.With the transfer accounting in place, this code
> can
> >> be greatly simplified to just use the current operator memory
> allocation.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
> >> challapallirahul@gmail.com> wrote:
> >>
> >> > I installed the latest master and ran this query. So
> >> > planner.memory.max_query_memory_per_node should have been the default
> >> > value. I switched back to 1.4.0 branch and this query completed
> >> > successfully.
> >> >
> >> > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
> >> > adeneche@maprtech.com
> >> > > wrote:
> >> >
> >> > > Rahul,
> >> > >
> >> > > How much memory was assigned to the sort operator (
> >> > > planner.memory.max_query_memory_per_node) ?
> >> > >
> >> > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> >> > > challapallirahul@gmail.com> wrote:
> >> > >
> >> > > > I am seeing an OOM error while executing a simple CTAS query.
I
> >> raised
> >> > > > DRILL-4324 for this. The query mentioned in the JIRA used to
> >> complete
> >> > > > successfully without any issue prior to 1.5. Any idea what could
> >> have
> >> > > > caused the regression?
> >> > > >
> >> > > > - Rahul
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Abdelhakim Deneche
> >> > >
> >> > > Software Engineer
> >> > >
> >> > >   <http://www.mapr.com/>
> >> > >
> >> > >
> >> > > Now Available - Free Hadoop On-Demand Training
> >> > > <
> >> > >
> >> >
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message