Mailing-List: contact dev-help@tajo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tajo.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CACZfFK6Odj3a0_MB17+-Qj23DDCbaQpOK4i2pq0SBFyC53K5aQ@mail.gmail.com>
References: 
 <CAOeZVif27M4roKnSHS139uF6KLmHWaDE2gFWTT+h5bcJu3zRSg@mail.gmail.com>
	<CACZfFK47OcRrKgq9UVob3tHw2+a2TCxxsXG6sFG+ygDE-T2Rtw@mail.gmail.com>
	<CAOeZVicb1hj9vCL_R8qVoHmgxi5kbWkqc=Y0G97cmu1t6vSFaA@mail.gmail.com>
	<CACZfFK6Odj3a0_MB17+-Qj23DDCbaQpOK4i2pq0SBFyC53K5aQ@mail.gmail.com>
Date: Wed, 17 Jun 2015 22:46:22 +0530
Message-ID: 
 <CAOeZVicW8VN3qUEmprNLm+iSCF095TRvMipMzVxRg7C93RfkvA@mail.gmail.com>
Subject: Re: Parallel Aggregates
From: Atri Sharma <atri.jiit@gmail.com>
To: dev@tajo.apache.org
Content-Type: multipart/alternative; boundary=001a114da6f4ad61860518b9d9c8

--001a114da6f4ad61860518b9d9c8
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thank you.

Is there any improvement in aggregates that we are looking at please?
On 16 Jun 2015 17:07, "Jihoon Son" <jihoonson@apache.org> wrote:

> In Tajo, aggregation is very similar to that in Hadoop MapReduce.
> Let me consider an example. Given a query of "select *k*, count(*) from *=
t*
> group by *k*", Tajo generates a LogicalPlan as follows.
>
> group by (k)
>        |
>    scan (t)
>
> This LogicalPlan is translated into a MasterPlan as follows.
>
> -----------------
>      Stage2
>   group by *k*
> -----------------
>           |
> shuffle tuples with *k*
>           |
> -----------------
>      Stage1
>   group by *k*
>          |
>     scan *t*
> -----------------
>
> As you can see in this example, the query plan consists of 2 stages. Each
> stage is executed subsequently because the result of Stage 1 is used as t=
he
> input of Stage 2. Each stage is divided into multiple tasks for each inpu=
t
> split as follows.
>
> Stage1
>
> Task 1
> group by *k*
>        |
>   scan *t* (0 - 99)
>
> Task 2
> group by *k*
>        |
>   scan *t* (100 - 199)
> ...
>
> Each task is executed by a TajoWorker. As you can see, tasks of the first
> stage execute a local aggregation after scanning input split. This local
> aggregation result is shuffled among TajoWorkers with the aggregation key
> *k*. Then, the final aggregation is computed at the second stage.
>
> Stage1 and Stage2 are similar to Map and Reduce of MapReduce. The local
> aggregation of Stage1 is similar to the Combiner of Hadoop MapReduce.
>
> I hope that this will be helpful to you.
> If you have any further questions, please feel free to ask.
> Jihoon
>
> 2015=EB=85=84 6=EC=9B=94 16=EC=9D=BC (=ED=99=94) =EC=98=A4=EC=A0=84 7:28,=
 Atri Sharma <atri.jiit@gmail.com>=EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=B1:
>
> Thanks.
> >
> > What are your thoughts on parallel aggregation? Generating query plans
> that
> > allow states to be generated which can be executed independently and th=
en
> > states recombined?
> > On 16 Jun 2015 05:25, "Jihoon Son" <jihoonson@apache.org> wrote:
> >
> > > Hi Atri, thanks for your question.
> > >
> > > First of all, maybe you already did, I recommend that you read this
> > article
> > > <
> > >
> >
> http://www.hadoopsphere.com/2015/02/technical-deep-dive-into-apache-tajo.=
html
> > > >
> > > before you start implementation. This is written by Hyunsik, and
> contains
> > > the description of Tajo's overall infrastructure. Afterwards, I think
> > that
> > > you may ask more detailed question.
> > >
> > > Here, I'll roughly list some important classes for aggregate
> > > implementation.
> > >
> > >    - SQLParser.g4 contains our SQL parsing rules. It is written in
> antlr.
> > >    - SQLAnalyzer is our parser based on rules defined at SQLParser.g4=
.
> > >    - SQLAnalyzer translates a SQL query into a tree of Expr which
> > >    represents an algebraic expression.
> > >    - LogicalPlanner translates the Expr tree into a LogicalPlan that
> > >    logically describes how the given query will be executed.
> > >    - GlobalPlanner translates the LogicalPlan into a MasterPlan
> > >    (distributed query execution plan) that describes how the given
> query
> > > will
> > >    be executed in distributed cluster.
> > >    - Once a MasterPlan is created, QueryMaster starts to execute quer=
y
> > >    processing. A query consists of multiple stages, which are
> > individually
> > >    processed in some order.
> > >       - For example, a simple aggregation query is executed in two
> > stages,
> > >       each of which is for parallel aggregation and combining
> aggregates.
> > > These
> > >       stages are executed sequentially.
> > >    - A stage is concurrently processed by multiple tasks, and is
> executed
> > >    by TajoWorker.
> > >    - Each task contains meta information for input data and a
> LogicalPlan
> > >    of the stage. This LogicalPlan is translated into PhysicalExec by
> > >    PhysicalPlanner.
> > >    - PhysicalExec describes how the query is actually executed.
> > >       - For example, there are two types of AggregationExec,
> > >       i.e., HashAggregateExec and SortAggregateExec, for hash-based
> > > aggregation
> > >       and sort-based aggregation, respectively.
> > >
> > > Best regards,
> > > Jihoon
> > >
> > > 2015=EB=85=84 6=EC=9B=94 15=EC=9D=BC (=EC=9B=94) =EC=98=A4=ED=9B=84 1=
1:32, Atri Sharma <atri.jiit@gmail.com>=EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=
=B1:
> > >
> > > > Folks,
> > > >
> > > > I am looking into parallel aggregates/combining aggregates. I have =
a
> > plan
> > > > around it which I think can work.
> > > >
> > > > Please update me on current infrastructure and point me around the
> > > existing
> > > > code base. Also, ideas would be most welcome around it.
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Atri
> > > > *l'apprenant*
> > > >
> > >
> >
>

--001a114da6f4ad61860518b9d9c8--