Mailing-List: contact dev-help@tajo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tajo.apache.org
MIME-Version: 1.0
References: 
 <CAOeZVif27M4roKnSHS139uF6KLmHWaDE2gFWTT+h5bcJu3zRSg@mail.gmail.com>
 <CACZfFK47OcRrKgq9UVob3tHw2+a2TCxxsXG6sFG+ygDE-T2Rtw@mail.gmail.com>
 <CAOeZVicb1hj9vCL_R8qVoHmgxi5kbWkqc=Y0G97cmu1t6vSFaA@mail.gmail.com>
 <CACZfFK6Odj3a0_MB17+-Qj23DDCbaQpOK4i2pq0SBFyC53K5aQ@mail.gmail.com>
 <CAOeZVicW8VN3qUEmprNLm+iSCF095TRvMipMzVxRg7C93RfkvA@mail.gmail.com>
In-Reply-To: 
 <CAOeZVicW8VN3qUEmprNLm+iSCF095TRvMipMzVxRg7C93RfkvA@mail.gmail.com>
From: Jihoon Son <jihoonson@apache.org>
Date: Wed, 17 Jun 2015 21:12:44 +0000
Message-ID: 
 <CACZfFK6LycaOt+FvENiOeZ_ej2T4er_vvuMRURpG8ru=Ci6oWA@mail.gmail.com>
Subject: Re: Parallel Aggregates
To: dev@tajo.apache.org
Content-Type: multipart/alternative; boundary=001a1137a7f68c14cc0518bd2799

--001a1137a7f68c14cc0518bd2799
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

As far as I know, there aren't any plans for improvement except in distinct
aggregation. I think that our code for distinct aggregation is too
complicated, and the performance also should be improved.

So, when you design the implementation of your algorithm on Tajo, you don't
have to consider distinct aggregation part, I think.

2015=EB=85=84 6=EC=9B=94 18=EC=9D=BC (=EB=AA=A9) =EC=98=A4=EC=A0=84 2:16, A=
tri Sharma <atri.jiit@gmail.com>=EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=B1:

> Thank you.
>
> Is there any improvement in aggregates that we are looking at please?
> On 16 Jun 2015 17:07, "Jihoon Son" <jihoonson@apache.org> wrote:
>
> > In Tajo, aggregation is very similar to that in Hadoop MapReduce.
> > Let me consider an example. Given a query of "select *k*, count(*) from
> *t*
> > group by *k*", Tajo generates a LogicalPlan as follows.
> >
> > group by (k)
> >        |
> >    scan (t)
> >
> > This LogicalPlan is translated into a MasterPlan as follows.
> >
> > -----------------
> >      Stage2
> >   group by *k*
> > -----------------
> >           |
> > shuffle tuples with *k*
> >           |
> > -----------------
> >      Stage1
> >   group by *k*
> >          |
> >     scan *t*
> > -----------------
> >
> > As you can see in this example, the query plan consists of 2 stages. Ea=
ch
> > stage is executed subsequently because the result of Stage 1 is used as
> the
> > input of Stage 2. Each stage is divided into multiple tasks for each
> input
> > split as follows.
> >
> > Stage1
> >
> > Task 1
> > group by *k*
> >        |
> >   scan *t* (0 - 99)
> >
> > Task 2
> > group by *k*
> >        |
> >   scan *t* (100 - 199)
> > ...
> >
> > Each task is executed by a TajoWorker. As you can see, tasks of the fir=
st
> > stage execute a local aggregation after scanning input split. This loca=
l
> > aggregation result is shuffled among TajoWorkers with the aggregation k=
ey
> > *k*. Then, the final aggregation is computed at the second stage.
> >
> > Stage1 and Stage2 are similar to Map and Reduce of MapReduce. The local
> > aggregation of Stage1 is similar to the Combiner of Hadoop MapReduce.
> >
> > I hope that this will be helpful to you.
> > If you have any further questions, please feel free to ask.
> > Jihoon
> >
> > 2015=EB=85=84 6=EC=9B=94 16=EC=9D=BC (=ED=99=94) =EC=98=A4=EC=A0=84 7:2=
8, Atri Sharma <atri.jiit@gmail.com>=EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=B1:
> >
> > Thanks.
> > >
> > > What are your thoughts on parallel aggregation? Generating query plan=
s
> > that
> > > allow states to be generated which can be executed independently and
> then
> > > states recombined?
> > > On 16 Jun 2015 05:25, "Jihoon Son" <jihoonson@apache.org> wrote:
> > >
> > > > Hi Atri, thanks for your question.
> > > >
> > > > First of all, maybe you already did, I recommend that you read this
> > > article
> > > > <
> > > >
> > >
> >
> http://www.hadoopsphere.com/2015/02/technical-deep-dive-into-apache-tajo.=
html
> > > > >
> > > > before you start implementation. This is written by Hyunsik, and
> > contains
> > > > the description of Tajo's overall infrastructure. Afterwards, I thi=
nk
> > > that
> > > > you may ask more detailed question.
> > > >
> > > > Here, I'll roughly list some important classes for aggregate
> > > > implementation.
> > > >
> > > >    - SQLParser.g4 contains our SQL parsing rules. It is written in
> > antlr.
> > > >    - SQLAnalyzer is our parser based on rules defined at
> SQLParser.g4.
> > > >    - SQLAnalyzer translates a SQL query into a tree of Expr which
> > > >    represents an algebraic expression.
> > > >    - LogicalPlanner translates the Expr tree into a LogicalPlan tha=
t
> > > >    logically describes how the given query will be executed.
> > > >    - GlobalPlanner translates the LogicalPlan into a MasterPlan
> > > >    (distributed query execution plan) that describes how the given
> > query
> > > > will
> > > >    be executed in distributed cluster.
> > > >    - Once a MasterPlan is created, QueryMaster starts to execute
> query
> > > >    processing. A query consists of multiple stages, which are
> > > individually
> > > >    processed in some order.
> > > >       - For example, a simple aggregation query is executed in two
> > > stages,
> > > >       each of which is for parallel aggregation and combining
> > aggregates.
> > > > These
> > > >       stages are executed sequentially.
> > > >    - A stage is concurrently processed by multiple tasks, and is
> > executed
> > > >    by TajoWorker.
> > > >    - Each task contains meta information for input data and a
> > LogicalPlan
> > > >    of the stage. This LogicalPlan is translated into PhysicalExec b=
y
> > > >    PhysicalPlanner.
> > > >    - PhysicalExec describes how the query is actually executed.
> > > >       - For example, there are two types of AggregationExec,
> > > >       i.e., HashAggregateExec and SortAggregateExec, for hash-based
> > > > aggregation
> > > >       and sort-based aggregation, respectively.
> > > >
> > > > Best regards,
> > > > Jihoon
> > > >
> > > > 2015=EB=85=84 6=EC=9B=94 15=EC=9D=BC (=EC=9B=94) =EC=98=A4=ED=9B=84=
 11:32, Atri Sharma <atri.jiit@gmail.com>=EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=
=B1:
> > > >
> > > > > Folks,
> > > > >
> > > > > I am looking into parallel aggregates/combining aggregates. I hav=
e
> a
> > > plan
> > > > > around it which I think can work.
> > > > >
> > > > > Please update me on current infrastructure and point me around th=
e
> > > > existing
> > > > > code base. Also, ideas would be most welcome around it.
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > *l'apprenant*
> > > > >
> > > >
> > >
> >
>

--001a1137a7f68c14cc0518bd2799--