Mailing-List: contact dev-help@tajo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tajo.apache.org
Received-SPF: pass (athena.apache.org: domain of coderplay@gmail.com
 designates 209.85.220.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <2C704A84-FE00-4A90-8CE4-A46629DB8F27@linkedin.com>
References: 
 <CAM=XDd-ic-A1uV4eDy-svyhsHPM9=FEfX1FhJsB7jRGakKCQXA@mail.gmail.com>
	<CACZfFK44uP9_vSx+nUdEyQ0e5sYQH98wS7w4j7gErMyAFyPyRA@mail.gmail.com>
	<CAM=XDd_upNK7891GovyT_Q6adcc-o67Mh=i7ztzHnDdw9N_G+w@mail.gmail.com>
	<CANpXwBCAHFM4AKQwrc6XJ=W6R6mCmUBgPU4PWNFhN7C6SB3zKw@mail.gmail.com>
	<CAM=XDd8SNN8Qdm+kQPNg_eadzvMj_KfF4pDTU-6wV8DiNyuC6A@mail.gmail.com>
	<CAM=XDd_qFVQTXunM9aS0WjG1f69kTjc2hhSTMeM6JXB41ne+GA@mail.gmail.com>
	<CALO_SpRUpKn1+620Pfj7qtobsEyq65fuYghMwgfSoYpA0tF7UA@mail.gmail.com>
	<CAM=XDd-cig-JZ+Ps5H7riyVsH26t2hjozcL3iAzWAONpNZzo=A@mail.gmail.com>
	<CACZfFK6TrQuCU7pPw=O51YU6YT1j_GbLSWNa_Y0BDXuKiZh1Nw@mail.gmail.com>
	<CAM=XDd9=m06tkfH7aBvJ3jQJR3Ci7P-Bf0r2w7hgTv56aBQPwQ@mail.gmail.com>
	<05D8FDE4-46C7-4C6F-9446-7CB89BA440CB@aol.com>
	<2C704A84-FE00-4A90-8CE4-A46629DB8F27@linkedin.com>
Date: Mon, 14 Apr 2014 13:05:54 -0700
Message-ID: 
 <CALO_SpSVneVE00PWGHaS2mdbuCPcnhnZ9qRCU6mHvQ_iE6T8Cg@mail.gmail.com>
Subject: Re: [DISCUSS] 0.8.0 release and next roadmap
From: Min Zhou <coderplay@gmail.com>
To: dev@tajo.apache.org
Content-Type: multipart/alternative; boundary=047d7b2ed25505536704f70636b9

--047d7b2ed25505536704f70636b9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Until today realized that my reply haven't been sent.

+1

Totally agree with Hyunsik. 0.9 is more appropriate for the next release.

Min


On Mon, Apr 14, 2014 at 12:31 PM, David Chen <dchen@linkedin.com> wrote:

> +1
>
> I agree with Hyunsik as well. I think since 1.0 increments the major
> version number, it should be used for a particularly significant release.=
 :)
>
> Thanks,
> David
>
>
> On Apr 13, 2014, at 7:51 PM, Alvin Henrick <share.code@aol.com> wrote:
>
> > +1 Hyunsik.
> >
> > Thanks!
> > Warm Regards,
> > Alvin.
> >
> > On Apr 11, 2014, at 8:30 AM, Hyunsik Choi wrote:
> >
> >> Hi folks,
> >>
> >> I'd like to discuss the next version number. In Jira, we have
> provisionally
> >> used 1.0, and we didn't decide the next major version. I propose 0.9 a=
s
> the
> >> next major version. What do you think about this?
> >>
> >> Regards,
> >> Hyunsik
> >>
> >>
> >> On Thu, Apr 10, 2014 at 11:05 AM, Jihoon Son <jihoonson@apache.org>
> wrote:
> >>
> >>> Min, thanks for reminding us!
> >>> It's a mandatory issue.
> >>> We need to implement that feature ASAP.
> >>>
> >>> Thanks,
> >>> Jihoon
> >>>
> >>>
> >>> 2014-04-10 3:19 GMT+09:00 Hyunsik Choi <hyunsik@apache.org>:
> >>>
> >>>> Min,
> >>>>
> >>>> Yes, you are right. I'm thinking it everyday, but I missed it. Thank
> you
> >>>> for reminding me. It would be achieved by modifying Query class to
> >>> execute
> >>>> independent execution blocks in parallel. I'll add it to the wiki.
> >>>>
> >>>> Thanks,
> >>>> Hyunsik
> >>>>
> >>>>
> >>>> On Thu, Apr 10, 2014 at 2:43 AM, Min Zhou <coderplay@gmail.com>
> wrote:
> >>>>
> >>>>> Yeah.. Another issue,  seems a query like A join B. Tajo will scan =
A
> at
> >>>>> first stage, after that in the 2nd stage scan B. Doesn't run it in
> >>>>> parallel, right?
> >>>>>
> >>>>>
> >>>>> Min
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 9, 2014 at 10:10 AM, Hyunsik Choi <hyunsik@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> I've just updated the roadmap page. Please take a look at the
> section
> >>>>>> 'After 0.8.0'
> >>>>>> https://cwiki.apache.org/confluence/display/TAJO/Tajo+Roadmap
> >>>>>>
> >>>>>> If there are missed or additional ideas, feel free to add them on
> >>> that
> >>>>>> page or suggest them here. After we discuss them more, we would
> >>> decide
> >>>>>> their priorities.
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Hyunsik
> >>>>>>
> >>>>>> On Sat, Apr 5, 2014 at 12:06 AM, Hyunsik Choi <hyunsik@apache.org>
> >>>>> wrote:
> >>>>>>> Hi Hyoungjun,
> >>>>>>>
> >>>>>>> Yes, TPC-H and TPC-DS scripts for Tajo are necessary. If we provi=
de
> >>>>>>> users with some prepared benchmark environment, users can test Ta=
jo
> >>>>>>> easily. I'll file your idea on the wiki. Thank you for your
> >>>>>>> suggestion.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Hyunsik
> >>>>>>>
> >>>>>>> On Fri, Apr 4, 2014 at 11:48 PM, =EA=B9=80=ED=98=95=EC=A4=80 <bab=
okim@gmail.com> wrote:
> >>>>>>>> Hi Hyunsik ,
> >>>>>>>>
> >>>>>>>> I did benchmark test with TPC-H, TPC-DS data. Benchmark script
> >>> like
> >>>>> hive
> >>>>>>>> and impala is more helpful to test.
> >>>>>>>>
> >>>>>>>> https://github.com/rxin/TPC-H-Hive
> >>>>>>>> https://github.com/cartershanklin/hive-testbench
> >>>>>>>> https://github.com/cloudera/impala-tpcds-kit
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>> Hyoungjun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 2014-04-04 23:40 GMT+09:00 Hyunsik Choi <hyunsik@apache.org>:
> >>>>>>>>
> >>>>>>>>> Hi Jihoon,
> >>>>>>>>>
> >>>>>>>>> CUBE and ROLL-UP are key features for analytic problems. I file=
d
> >>> it
> >>>>> on
> >>>>>> the
> >>>>>>>>> wiki.
> >>>>>>>>>
> >>>>>>>>> TAJO-266 and TAJO-161 will give more optimization opportunities
> >>> to
> >>>>>>>>> logical planning and distributed query planning. But, I'm not
> >>> sure
> >>>> it
> >>>>>>>>> can be included in short-term roadmap. They are necessary, but
> >>> they
> >>>>>>>>> are not required right now. In my view, it would be reasonable =
to
> >>>>>>>>> schedule them on long-term roadmap.
> >>>>>>>>>
> >>>>>>>>> Warm regards,
> >>>>>>>>> Hyunsik
> >>>>>>>>>
> >>>>>>>>> On Fri, Apr 4, 2014 at 3:01 PM, Jihoon Son <jihoonson@apache.or=
g
> >>>>
> >>>>>> wrote:
> >>>>>>>>>> Hi Hyunsik,
> >>>>>>>>>> I'm very glad that we can release the next version, soon.
> >>>>>>>>>> Also, appreciate for the guideline of the next roadmap.
> >>>>>>>>>>
> >>>>>>>>>> Addition to the aforementioned features, I have the two
> >>>>> suggestions.
> >>>>>>>>>> First is the support of CUBE operator (TAJO-259). Acutally, I
> >>>>>> started it
> >>>>>>>>>> quite a long time ago, but it is delayed due to the lower
> >>>> priority
> >>>>>> than
> >>>>>>>>>> other stability issues. But, since this operator is widely use=
d
> >>>> in
> >>>>>>>>> analytic
> >>>>>>>>>> applications, we need to add this feature as soon as possible.
> >>>> So,
> >>>>>> in my
> >>>>>>>>>> opinion, it would be good to add this feature to the next
> >>>> roadmap.
> >>>>>>>>>>
> >>>>>>>>>> Second is the advanced query optimization. TAJO-266 is an issu=
e
> >>>> for
> >>>>>>>>> making
> >>>>>>>>>> the query plan more flexible. After that, we can employ the
> >>>> plenty
> >>>>>>>>>> optimization opportunities like described in TAJO-161.
> >>>>>>>>>>
> >>>>>>>>>> How do you guys think about these issues?
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jihoon
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 2014-04-04 14:24 GMT+09:00 Hyunsik Choi <hyunsik@apache.org>:
> >>>>>>>>>>
> >>>>>>>>>>> Hi folks,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm very happy to see that our community is growing! Also,
> >>> It's
> >>>> a
> >>>>>>>>> pleasure
> >>>>>>>>>>> to discuss the Tajo 0.8.0 release. Recently, I've tested
> >>> various
> >>>>>>>>> features
> >>>>>>>>>>> in various contexts, and tried to figure out if there are any
> >>>>>> critical
> >>>>>>>>>>> problems. I think that there are only a few issues and we can
> >>>>>> release
> >>>>>>>>> 0.8.0
> >>>>>>>>>>> next week. If there are further issues to be solved before th=
e
> >>>>> 0.8.0
> >>>>>>>>>>> release, feel free to suggest ideas.
> >>>>>>>>>>>
> >>>>>>>>>>> Also, I'd like to discuss our next roadmap. We are open to an=
y
> >>>>>>>>> suggestion
> >>>>>>>>>>> from users, contributors, and committers. Please fire away!
> >>>>>>>>>>>
> >>>>>>>>>>> I'm thinking that our next stage should focus on improving th=
e
> >>>> way
> >>>>>> Tajo
> >>>>>>>>>>> runs in thousands of large cluster nodes and for a number of
> >>>>>> concurrent
> >>>>>>>>>>> users. The key issues associated with this include the
> >>>> following:
> >>>>>>>>>>>
> >>>>>>>>>>> * High availability
> >>>>>>>>>>> * Multi-tenancy scheduling
> >>>>>>>>>>> * More stability
> >>>>>>>>>>> * Improved shuffle
> >>>>>>>>>>>
> >>>>>>>>>>> The current work status is as follows. Min is working on
> >>> Tajo's
> >>>>> new
> >>>>>>>>>>> scheduler (TAJO-540) based on sparrow. I'll support him. As
> >>> far
> >>>>> as I
> >>>>>>>>> know,
> >>>>>>>>>>> Alvin is working on TajoMaster HA (TAJO-704). Also, some guys
> >>>>>> including
> >>>>>>>>>>> myself are investigating and solving the issues which occur i=
n
> >>>>> large
> >>>>>>>>>>> clusters. These issues should be solved in order to make Tajo
> >>> a
> >>>>>> complete
> >>>>>>>>>>> enterprise-ready production.
> >>>>>>>>>>>
> >>>>>>>>>>> In addition, there are some SQL feature support issues. Many
> >>>>>> analytic
> >>>>>>>>>>> problems require window functions. Also, in-subquery and
> >>> scalar
> >>>>>> subquery
> >>>>>>>>>>> should be supported. So, I'd like to schedule them with high
> >>>>>> priority.
> >>>>>>>>> In
> >>>>>>>>>>> my view, there will be very few SQL support issues if Tajo
> >>>>> provides
> >>>>>>>>> these
> >>>>>>>>>>> features.
> >>>>>>>>>>>
> >>>>>>>>>>> Besides those areas, David is working on a nested schema and
> >>> its
> >>>>>> related
> >>>>>>>>>>> work (TAJO-710). I guess this will take quite a while because
> >>> it
> >>>>>>>>> requires a
> >>>>>>>>>>> lot of hard work. So, it would be great to schedule the neste=
d
> >>>>>> schema
> >>>>>>>>>>> loosely. That's just my thoughts, anyhow.
> >>>>>>>>>>>
> >>>>>>>>>>> Aside from the discussion of our roadmap, I'd like to suggest
> >>>> that
> >>>>>> we
> >>>>>>>>> need
> >>>>>>>>>>> to release more frequently after the 0.8.0 release. So far,
> >>>> there
> >>>>>> has
> >>>>>>>>> been
> >>>>>>>>>>> a long period between each release because Tajo is undergoing
> >>>>> heavy
> >>>>>>>>>>> development. By 'releasing early, releasing often', we will
> >>> make
> >>>>>> more
> >>>>>>>>>>> tighter feedback loop between users and developers.
> >>>>>>>>>>>
> >>>>>>>>>>> I think that there are many additional many interesting issue=
s
> >>>> to
> >>>>> be
> >>>>>>>>>>> included in our roadmap. Feel free to suggest your idea. We
> >>> will
> >>>>>> arrange
> >>>>>>>>>>> our short-term roadmap and long-term roadmap based on your
> >>>>>> suggestions.
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you all so much for your contribution!
> >>>>>>>>>>>
> >>>>>>>>>>> Warm Regards,
> >>>>>>>>>>> Hyunsik
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Tajo - Big Data Warehouse System on Hadoop
> >>>>>>>> http://tajo.apache.org/
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> My research interests are distributed systems, parallel computing a=
nd
> >>>>> bytecode based virtual machine.
> >>>>>
> >>>>> My profile:
> >>>>> http://www.linkedin.com/in/coderplay
> >>>>> My blog:
> >>>>> http://coderplay.javaeye.com
> >>>>>
> >>>>
> >>>
> >
>
>


--=20
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

--047d7b2ed25505536704f70636b9--