pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Ryaboy <dvrya...@gmail.com>
Subject Re: Next Pig release proposal
Date Tue, 25 Oct 2011 00:38:08 GMT
To be a little more concrete about what I am saying here -- I don't think we
should put a "1.0" label on any *.0 release. 0.8.1 is pretty solid; 0.9.0
has some holes, 0.9.1 is better. If we put 1.0 on what is currently being
thought of as 0.10, it will have some stability / usability issues (things
tend to show up after we make a release and people in the wild start trying
it), and those issues will make a poor impression on those who expect 1.0 to
be shiny and polished after so much time. I'm in favor of waiting a couple
of dot releases, promoting a stabilized release into 1.0, and going from
there. So, pictorially:

-- trunk --- 0.11-dev ----------0.12-dev------------------| 1.2-dev!
    \               \
     \               \ ---------------- 0.11.0 --------------------| 1.1.0!
      \
       \------- 0.10.0 ------- 0.10.1 ------- 0.10.2 --------| 1.0.0 !!

On Mon, Oct 24, 2011 at 12:43 PM, Dmitriy Ryaboy <dvryaboy@gmail.com> wrote:

> I am good with Scheme 2.
>
> We are finding a fair number of issues trying to move from Pig 0.8.1 to
> 0.9, and I don't think those issues are fixed in 10, either.. not sure that
> this "stabilization" process has happened yet.
>
> D
>
>
> On Mon, Oct 24, 2011 at 11:59 AM, Daniel Dai <daijy@hortonworks.com>wrote:
>
>> Yes, we need a versioning scheme. There are two versioning scheme I can
>> think of:
>>
>> Scheme 1:
>> <major>.<patch>
>> <major> will be the feature rich release every 3 month
>> <patch> will be the bug fix release when necessary
>>
>> Nov release will be 1.0, Feb release will be 2.0. There will be 1.1, 2.1
>> etc
>> for bug fixes.
>>
>> Scheme 2:
>> <major>.<minor>.<patch>
>> Most of our 3 month release will be counted as <minor> release unless
>> there
>> are major user facing/disruptive changes.
>>
>> Nov release will be 1.0.0, Feb release will be 1.1.0. There will be 1.0.1,
>> 1.1.1 etc for bug fixes.
>>
>> I personally prefer scheme 2, increasing major version too frequently
>> might
>> be confusing to users. How's other folks feel?
>>
>> Daniel
>>
>>
>> On Sat, Oct 22, 2011 at 2:31 AM, Gianmarco De Francisci Morales <
>> gdfm@apache.org> wrote:
>>
>> > Hi,
>> >
>> > just my 2 cents.
>> >
>> > I think the issue here is not 1.0 vs 0.10, but what's the versioning
>> scheme
>> > we want to use for Pig.
>> > Up to now it has been just an increasing number after a '0.' prefix,
>> > changed
>> > when the community felt it was time. I think this works well for a small
>> > project, but it is somewhat fuzzy.
>> >
>> > I like the idea of having <major>.<minor>.<patch> versions
like many
>> other
>> > projects. It's a very clear and almost standard way of versioning a
>> piece
>> > of
>> > software. It has clear rules on when to change each of the numbers, and
>> > lets
>> > the user get an idea of backward compatibility at a glance.
>> >
>> > So, to conclude, I am in favor of going 1.0 (or 1.0.0) as long as we
>> decide
>> > a clear versioning policy (whichever it is).
>> > So that the 1.0 milestone would mark the beginning of our new policy.
>> >
>> > Cheers,
>> > --
>> > Gianmarco
>> >
>> >
>> >
>> > On Fri, Oct 21, 2011 at 23:10, <Milind.Bhandarkar@emc.com> wrote:
>> >
>> > > If one were to rewrite input and output formats to use the webhdfs://
>> > > APIs, this would not be an issue, right ?
>> > >
>> > > - milind
>> > >
>> > >
>> > > On 10/21/11 1:50 PM, "Santhosh Srinivasan" <sms@yahoo-inc.com> wrote:
>> > >
>> > > >If I was not clear in my earlier email, I apologize for the lack of
>> > > >clarity. I am no longer in favour of waiting for Hadoop API stability
>> > > >across Hadoop versions. It's a pipe dream.
>> > > >
>> > > >When we had PigInputFormat and PigOutputFormat, your reasoning would
>> be
>> > > >spot on. I am concerned about the following. Our tight integration
>> with
>> > > >Hadoop due to the use of Input and Output format might lead to a
>> break
>> > in
>> > > >backward compatibility. I am not sure if the comparison with that of
>> > Java
>> > > >is valid. Probably a majority of the users don't use JNI. Its very
>> hard
>> > > >to use Pig without writing custom load and store functions. The
>> default
>> > > >load and store don't suffice for a majority of use cases that I have
>> > > >observed.
>> > > >
>> > > >I am trying to get all factors that might influence this decision.
>> From
>> > > >the few emails that have been exchanged since yesterday, we have the
>> > > >following factors:
>> > > >
>> > > >1. Hadoop 0.20.205 (support for Append)
>> > > >2. Hadoop 0.22
>> > > >3. Hadoop 0.23
>> > > >4. Maturity of the new parser
>> > > >5. Stability of the new logical plan
>> > > >6. Other components in the eco system.
>> > > >       - Avro (1.5.4, 1.4.1, ...)
>> > > >       - Cassandra (1.0.0, 0.8.7, ...)
>> > > >       - Chukwa (0.4.0, 0.3.0, ...)
>> > > >       - Hama (0.3.0, 0.2.0, ...)
>> > > >       - Hbase (0.90.4, 0.90.3, 0.90.2, 0.90.1, ...)
>> > > >       - Hive (Releases - 0.7.1, 0.7.0, 0.6.0, ...)
>> > > >       - Zookeeper (3.3.3, 3.3.2, 3.2.2, 3.1.2, ...)
>> > > >
>> > > >Santhosh
>> > > >
>> > > >
>> > > >-----Original Message-----
>> > > >From: Thejas Nair [mailto:thejas@hortonworks.com]
>> > > >Sent: Friday, October 21, 2011 11:22 AM
>> > > >To: dev@pig.apache.org
>> > > >Subject: Re: Next Pig release proposal
>> > > >
>> > > >
>> > > >Santosh,
>> > > >I thought you meant API stability for hadoop across major versions,
>> but
>> > I
>> > > >guess you are referring to stability within 0.23 versions. But
>> argument
>> > > >applies to that as well, if 0.23.1 is not compatible with 0.23.0, we
>> > need
>> > > >to call the release for 0.23.1 as 'pig 1.x for 0.23.1 api' .
>> > > >
>> > > >We just need to communicate to the users that the
>> > > >InputFormat/OutputFormat api's (and any anything else we expose from
>> > > >hadoop) depends on the hadoop version they are using.
>> > > >
>> > > >I think it is just like different JNI libraries that you would write
>> for
>> > > >different OS. But the java version remains the same across OSs.
>> > > >
>> > > >-Thejas
>> > > >
>> > > >
>> > > >On 10/21/11 10:59 AM, Santhosh Srinivasan wrote:
>> > > >> Thejas,
>> > > >>
>> > > >> I guess you did not read my email completely. You are referring
to
>> the
>> > > >>premise without examining the conclusion. I am repasting my entire
>> > email
>> > > >>to avoid confusion (I hate truncated references). If you could
>> respond
>> > > >>again, it will bring us onto the same page.
>> > > >>
>> > > >> <email>
>> > > >>
>> > > >> Ref: http://tinyurl.com/4ng8upa (last discussion on 1.0)
>> > > >>
>> > > >> How far have we progressed from our last discussion in March.
There
>> > was
>> > > >>no consensus on the 1.0 release. Opinions ranged from having more
>> > > >>releases to bake in the maturity of the new parser and logical
plan
>> > > >>changes to compatibility with Hadoop API (was compared to Social
>> > > >>Security - a very hot topic these days).
>> > > >>
>> > > >> My concerns were around Hadoop API stability. I have heard that
the
>> > > >>APIs will not be stable for at least 1 year. This is taking me
away
>> > from
>> > > >>the Hadoop API stability factor (They passed healthcare in that
>> > > >>duration. Really!) Do we want compatibility with 0.23 as a gating
>> > factor
>> > > >>- not sure if this is anywhere close to getting done in the near
>> > future.
>> > > >>Will we support append (0.20.205)?
>> > > >>
>> > > >> Btw, Hbase has been doing 0.90.1, 0.90.2, etc. So we can take
a
>> look
>> > at
>> > > >>this option too.
>> > > >>
>> > > >> Santhosh
>> > > >>
>> > > >>
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]
>> > > >> Sent: Thursday, October 20, 2011 4:40 PM
>> > > >> To: dev@pig.apache.org
>> > > >> Subject: Next Pig release proposal
>> > > >>
>> > > >> Hi,
>> > > >>
>> > > >> Here is what I propose we do for the next Pig release:
>> > > >>
>> > > >>
>> > > >> (1)    Branch early next week - we have major features  and many
>> bug
>> > > >>fixes in and will be fixing remaining bugs on the branch
>> > > >>
>> > > >> (2)    Publish the release by 11/15 - that will give us a couple
of
>> > > >>weeks to stabilize the branch and get last minute bug fixes in
>> > > >>
>> > > >> (3)    Make this release a 1.0 release. Reasons to go for 1.0
and
>> not
>> > > >>0.10
>> > > >>
>> > > >> a.       This release has minimal number of features and was
>> focused
>> > on
>> > > >>code stabilization and bug fixes. We believe it will be a stable
>> > release
>> > > >>
>> > > >> <email/>
>> > > >>
>> > > >> Thanks,
>> > > >> Santhosh
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Thejas Nair [mailto:thejas@hortonworks.com]
>> > > >> Sent: Friday, October 21, 2011 10:45 AM
>> > > >> To: dev@pig.apache.org
>> > > >> Subject: Re: Next Pig release proposal
>> > > >>
>> > > >> On 10/20/11 4:58 PM, Santhosh Srinivasan wrote:
>> > > >>> Ref: http://tinyurl.com/4ng8upa (last discussion on 1.0)
>> > > >>>
>> > > >>> How far have we progressed from our last discussion in March.
>> There
>> > > >>>was no consensus on the 1.0 release. Opinions ranged from having
>> more
>> > > >>>releases to bake in the maturity of the new parser and logical
plan
>> > > >>>changes to compatibility with Hadoop API (was compared to Social
>> > > >>>Security - a very hot topic these days).
>> > > >>>
>> > > >>> My concerns were around Hadoop API stability.
>> > > >>
>> > > >> Over the next year or so, there are going to be two API versions
of
>> > > >>hadoop to be supported - 0.20.x api's and 0.23 apis, as we will
have
>> > > >>userbase on both.
>> > > >>
>> > > >> I think it is just a matter of releasing pig 1.0 for 0.20.x api's
>> and
>> > > >>1.0 for 0.23.x api's.  We will have to come up with a numbering
>> scheme
>> > > >>that reflects 'for hadoop version X' in our pig releases, regardless
>> of
>> > > >>it being 0.10 or 1.0.
>> > > >>
>> > > >> As there will be support for different api's of hadoop in pig
>> > releases,
>> > > >>I don't see a reason why the hadoop api stability should stop pig
>> from
>> > > >>going 1.0 .
>> > > >>
>> > > >> -Thejas
>> > > >
>> > > >
>> > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message