Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <50548833.3040807@xebia.com>
Date: Sat, 15 Sep 2012 19:22:51 +0530
From: Rahul <rsharma@xebia.com>
User-Agent: Mozilla/5.0 (Windows NT 6.0; WOW64;
 rv:15.0) Gecko/20120907 Thunderbird/15.0.1
MIME-Version: 1.0
To: <crunch-dev@incubator.apache.org>
Subject: Re: Focus of our next release?
References: <20120914172357.GA8622@mafr.de>
 <CAH29n6PKNVp33uUnJT8XKSLJtzXRraf7n6CrJthxijRdpekcmw@mail.gmail.com>
 <20120915095543.GA20293@mafr.de>
In-Reply-To: <20120915095543.GA20293@mafr.de>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit

I agree that API refactoring should be done in smaller steps. This could 
be done in multiple releases also. Maybe for this release we could have 
some goal like just making some splits eg API and impl. For the rest we 
can continue on them also but they should not hold back the release.

Also +1 to the idea of having support of different Apache projects like 
solr/hcatalog/Cassandra etc. But again we can have some specific ones 
that we will make in this sprint.

I also want to know when do we want to make the next release ? Will it 
be linked to the things that we want to fix or do we have a set 
time-frame ? I feel that the next release should not take much long.

On 15-09-2012 15:25, Matthias Friedrich wrote:
> Hi,
>
> On Friday, 2012-09-14, Josh Wills wrote:
>> I like the idea of having themes for releases. In my head, the theme of
>> this release could be either
>>
>> a) Hacking the new MSCRPlanner code, esp. to add the ability to fuse
>> different MSCR jobs into a single instance that it enables, or
>> b) data access/integration points-- things like solr, hcatalog, hbase,
>> cassandra, jdbc, etc. as input and output sources for Crunch pipelines, or
>> c) API refactoring-- the crunch-api/crunch-impl/crunch-lib split, or
> I would see c) as part of a larger mission for improving documentation
> and usability. An immediate benefit would be that we don't have to
> provide javadoc for each and every class, only for those packages that
> are client-facing. Higher perceived quality with less work for us.
>
> I wouldn't make it a separate release though, perhaps we can do this
> in a series of smaller steps, starting with the crunch-api split.
> Refactorings like this usually turn into long frustrating monster
> tasks that prevent other progress. I'd really like to avoid that.
>
> But, before spending any more time on this, I think we should all
> agree that this is what we really want. Somehow I got the impression
> that you're not fully convinced that this refactoring is necessary or
> even a good idea. To me it would feel like people trying to rearrange
> the furniture in my living room. Let's discuss this here before we
> produce any more patches.
>
>> d) working on a PStream API that would let people apply DoFns to streams
>> and would build on top of things like WalMart's mupd8 or Storm or whatever.
>>
>> Of course, this is in addition to whatever fixes and new lib functions we
>> want to add over time. I don't want anything heavyweight, but those are
>> some of the larger-scale things that we'll need to tackle as a community,
>> and I would think of completing each of those big things as corresponding
>> to a release.
> Sounds good to me.
>
> Regards,
>    Matthias