Return-Path: X-Original-To: apmail-jackrabbit-dev-archive@www.apache.org Delivered-To: apmail-jackrabbit-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 07806D9AD for ; Fri, 3 Aug 2012 18:01:10 +0000 (UTC) Received: (qmail 57599 invoked by uid 500); 3 Aug 2012 18:01:09 -0000 Delivered-To: apmail-jackrabbit-dev-archive@jackrabbit.apache.org Received: (qmail 57550 invoked by uid 500); 3 Aug 2012 18:01:09 -0000 Mailing-List: contact dev-help@jackrabbit.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@jackrabbit.apache.org Delivered-To: mailing list dev@jackrabbit.apache.org Received: (qmail 57542 invoked by uid 99); 3 Aug 2012 18:01:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Aug 2012 18:01:09 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rhauch@gmail.com designates 209.85.161.170 as permitted sender) Received: from [209.85.161.170] (HELO mail-gg0-f170.google.com) (209.85.161.170) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Aug 2012 18:01:02 +0000 Received: by ggnf2 with SMTP id f2so2109759ggn.1 for ; Fri, 03 Aug 2012 11:00:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:message-id:in-reply-to:references:subject:x-mailer :mime-version:content-type; bh=Sj9V93IJprO4McUGwvg+zssda5tvNIIGP3o9ZKoP0KA=; b=DmUe92GcyOX2H4J8y3agh9tsl1jYXTmWIPp59s/nI8kbwY61buFDPAj/SfUyS1tymK cC9uk56m+y5PMCvwE5uBXvsuklLzz6H4fM408ykooXWDB3M+Om2YvpAB9RoF2X+HTR5s UbSjFq1y98CtL5jQiKR5Ao4XKfeQuVAz2U+vIlbfgBZZ6K2MpMzWFUoX+fZ18ggL/eHR lYQ2FLExmu8MK82ADEiyr/MSVgjx93q7Zn0m/kcw2XMQzckSgJQVnsQdfQUH5NerfheI CAPEdFgm0Dt04j+COVM9EpwCqV8HqW3nC0mD1KkqtoYAVxt4WH4hoLPn3pjh+bcmj7PI lPgA== Received: by 10.50.186.162 with SMTP id fl2mr11917578igc.44.1344016840645; Fri, 03 Aug 2012 11:00:40 -0700 (PDT) Received: from [10.15.208.180] (nat-pool-rdu.redhat.com. [66.187.233.202]) by mx.google.com with ESMTPS id gz1sm412108igc.16.2012.08.03.11.00.38 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 03 Aug 2012 11:00:39 -0700 (PDT) Date: Fri, 3 Aug 2012 13:00:42 -0500 From: Randall Hauch To: dev@jackrabbit.apache.org Message-ID: <991FFF8A1E1A4815961E7B10211299D0@gmail.com> In-Reply-To: References: <4FD0D483.1060804@apache.org> <4FD0DDD1.1050204@gmx.de> <9B7EFE8783454ABDBDCCE42EACE6439B@gmail.com> Subject: Re: JSR-283 official TCK and the 'jackrabbit-jcr-tests' X-Mailer: sparrow 1.6.2 (build 1143.6) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="501c11ca_79f0d62f_890e" --501c11ca_79f0d62f_890e Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Thanks, Jukka. See below for a few responses/questions. As always, thanks again! On Friday, August 3, 2012 at 11:51 AM, Jukka Zitting wrote: > Hi, > > On Fri, Aug 3, 2012 at 5:21 PM, Randall Hauch wrote: > > The Jackrabbit TCK page [2] states that Jackrabbit's JCR Tests module is > > used within the official JSR-283 TCK [3] but is not the official TCK test. > > Obviously the former have been maintained and continue to improve, but the > > official JSR-283 TCK tests appear to have been released last on August 19, > > 2009. Is it possible that the official JSR-283 TCK tests can be updated, and > > would having a separate release schedule help with this in any way? > > > > > The only way to update the official JSR-283 TCK is through the JCP > maintenance process [1]. > > > (If the official TCK tests cannot be updated, then I presume any project wanting > > to claim JSR-283 compliance would have to run the official TCKs and appeal each > > of the older incorrect tests by stating the issue and perhaps using the > > updated/corrected tests Jackrabbit JCR Tests module as "corrections". > > > > > Yes, the appeal process is documented in [2] and relies on an exclude > list of buggy tests maintained by the spec lead. In practice, if you > have pointers to relevant jackrabbit-jcr-tests bug reports and can > demonstrate that your implementation passes the test after that issue > is fixed, the spec lead will be happy to put the test case on the > exclude list for you. > > What is the preferred way to demonstrate that our implementation passes all of the JCR Tests? We obviously run them in our builds, but we could also set up a separate Maven project to do that. I presume there's no way to easily run the official TCK tests but forcibly use the latest unofficial JCR Tests. Is there a preferred point of contact, other than jcr@day.com ? > > > Is there anything that makes this easier with a non-updated TCK? > > > It's a more light-weight process than doing a maintenance release through JCP. > > > How does Jackrabbit show spec and TCK compliance? > > We rely on the appeals process and exclude list as described above. In > practice that means that we're good as long as Jackrabbit passes the > latest version of the tests in jackrabbit-jcr-tests. > > > The official TCK and Jackrabbit's JCR tests unsurprisingly do not completely > > cover the specification, and doing so would require a significant amount of > > effort. However, there may be opportunity over time for projects to propose > > adding new tests. > > > > > Agreed. Especially with multiple active JCR implementations I think > there's a shared incentive to maintain compatibility beyond that of > what the official TCK tests. There's still plenty of place for healthy > competition on performance, scalability, maintainability and various > other features and "-ilities" that fall outside the scope of JCR. > > Thinking further, a standalone test codebase would also be a great > place to cooperate on things like performance benchmarks or other > tests that go beyond the scope of the JCR spec but that would still > add value to implementors of content repositories. > > It would be great to collaborate on this. We also have the start of a performance test framework (as does Jackrabbit) that requires nothing but Maven and a JDK; the set of tests is minimal. > > > If the JCR Tests' release cycle were independent from the Jackrabbit's > > release cycle, then the process of adding new tests and releasing updated > > JCR tests might be easier. On the other hand, verification that the tests are > > valid may become a bit harder. > > > > > Not necessarily. If the tests were maintained outside the main > Jackrabbit trunk, it would be easier set up a CI build that runs all > updates to the TCK codebase against the official JCR RI and latest > stable versions of Jackrabbit, ModeShape and other JCR implementations > (with excludes for tests that are known issues for each particular > implementation). That should make it much easier than now to catch > problems where the TCK codebase makes assumptions based on just a > single implementation. > > Setting up CI to automatically test against the latest stable releases is a great idea. > > > For example, the ModeShape project has around 70 additional tests [4] that > > we'd be willing to donate. Most of these are around verifying administration > > privileges (e.g., registering namespaces, node types, etc., especially for > > anonymous users), versioning, and locking. > > > > > That would be awesome! > > We also have some extra generic JCR tests, written for the jcr2spi > component, that could and ideally should be used as a part of the TCK. > We have a lot of query-related tests, too. But those are more basic JUnit tests that don't extend the AbstractJcrTest base class. It'd take a bit more work, but we could probably easily convert them to something compatible with JCR Tests. > > > Additionally, the Oak effort is already running the JCR tests and has > > recently found/fixed issues, too. Can those participating in Oak offer how > > much they might expect to simply reuse vs. add new tests? > > > > > > For now our goal is just to pass the already existing TCK tests, but > I'm sure that over time we'll encounter cases where existing clients > that assume only standard JCR functionality fail on Oak because of > some problems in our code. Capturing such cases as TCK tests would be > quite useful. > > [1] http://jcp.org/en/procedures/jcp2#5 > [2] http://www.day.com/content/day/en/products/jcr/jsr-283/_jcr_content/par/download_1/file.res/jsr-283-tck-appeal.pdf > > BR, > > Jukka Zitting --501c11ca_79f0d62f_890e Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Thanks, Jukka. See below for a few responses/questions. As always, th= anks again=21
=20

On =46riday, August 3,= 2012 at 11:51 AM, Jukka Zitting wrote:

Hi,

On = =46ri, Aug 3, 2012 at 5:21 PM, Randall Hauch <rhauch=40gmail.com> wrote:
The Jackrabbit TCK page =5B2=5D states that Jackr= abbit's JCR Tests module is
used within the official JSR-283 TC= K =5B3=5D but is not the official TCK test.
Obviously the forme= r have been maintained and continue to improve, but the
officia= l JSR-283 TCK tests appear to have been released last on August 19,
=
2009. Is it possible that the official JSR-283 TCK tests can be upda= ted, and
would having a separate release schedule help with thi= s in any way=3F

The only way = to update the official JSR-283 TCK is through the JCP
maintenan= ce process =5B1=5D.

(If the official TCK tests cannot be updated, then I presume any p= roject wanting
to claim JSR-283 compliance would have to run th= e official TCKs and appeal each
of the older incorrect tests by= stating the issue and perhaps using the
updated/corrected test= s Jackrabbit JCR Tests module as =22corrections=22.

Yes, the appeal process is documented in =5B2=5D a= nd relies on an exclude
list of buggy tests maintained by the s= pec lead. In practice, if you
have pointers to relevant jackrab= bit-jcr-tests bug reports and can
demonstrate that your impleme= ntation passes the test after that issue
is fixed, the spec lea= d will be happy to put the test case on the
exclude list for yo= u.
What is the preferred way to= demonstrate that our implementation passes all of the JCR Tests=3F We ob= viously run them in our builds, but we could also set up a separate Maven= project to do that. I presume there's no way to easily run the official = TCK tests but forcibly use the latest unofficial JCR Tests. Is there a pr= eferred point of contact, other than jcr=40day.com =3F
&nb= sp;
<= div>
Is there anything that makes= this easier with a non-updated TCK=3F

<= div>It's a more light-weight process than doing a maintenance release thr= ough JCP.

How does Jackrabbit show spec and TCK compliance=3F

We rely on the appeals process and exclude = list as described above. In
practice that means that we're good= as long as Jackrabbit passes the
latest version of the tests i= n jackrabbit-jcr-tests.

The official TCK and Jackrabbit's JCR tests unsurprisingly do = not completely
cover the specification, and doing so would requ= ire a significant amount of
effort. However, there may be oppor= tunity over time for projects to propose
adding new tests.

Agreed. Especially with multiple = active JCR implementations I think
there's a shared incentive t= o maintain compatibility beyond that of
what the official TCK t= ests. There's still plenty of place for healthy
competition on = performance, scalability, maintainability and various
other fea= tures and =22-ilities=22 that fall outside the scope of JCR.
Thinking further, a standalone test codebase would also be a= great
place to cooperate on things like performance benchmarks= or other
tests that go beyond the scope of the JCR spec but th= at would still
add value to implementors of content repositorie= s.
It would be great to collabo= rate on this. We also have the start of a performance test framework (as = does Jackrabbit) that requires nothing but Maven and a JDK; the set of te= sts is minimal.

If the = JCR Tests' release cycle were independent from the Jackrabbit's
release cycle, then the process of adding new tests and releasing update= d
JCR tests might be easier. On the other hand, verification th= at the tests are
valid may become a bit harder.

Not necessarily. If the tests were maintaine= d outside the main
Jackrabbit trunk, it would be easier set up = a CI build that runs all
updates to the TCK codebase against th= e official JCR RI and latest
stable versions of Jackrabbit, Mod= eShape and other JCR implementations
(with excludes for tests t= hat are known issues for each particular
implementation). That = should make it much easier than now to catch
problems where the= TCK codebase makes assumptions based on just a
single implemen= tation.
Setting up CI to automa= tically test against the latest stable releases is a great idea.
 
<= div>

=46or example,= the ModeShape project has around 70 additional tests =5B4=5D that
<= div>we'd be willing to donate. Most of these are around verifying adminis= tration
privileges (e.g., registering namespaces, node types, e= tc., especially for
anonymous users), versioning, and locking.<= /div>

That would be awesome=21

We also have some extra generic JCR tests, written f= or the jcr2spi
component, that could and ideally should be used= as a part of the TCK.

We have a lot of query-related tests, too. But those are more= basic JUnit tests that don't extend the AbstractJcrTest base class. It'd= take a bit more work, but we could probably easily convert them to somet= hing compatible with JCR Tests.
 
Additionally, the Oak effort is already runni= ng the JCR tests and has
recently found/fixed issues, too. Can = those participating in Oak offer how
much they might expect to = simply reuse vs. add new tests=3F
=46or now our goal is just to pass the already existing TCK t= ests, but
I'm sure that over time we'll encounter cases where e= xisting clients
that assume only standard JCR functionality fai= l on Oak because of
some problems in our code. Capturing such c= ases as TCK tests would be
quite useful.

=5B1=5D http://j= cp.org/en/procedures/jcp2=235

BR,

Jukka Zitting =20 =20 =20 =20
=20

--501c11ca_79f0d62f_890e--