Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <dev@tomcat.apache.org>
Message-ID: <51FE7DD3.1050008@apache.org>
Date: Sun, 04 Aug 2013 18:14:11 +0200
From: Mark Thomas <markt@apache.org>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8;
 rv:17.0) Gecko/20130620 Thunderbird/17.0.7
MIME-Version: 1.0
To: Tomcat Developers List <dev@tomcat.apache.org>
Subject: Re: [VOTE] Release Apache Tomcat 8.0.0-RC1
References: <51FACAD0.3030906@apache.org> <51FCD616.7070603@kippdata.de>
In-Reply-To: <51FCD616.7070603@kippdata.de>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On 03/08/2013 12:06, Rainer Jung wrote:
> Overview:
> 
> - one crash in tcnative during unit tests
> - MBeans indicate common loader is gone (is that expected?
>   At least the common.loader entry still exists in catalina.properties)
Expected as StandardClassLoader has been removed. That decision is
reversible of there is a use case for having the class loader visible in
JMX.

> - some MBean data gone (resource cache statistics, environment content)
Resource cache is expected. Env is not. Might need to add more JMX stats
to new WebResource implementation.

> - Catalina:j2eeType=WebModule,name=//localhost attribute
>   workDir switched from work/Catalina/localhost/_
>   to work/Catalina/localhost/ROOT
Expected.

> - new SEVERE and Exceptions in unit test output, maybe expected
Unless the test fails then these should be expected.

> - bayeaux and tomcat-lite modules not bundled
Expected. I won't be copying these to tc8.0.x/trunk

> - MD5 OK
> - signatures OK
> - key in KEYS file
> - gz and zip for src and bin consistent
> ! src consistent with svn tag except:
>   - modules bayeux and tomcat-lite not in src
See above.

>   - file
> res/META-INF/tomcat-websocket.jar/services/javax.servlet.ServletContainerInitializer
>     has Unix line ends in svn, but the src archives
>     for Unix and Windows both contain it with DOS line ends
It wasn't treated as a text file. Now fixed.

> ! builds fine but
>   - several warning about unsafe or unchecked operations.
>     See full list at end of mail.
I was expecting all of those to be unfixable. To double-check I disabled
Eclipse's hide unavoidable generics errors and to my surprise ~30% were
easily fixable. These have been fixed. The rest do look to be unavoidable.

Digging into the 'fixes' all they appear to be doing is using a JRE
method that already has @SupressWarnings so arguably Eclipse was right.
Even so, I am no longer as strongly +1 for using this option as I have been.

> - build result looks consistent with binaries
> - no checkstyle complaints
> - only few Javadoc warnings, I fixed those
> ! Unit tests few failures and errors:
These need looking at. Note: Many of the tests are timing sensitive and
do not fail consistently. Making the tests less sensitive to timing is
an ongoing task.

> 
>   - Timing: TestWebSocketFrameClient.testConnectToServerEndpointSSL()
>     took 63-66 seconds, but expects only 60 seconds.
> 
> 
>   - On NIO in addition (same test):
> 
> java.lang.NullPointerException
>         at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.longPoll(Http11NioProtocol.java:276)
>         at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:692)
>         at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
>         at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
>         at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
> 
> 
>   - On APR in addition a crash (core) in:
> 
>  b76dfe74 Java_org_apache_tomcat_jni_Socket_recv (30b528, b30ff188,
> b30fd09c, b29500, b30ff214, 0) + 34
>  fbc0f780 * org/apache/tomcat/jni/Socket.recv(J[BII)I+32362
>  fbc0f72c * org/apache/tomcat/jni/Socket.recv(J[BII)I+0
>  fbc068ec *
> org/apache/coyote/http11/upgrade/AprServletInputStream.doRead(Z[BII)I+73
> (line 101)
>  fbc068ec *
> org/apache/coyote/http11/upgrade/AbstractServletInputStream.read([BII)I+20
> (line 229)
>  fbc068ec *
> org/apache/tomcat/websocket/server/WsFrameServer.onDataAvailable()V+46
> (line 89)
>  fbc06b84 *
> org/apache/tomcat/websocket/server/WsHttpUpgradeHandler$WsReadListener.onDataAvailable()V+4
> (line 384)
>  fbc0771c *
> org/apache/coyote/http11/upgrade/AbstractServletInputStream.onDataAvailable()V+11
> (line 337)
>  fbc06b84 *
> org/apache/coyote/http11/upgrade/AbstractProcessor.upgradeDispatch(Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;+11
> (line 189)
>  fbc073e0 *
> org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;+171
> (line 1221)
>  fbc06848 *
> org/apache/coyote/http11/Http11AprProtocol$Http11ConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;+76
> (line 546)
>  fbc073e0 *
> org/apache/tomcat/util/net/AprEndpoint$SocketProcessor.doRun()V+26 (line
> 4492)
>  fbc06b84 *
> org/apache/tomcat/util/net/AprEndpoint$SocketProcessor.run()V+55 (line 4468)
> ...
> 
> GDB tells my:
> 
> #7  0xb76dfe74 in Java_org_apache_tomcat_jni_Socket_recv (e=0x30b528,
> o=0xb30ff188, sock=<optimized out>, buf=0xb30ff214, offset=0,
> toread=8192) at src/network.c:733
> 733             if ((ss = (*s->net->recv)(s->opaque, sb, &nbytes)) ==
> APR_SUCCESS)
> 
> #7  0xb76dfe74 in Java_org_apache_tomcat_jni_Socket_recv (e=0x30b528,
> o=0xb30ff188, sock=<optimized out>, buf=0xb30ff214, offset=0,
> toread=8192) at src/network.c:733
>         sb =
> "\000\006\026\200\000\fÞÌ³\017Ñ\000·\205{Í·\205{Ø³\017Ñ\234³\017Ñ\030³\017Ð¨·\227(p»\204H\200ò\032\226x\000\000\000\023\000\000\000\000ûÁ\227hò\032\227\020³\017Ñ8³\017Ñ8ûÀhH\000\000\b¹³\017Ñh³\017ÑHûÀhH³\017ÑP\000\000\002°·\211+Ø\000\000\000u\000\000\000\000ò\032\227
> ò\032\226\210ò\032\226\210\000\000\000u\000\000\000\000³\017ÑhûÀhH»\204Hh¼Ð¬h\000\000\001\032\000\000\000\000³\017Ñ\230·\225x'·\225xH³\017Ò$³\017Ñ ³\017Ñ8·\225}ø\000\060´\000\000\000\000\000\000\000\000¶\000\000\000\000ûÁÄÀ"...
>         s = 0xb29500
>         nbytes = 8192
>         ss = <optimized out>
> 
> (gdb) print *s
> $5 = {pool = 0xb294c0, child = 0x0, sock = 0x0, opaque = 0xb297d8,
>   jsbbuff = 0x330398 "HTTP/1.1 101 Switching Protocols\r\nServer:
> Apache-Coyote/1.1\r\nUpgrade: websocket\r\nConnection:
> upgrade\r\nSec-WebSocket-Accept: tPEj68f24a5073zco+NHsEUCKpY=\r\nDate:
> Fri, 02 Aug 2013 12:52:53 GMT\r\n\r\n",
>   jrbbuff = 0xbfbc78 "GET /firehose HTTP/1.1\r\nSec-WebSocket-Version:
> 13\r\nUpgrade: websocket\r\nHost: localhost:59353\r\nSec-WebSocket-Key:
> qzlX6/aQxs4TAKyKCb+XLg==\r\nConnection: upgrade\r\n\r\n", net = 0x0, pe
> = 0xb87958, last_active = 0, timeout = -2}
> 
> The reason for the crash is likely:
> 
> (gdb) print s->net
> $7 = (tcn_nlayer_t *) 0x0
> 
> So dereferencing it in (*s->net->recv) gives a segfault.
> 
> 
>   - TestWsWebSocketContainer:
> 
> Testcase: testMaxMessageSize03 took 0.426 sec
>         Caused an ERROR
> The WebSocket session has been closed and no method (apart from close())
> may be called on a closed session
> java.lang.IllegalStateException: The WebSocket session has been closed
> and no method (apart from close()) may be called on a closed session
>         at
> org.apache.tomcat.websocket.WsSession.checkState(WsSession.java:607)
>         at
> org.apache.tomcat.websocket.WsSession.getUserProperties(WsSession.java:556)
>         at
> org.apache.tomcat.websocket.WsRemoteEndpointImplBase.getBlockingSendTimeout(WsRemoteEndpointImplBase.java:360)
>         at
> org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendPartialString(WsRemoteEndpointImplBase.java:195)
>         at
> org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendString(WsRemoteEndpointImplBase.java:155)
>         at
> org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:37)
>         at
> org.apache.tomcat.websocket.TestWsWebSocketContainer.doMaxMessageSize(TestWsWebSocketContainer.java:859)
>         at
> org.apache.tomcat.websocket.TestWsWebSocketContainer.testMaxMessageSize03(TestWsWebSocketContainer.java:830)
> 
> 
>   - TestWebappClassLoaderExecutorMemoryLeak NIO:
> 
> Testcase: testTimerThreadLeak took 4.932 sec
>         FAILED
> null
> junit.framework.AssertionFailedError
>         at
> org.apache.catalina.loader.TestWebappClassLoaderExecutorMemoryLeak.testTimerThreadLeak(TestWebappClassLoaderExecutorMemoryLeak.java:72)
> 
> 
> 
> - JMX MBean-Comparison with TC 7:
> 
>   - MBean Catalina:type=ServerClassLoader,name=common no longer exists
Expected.

>   - MBeans j2eeType=WebModule attribute
>     parentClassLoader is no longer of type
>     org.apache.catalina.loader.StandardClassLoader
>     but instead java.net.URLClassLoader
Expected.

>   - MBeans j2eeType=Servlet and j2eeType=WebModule no longer
>     have attributes eventProvider and statisticsProvider
Expected. They were hard-coded to false so were removed.

>   - MBeans j2eeType=WebModule no longer have attributes
>     saveConfig, staticResources and mappingObject
saveConfig - removed as was unused
staticResoucres - new WebResources impl
mappingObject - removed (unnecessary duplicate)

>   - MBeans type=Cache replaced by type=WebResourceRoot
>     - old attributes accessCount, hitsCount, desiredEntryAccessRatio,
>       cacheSize, maxAllocateIterations removed
>     - new attributes cacheMaxObjectSize, cacheTtl, cachingAllowed,
>       allowLinking, stateName, objectName, domain added
All expected. Attributes can be added to new impl as required.

>   - MBean type=NamingResourcesno longer provides the arrays
>     resources, environments and resourceLinks and attribute container.
>     modelerType changed from
>     org.apache.catalina.mbeans.NamingResourcesMBean
>     to org.apache.tomcat.util.modeler.BaseModelMBean
That is not expected. Need to look into that.

>   - MBeans type=Loader no longer have attributes repositories,
>     repositoriesString and searchExternalFirst.
>     Probably replaced by the new Resources impl
Correct.

>   - MBeans type=WebappClassLoader no longer has attribute
>     searchExternalFirst.
>     Probably replaced by the new Resources impl.
Corrct.

>   - ThreadPool, RequestProcessor and GlobalRequestProcessor now
>     for nio instead of bio as default; corresponding name changes
>     in thread names, class names etc. and increased maxConnections
Expected.

>   - nio based ThreadPools switch useComet, usePolling and
>     useCometTimeout from "false" to "true", for http also
>     useSendfile switched from "false" to "true".
>     New attributes keepAliveCount, pollerThreadCount,
>     pollerThreadPriority, selectorTimeout, selectorPool and
>     oomParachute
Expected.

>   - nio based ProtocolHandler new attributes connectionCount,
>     cometSupported, cometTimeoutSupported and sendfileSupported.
>     For http in addition pollerThreadCount, pollerThreadPriority,
>     acceptorThreadPriority, useSendfile, endpoint and selectorTimeout.
>     Attribute disableKeepAlivePercentage only existed for http and
>     is gone there now.
Expected.

>   - Example Servlets wsChat, wsEchoMessage, wsEchoStream, wsSnake
>     removed
Need to look in to JMX visibility of new WebSocket impl.

>   - Filter o.a.t.websocket.server.WsFilter added to all 5 webapps
>     (ROOT, docs, examples, manager, host-manager)
Expected.

>   - order of jstl.jar and standard.jar in loader for /examples
>     switched
Odd but should not be an issue.

>   - MBean type=Environment now className
>     org.apache.tomcat.util.descriptor.web.ContextEnvironment
>     instead of org.apache.catalina.deploy.ContextEnvironment
Expected.

>   - type=Mapper singleton, no longer per connector
Expected.

>   - Catalina:type=Valve,host=localhost,name=AccessLogValve
>     locale changed from en_US to en, fileDateFormat changed
>     from yyyy-MM-dd to .yyyy-MM-dd, prefix changed from
>     localhost_access_log. to localhost_access_log (prefix and
>     fileDateFormat changes OK), resolveHosts attribute gone (unused),
>     info attribute gone (not a big loss)
Expected.

>   - Catalina:type=Valve,host=localhost,name=ErrorReportValve
>     attribute info gone (not a big loss)
Expected.

>   - Catalina:type=Engine attribute baseDir replaced by catalinaBase
Expected.

>   - Catalina:j2eeType=WebModule,name=//localhost attribute
>     ignoreAnnotations switched from "true" to "false",
Need to look in to that.

>     attribute workDir switched from work/Catalina/localhost/_
>     to work/Catalina/localhost/ROOT
Expected.

>   - Catalina:type=Deployer,host=localhost attribute
>     configClass: org.apache.catalina.startup.ContextConfig gone
Expected.

>   - new (platform) MBeans java.nio:type=BufferPool,name=direct
>     and java.nio:type=BufferPool,name=mapped
OK.

>   - platform MBean type=Logging: LoggerNames javax, javax.management and
>     javax.management.mbeanserver removed, org.apache.tomcat.websocket
>     and org.apache.tomcat.websocket.WsWebSocketContainer added
OK.

>   - platform MBean java.lang:type=Threading shows 4 more (daemon)
>     threads
OK.

>   - platform MBean java.lang:type=OperatingSystem show 88 open file
>     descriptors instead of 61
Worth just checking what those files are.

>   - platform MBean java.lang:type=Runtime shows 3 system properties
>     *.jarsToSkip replaced by t.u.s.StandardJarScanFilter.jarsToScan and
>     t.u.s.StandardJarScanFilter.jarsToSkip
Expected.

> Build and tests were done using Java 1.7.0_25. OS was Solaris 10 Sparc,
> tcnative was 1.1.27 based on APR 1.4.8 and OpenSSL 1.0.1e (plus a few
> patches).
> 
> Unit test warnings (very similar to 7.0.35 and before, but two new ones,
> one NPE in access log, one testResources warning):
> 
> - org.apache.tomcat.util.descriptor.web.TestWebXmlOrdering
>   - (old) BIO, NIO and APR: [main]
>     org.apache.tomcat.util.descriptor.web.WebXml.orderWebFragments
>     Used a wrong fragment name z at web.xml absolute-ordering tag!
>   - (new) BIO, NIO and APR: [main]
>     org.apache.tomcat.util.descriptor.web.WebXml.setVersion
>     Unknown version string [0.0]. Default version will be used.
Look OK.

> - org.apache.tomcat.util.net.TestCustomSsl
>   - (old) BIO twice "Exception getting SSL attributes"
>     in org.apache.coyote.http11.Http11Processor actionInternal
>     exception is:
>     javax.net.ssl.SSLProtocolException: handshake alert: no_certificate
>   - (old) NIO twice "Exception re-negotiating SSL connection"
>     in org.apache.coyote.http11.Http11NioProcessor actionInternal
Need to check.

> - org.apache.catalina.startup.TestHostConfigAutomaticDeployment
>   - (new) BIO, NIO and APR: [main] 3 times each
>     org.apache.catalina.startup.HostConfig deployDescriptor
>     A docBase .../webapps/myapp inside the host appBase has been
>     specified, and will be ignored
>   - (new) BIO, NIO and APR: [main] 2 times each
>     org.apache.catalina.startup.HostConfig deployWARs
>     The directory [.../webapps/myapp] will be ignored because the WAR
>     [.../webapps/myapp.war] takes priority and unpackWARs is false
Expected.

> - tribes: several warnings (41), numbers up by 22 from 7.0.42.
>   Mostly same types of messages.
Probably OK but should check.

> Unit test SEVERE messages: 439 total (+218 rel TC 7).
That seems rather high.

> About 200 of those are of types not observed for TC 7:
> 
> Count  Message
> 
>   48 : ContainerBase.addChild: start:
Need to look into those.

>   48 : Error deploying configuration descriptor
> .../conf/Tomcat/localhost/myapp.xml
Related to the above.

>   12 : Error listenerStart
Need to look into these.

>   12 : For security constraints with URL pattern [/] only the HTTP
> methods [POST] are covered. All other methods are uncovered.
>    6 : For security constraints with URL pattern [/] only the HTTP
> methods [GET] are covered. All other methods are uncovered.
>    6 : For security constraints with URL pattern [/test] only the HTTP
> methods [POST] are covered. All other methods are uncovered.
>    3 : For security constraints with URL pattern [/test] only the HTTP
> methods [PUT] are covered. All other methods are uncovered.
>    3  [main]
> org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods
> For security constraints with URL pattern [/test] only the HTTP methods
> [GET] are covered. All other methods are uncovered.
>    3  [main]
> org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods
> For security constraints with URL pattern [/test] only the HTTP methods
> [POST] are covered. All other methods are uncovered.
>    3  [main]
> org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods
> For security constraints with URL pattern [/test] the HTTP methods [GET]
> are uncovered.
>    3  [main]
> org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods
> For security constraints with URL pattern [/test] the HTTP methods
> [POST] are uncovered.
All expected. These are new tests for the new 3.1 uncovered methods feature.

>    9 : Error reading request, ignored
>    9 : Exception fixing docBase for context [/myapp]
>    6  [localhost-startStop-1]
> org.apache.catalina.core.StandardContext.listenerStart Exception sending
> context initialized event to listener instance of class
> websocket.echo.WsConfigListener
>    6  [localhost-startStop-1]
> org.apache.catalina.core.StandardContext.startInternal Context
> [/examples] startup failed due to previous errors
>    6  [localhost-startStop-1]
> org.apache.catalina.core.StandardContext.startInternal Error listenerStart
>    6 : Context [/examples] startup failed due to previous errors
>    6 : Context [/myapp] startup failed due to previous errors
>    6 : Exception sending context initialized event to listener instance
> of class websocket.echo.WsConfigListener
>    6 : Skipped installing application listeners due to previous error(s)
>    3 : No error handling configured for
> [org.apache.tomcat.websocket.pojo.TestPojoEndpointBase$Bug54716] and the
> following error occurred
>    1 : Error configuring application listener of class
> org.apache.tomcat.util.descriptor.web.ApplicationListener@12cc06
>    1 : Error configuring application listener of class
> org.apache.tomcat.util.descriptor.web.ApplicationListener@1605b09
>    1 : Error configuring application listener of class
> org.apache.tomcat.util.descriptor.web.ApplicationListener@17024fc
>    1 : Error configuring application listener of class
> org.apache.tomcat.util.descriptor.web.ApplicationListener@18896b4
>    1 : Error configuring application listener of class
> org.apache.tomcat.util.descriptor.web.ApplicationListener@4114a7
>    1 : Error configuring application listener of class
> org.apache.tomcat.util.descriptor.web.ApplicationListener@f4b327
The above error message needs fixing.
The above group of messages needs checking.

> The rest looks very similar to TC 7:
> 
> Most of those are of type "Servlet.service() for servlet ... threw
> exception" (137 times, unchanged).
> 
> Second biggest block (unchanged):
> 
> Count    message
>   27     [junit] TIMESTAMP SEVERE [main]
> org.apache.tomcat.util.digester.Digester.startElement Begin event threw
> exception
>    3     [junit] TIMESTAMP SEVERE [main]
> org.apache.tomcat.util.digester.Digester.endElement End event threw
> exception
> 
> Third biggest block is 15 (-4) messages from the mem leak detection
> during context unload (same types as before), but in addition one new
> message
> 
> The web application [] is still processing a request that has yet to
> finish. This is very likely to create a memory leak. You can control the
> time allowed for requests to finish by using the unloadDelay attribute
> of the standard Context implementation.
> 
> in org.apache.catalina.connector.TestMaxConnections for BIO, NIO, APR 2
> times each.
> 
> 
> "Exception" in output of unit tests: 475 times (+256 rel. TC 7).
> 
> Top exception counts (some new entries):

Should be covered by looking at the warnings as per above.

> "unchecked" warnings during build:

See above.

Many thanks for such a thorough analysis.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org