Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: domain of dmikusa@pivotal.io designates
 74.125.82.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <5457DD0A.9030400@christopherschultz.net>
References: <5457D060.20606@sr.unh.edu>
	<CAAqVxAz0bWisa+ZTc-D_-4YCvDug-9vJ+NsTtPtCegbY1b8bFQ@mail.gmail.com>
	<5457DA64.4060004@sr.unh.edu>
	<5457DD0A.9030400@christopherschultz.net>
Date: Mon, 3 Nov 2014 15:23:28 -0500
Message-ID: 
 <CAAqVxAyxskkY_DUp1yxmbQMvF7Rwxx_FCSL-AWxPxa68ZNNmDw@mail.gmail.com>
Subject: Re: troubleshoot tomcat database connection
From: Daniel Mikusa <dmikusa@pivotal.io>
To: Tomcat Users List <users@tomcat.apache.org>
Content-Type: multipart/alternative; boundary=e89a8f3b9b699a4aae0506fa1e80

--e89a8f3b9b699a4aae0506fa1e80
Content-Type: text/plain; charset=UTF-8

On Mon, Nov 3, 2014 at 2:52 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Mark,
>
> On 11/3/14 2:41 PM, Mark Maciolek wrote:
> >
> > On 11/3/2014 2:29 PM, Daniel Mikusa wrote:
> >> On Mon, Nov 3, 2014 at 1:58 PM, Mark Maciolek <mlm@sr.unh.edu>
> >> wrote:
> >>
> >>> hi,
> >>>
> >>> system is running Scientific Linux release 6.5 rpm installed
> >>>
> >>> tomcat6-servlet-2.5-api-6.0.24-78.el6_5.noarch
> >>> tomcat6-javadoc-6.0.24-78.el6_5.noarch
> >>> tomcat6-lib-6.0.24-78.el6_5.noarch
> >>> tomcat-native-1.1.30-1.el6.x86_64
> >>> tomcat6-el-2.1-api-6.0.24-78.el6_5.noarch
> >>> tomcat6-admin-webapps-6.0.24-78.el6_5.noarch
> >>> apache-tomcat-apis-0.1-1.el6.noarch
> >>> tomcat6-jsp-2.1-api-6.0.24-78.el6_5.noarch
> >>> tomcat6-webapps-6.0.24-78.el6_5.noarch
> >>> tomcat-servlet-3.0-api-7.0.33-4.el6.noarch
> >>> tomcat6-6.0.24-78.el6_5.noarch
> >>> tomcat6-docs-webapp-6.0.24-78.el6_5.noarch
> >>>
> >> For what it's worth, 6.0.24 is pretty old.  You should strongly
> >> consider upgrading to 6.0.41, or whatever is the latest at the
> >> time you go to upgrade.
> >>
> >>
> >>> httpd-2.2.15-31.sl6.x86_64
> >>>
> >>> tomcat starts and I can reach the page on the 8080 port
> >>>
> >> Good!
> >>
> >>
> >>> It never starts the connection host name connection from the
> >>> server.xml
> >>>
> >> Hmm, not sure what you mean here.  Are you talking about virtual
> >> hosting with Tomcat?
> >>
> >> http://tomcat.apache.org/tomcat-6.0-doc/virtual-hosting-howto.html
> >>
> >
> >>
> not currently just this one site on the server serving jsp file.
> >>> I do a kill -3 on the tomcat pid and this is what the logs
> >>> shows:
> >>>
> >>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.24 Nov 3, 2014
> >>> 1:19:10 PM org.apache.catalina.startup.HostConfig
> >>> deployDescriptor INFO: Deploying configuration descriptor
> >>> host-manager.xml Nov 3, 2014 1:19:10 PM
> >>> org.apache.catalina.startup.HostConfig deployDescriptor INFO:
> >>> Deploying configuration descriptor manager.xml Nov 3, 2014
> >>> 1:19:10 PM org.apache.catalina.startup.HostConfig
> >>> deployDirectory INFO: Deploying web application directory ROOT
> >>> Nov 3, 2014 1:19:10 PM org.apache.catalina.startup.HostConfig
> >>> deployDirectory INFO: Deploying web application directory
> >>> examples Nov 3, 2014 1:19:11 PM
> >>> org.apache.catalina.startup.HostConfig deployDirectory INFO:
> >>> Deploying web application directory sample Nov 3, 2014 1:19:11
> >>> PM org.apache.catalina.startup.HostConfig deployDirectory INFO:
> >>> Deploying web application directory docs Nov 3, 2014 1:19:11 PM
> >>> org.apache.coyote.http11.Http11Protocol start INFO: Starting
> >>> Coyote HTTP/1.1 on http-8080 Nov 3, 2014 1:19:11 PM
> >>> org.apache.jk.common.ChannelSocket init INFO: JK: ajp13
> >>> listening on /0.0.0.0:8009 Nov 3, 2014 1:19:11 PM
> >>> org.apache.jk.server.JkMain start INFO: Jk running ID=0
> >>> time=0/54  config=null Nov 3, 2014 1:19:11 PM
> >>> org.apache.catalina.startup.Catalina start INFO: Server startup
> >>> in 1307 ms 2014-11-03 13:19:37 Full thread dump OpenJDK 64-Bit
> >>> Server VM (23.25-b01 mixed mode):
> >>>
> >>> "TP-Monitor" daemon prio=10 tid=0x00007feca454d800 nid=0x729f
> >>> in Object.wait() [0x00007feca2410000] java.lang.Thread.State:
> >>> TIMED_WAITING (on object monitor) at
> >>> java.lang.Object.wait(Native Method) - waiting on
> >>> <0x00000000ba74ea78> (a org.apache.tomcat.util.
> >>> threads.ThreadPool$MonitorRunnable) at
> >>> org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(
> >>> ThreadPool.java:565) - locked <0x00000000ba74ea78> (a
> >>> org.apache.tomcat.util. threads.ThreadPool$MonitorRunnable) at
> >>> java.lang.Thread.run(Thread.java:701)
> >>>
> >>> "TP-Processor4" daemon prio=10 tid=0x00007feca452f000
> >>> nid=0x729e runnable [0x00007feca2511000]
> >>> java.lang.Thread.State: RUNNABLE at
> >>> java.net.PlainSocketImpl.socketAccept(Native Method) at
> >>> java.net.AbstractPlainSocketImpl.accept(
> >>> AbstractPlainSocketImpl.java:375) at
> >>> java.net.ServerSocket.implAccept(ServerSocket.java:478) at
> >>> java.net.ServerSocket.accept(ServerSocket.java:446) at
> >>> org.apache.jk.common.ChannelSocket.accept(
> >>> ChannelSocket.java:307) at
> >>> org.apache.jk.common.ChannelSocket.acceptConnections(
> >>> ChannelSocket.java:661) at
> >>> org.apache.jk.common.ChannelSocket$SocketAcceptor.
> >>> runIt(ChannelSocket.java:872) at
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> >>> ThreadPool.java:690) at java.lang.Thread.run(Thread.java:701)
> >>>
> >>> "TP-Processor3" daemon prio=10 tid=0x00007feca452d000
> >>> nid=0x729d in Object.wait() [0x00007feca2612000]
> >>> java.lang.Thread.State: WAITING (on object monitor) at
> >>> java.lang.Object.wait(Native Method) - waiting on
> >>> <0x00000000ba74e3a8> (a org.apache.tomcat.util.
> >>> threads.ThreadPool$ControlRunnable) at
> >>> java.lang.Object.wait(Object.java:502) at
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> >>> ThreadPool.java:662) - locked <0x00000000ba74e3a8> (a
> >>> org.apache.tomcat.util. threads.ThreadPool$ControlRunnable) at
> >>> java.lang.Thread.run(Thread.java:701)
> >>>
> >>> "TP-Processor2" daemon prio=10 tid=0x00007feca4532000
> >>> nid=0x729c in Object.wait() [0x00007feca2713000]
> >>> java.lang.Thread.State: WAITING (on object monitor) at
> >>> java.lang.Object.wait(Native Method) - waiting on
> >>> <0x00000000ba74e010> (a org.apache.tomcat.util.
> >>> threads.ThreadPool$ControlRunnable) at
> >>> java.lang.Object.wait(Object.java:502) at
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> >>> ThreadPool.java:662) - locked <0x00000000ba74e010> (a
> >>> org.apache.tomcat.util. threads.ThreadPool$ControlRunnable) at
> >>> java.lang.Thread.run(Thread.java:701)
> >>>
> >>> "TP-Processor1" daemon prio=10 tid=0x00007feca452a000
> >>> nid=0x729b in Object.wait() [0x00007feca2814000]
> >>> java.lang.Thread.State: WAITING (on object monitor) at
> >>> java.lang.Object.wait(Native Method) - waiting on
> >>> <0x00000000ba74dc60> (a org.apache.tomcat.util.
> >>> threads.ThreadPool$ControlRunnable) at
> >>> java.lang.Object.wait(Object.java:502) at
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> >>> ThreadPool.java:662) - locked <0x00000000ba74dc60> (a
> >>> org.apache.tomcat.util. threads.ThreadPool$ControlRunnable) at
> >>> java.lang.Thread.run(Thread.java:701)
> >>>
> >>> "http-8080-Acceptor-0" daemon prio=10 tid=0x00007feca44fb000
> >>> nid=0x7299 runnable [0x00007feca2915000]
> >>> java.lang.Thread.State: RUNNABLE at
> >>> java.net.PlainSocketImpl.socketAccept(Native Method) at
> >>> java.net.AbstractPlainSocketImpl.accept(
> >>> AbstractPlainSocketImpl.java:375) at
> >>> java.net.ServerSocket.implAccept(ServerSocket.java:478) at
> >>> java.net.ServerSocket.accept(ServerSocket.java:446) at
> >>> org.apache.tomcat.util.net.DefaultServerSocketFactory.
> >>> acceptSocket(DefaultServerSocketFactory.java:61) at
> >>> org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(
> >>> JIoEndpoint.java:352) at java.lang.Thread.run(Thread.java:701)
> >>>
> >>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon
> >>> prio=10 tid=0x00007feca4501000 nid=0x7298 waiting on condition
> >>> [0x00007feca2a16000] java.lang.Thread.State: TIMED_WAITING
> >>> (sleeping) at java.lang.Thread.sleep(Native Method) at
> >>> org.apache.catalina.core.ContainerBase$
> >>> ContainerBackgroundProcessor.run(ContainerBase.java:1579) at
> >>> java.lang.Thread.run(Thread.java:701)
> >>>
> >>> "GC Daemon" daemon prio=10 tid=0x00007feca440e800 nid=0x7297
> >>> in Object.wait() [0x00007feca2d2d000] java.lang.Thread.State:
> >>> TIMED_WAITING (on object monitor) at
> >>> java.lang.Object.wait(Native Method) - waiting on
> >>> <0x00000000cdd49190> (a sun.misc.GC$LatencyLock) at
> >>> sun.misc.GC$Daemon.run(GC.java:117) - locked
> >>> <0x00000000cdd49190> (a sun.misc.GC$LatencyLock)
> >>>
> >>> "Low Memory Detector" daemon prio=10 tid=0x00007feca40f0000
> >>> nid=0x7295 runnable [0x0000000000000000]
> >>> java.lang.Thread.State: RUNNABLE
> >>>
> >>> "C2 CompilerThread1" daemon prio=10 tid=0x00007feca40ed800
> >>> nid=0x7294 waiting on condition [0x0000000000000000]
> >>> java.lang.Thread.State: RUNNABLE
> >>>
> >>> "C2 CompilerThread0" daemon prio=10 tid=0x00007feca40eb000
> >>> nid=0x7293 waiting on condition [0x0000000000000000]
> >>> java.lang.Thread.State: RUNNABLE
> >>>
> >>> "Signal Dispatcher" daemon prio=10 tid=0x00007feca40e9000
> >>> nid=0x7292 waiting on condition [0x0000000000000000]
> >>> java.lang.Thread.State: RUNNABLE
> >>>
> >>> "Finalizer" daemon prio=10 tid=0x00007feca409a800 nid=0x7291
> >>> in Object.wait() [0x00007feca3c65000] java.lang.Thread.State:
> >>> WAITING (on object monitor) at java.lang.Object.wait(Native
> >>> Method) - waiting on <0x00000000cdb55a98> (a
> >>> java.lang.ref.ReferenceQueue$ Lock) at
> >>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) -
> >>> locked <0x00000000cdb55a98> (a java.lang.ref.ReferenceQueue$
> >>> Lock) at
> >>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
> >>> at
> >>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
> >>>
> >>>
> >>>
> "Reference Handler" daemon prio=10 tid=0x00007feca4098800 nid=0x7290 in
> >>> Object.wait() [0x00007feca3d66000] java.lang.Thread.State:
> >>> WAITING (on object monitor) at java.lang.Object.wait(Native
> >>> Method) - waiting on <0x00000000cdb557c8> (a
> >>> java.lang.ref.Reference$Lock) at
> >>> java.lang.Object.wait(Object.java:502) at
> >>> java.lang.ref.Reference$ReferenceHandler.run(
> >>> Reference.java:133) - locked <0x00000000cdb557c8> (a
> >>> java.lang.ref.Reference$Lock)
> >>>
> >>> "main" prio=10 tid=0x00007feca4009000 nid=0x728d runnable
> >>> [0x00007fecabf19000] java.lang.Thread.State: RUNNABLE at
> >>> java.net.PlainSocketImpl.socketAccept(Native Method) at
> >>> java.net.AbstractPlainSocketImpl.accept(
> >>> AbstractPlainSocketImpl.java:375) at
> >>> java.net.ServerSocket.implAccept(ServerSocket.java:478) at
> >>> java.net.ServerSocket.accept(ServerSocket.java:446) at
> >>> org.apache.catalina.core.StandardServer.await(
> >>> StandardServer.java:389) at
> >>> org.apache.catalina.startup.Catalina.await(Catalina.java:657)
> >>> at
> >>> org.apache.catalina.startup.Catalina.start(Catalina.java:617)
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke(
> >>> NativeMethodAccessorImpl.java:57) at
> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> >>> DelegatingMethodAccessorImpl.java:43) at
> >>> java.lang.reflect.Method.invoke(Method.java:622) at
> >>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> >>>
> >>>
> at
> >>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >>>
> >>> "VM Thread" prio=10 tid=0x00007feca4089800 nid=0x728f runnable
> >>>
> >>> "VM Periodic Task Thread" prio=10 tid=0x00007feca40fb000
> >>> nid=0x7296 waiting on condition
> >>>
> >>> JNI global references: 231
> >>>
> >>> Heap def new generation   total 18432K, used 13769K
> >>> [0x00000000b9c00000, 0x00000000bb000000, 0x00000000cdb50000)
> >>> eden space 16384K,  74% used [0x00000000b9c00000,
> >>> 0x00000000ba7f1588, 0x00000000bac00000) from space 2048K,  75%
> >>> used [0x00000000bae00000, 0x00000000baf81118,
> >>> 0x00000000bb000000) to   space 2048K,   0% used
> >>> [0x00000000bac00000, 0x00000000bac00000, 0x00000000bae00000)
> >>> tenured generation   total 40896K, used 2039K
> >>> [0x00000000cdb50000, 0x00000000d0340000, 0x00000000f5a00000)
> >>> the space 40896K,   4% used [0x00000000cdb50000,
> >>> 0x00000000cdd4df20, 0x00000000cdd4e000, 0x00000000d0340000)
> >>> compacting perm gen  total 21248K, used 12267K
> >>> [0x00000000f5a00000, 0x00000000f6ec0000, 0x0000000100000000)
> >>> the space 21248K,  57% used [0x00000000f5a00000,
> >>> 0x00000000f65faf88, 0x00000000f65fb000, 0x00000000f6ec0000) No
> >>> shared spaces configured.
> >>>
> >>>
> >>> Is the issue with the No shared spaces configured or is it
> >>> more likely the java code being called to make the startup
> >>> connection?
> >>>
> >>> Any pointers in how to troubleshoot this problem would be
> >>> greatly appreciated.
> >>>
> >> It's not really clear what's failing here.  Can you be more
> >> specific about what isn't working?  Perhaps provide an example of
> >> what fails and what you expect should happen.
> >>
> >> Dan
> >>
> >
> > On a working server with similar setup this is what shows in the
> > catalina.out log
> >
> > 2014-11-03 11:34:59  Config Instatiated: {OS_OWNER=ag,
> > TOMCAT_GROUP=tomcat, DB_USER=dbuser, SUPPORT_PHONE=603-862-0348,
> > BASE_PATH=/data1/picturepost, TOMCAT_HOME=/usr/share/tomcat6,
> > MAP_FILE=/data1/picturepost/cgi-bin/modis.map,
> > CONVERT=/usr/bin/convert, DB_PORT=5432,
> >
> GOOGLE_MAPS_KEY=ABQIAAAAXLtB9cZEiOZ_WJC3vUyI2RQG0VXM1isChuPqZL77qjLf3s1C-hSFtmTt8t6XLAWqQA-40gXr1h4CJg,
> >
> >
> PICTURE_DIR=/data1/picturepost/data/pictures, APACHE_GROUP=apache,
> > MAX_FILE_UPLOAD_SIZE=10485760, MAX_DB_CONNECTION_TIME=120,
> > PICTURE_DIR_URL=/images/pictures,
> > APACHE_SERVER_NAME=picturepost.unh.edu,
> > SUPPORT_EMAIL=picturepostsupport@picturepost.unh.edu,
> > HOST=/data1/picturepost/logs/picturepost.log,
> > DOCUMENT_ROOT_DIR=/data1/picturepost/webapps/ROOT,
> > DB_PASSWORD=n0rMal1, DB_ADMIN_USER=postgres, MODE=live,
> > NUM_DB_CONNECTIONS=20, URL=http://picturepost.unh.edu,
> > NAME=picturepost, LOG=/data1/picturepost/logs/picturepost.log,
> > RECAPTCHA_PUBLIC_KEY=6Lckb8USAAAAAPU_hxR1X6lmNXO1goLP_vLPGlnF,
> > DB_HOST_IP=127.0.0.1,
> > LAYERS_FILE=/data1/picturepost/cgi-bin/modis_layers.map,
> > DATABASE=picturepost, PHPBB_PTA_SECRET=PAGoJcPsNJuN0GsjuOZpB,
> > DB_ADMIN_PASSWORD=,
> > RECAPTCHA_PRIVATE_KEY=6Lckb8USAAAAAAPOLqa9bls_ss80YStO5PyPseVK,
> > OS_GROUP=users, APACHE_ADDR_PORT=*:80,
> > CONFIG_FILE=/data1/picturepost/config/picturepost.cfg,
> > EMAIL_COMMAND=/usr/sbin/sendmail -oi -t}
>
> I've never seen anything like that ever before. Is that some custom
> component that is generating that logging?
>

+1 You probably need to ask your developer more about how to debug this, as
it doesn't seem like a Tomcat problem.


>
> > So on the server that is not working I never see the Config
> > Instatiated
> >
> > <Host name="centaur.sr.unh.edu" appBase="/web/opal/webapps">
> > <Context docBase="ROOT" path="" allowLinking="true"> <Parameter
> > name="BASE_PATH" value="/web/opal"/> </Context>
>
> The above is probably going to double-deploy your web application or
> otherwise do something weird. Don't put your <Context> in server.xml.
> Instead, declare the <Context> element in your web application's
> META-INF/context.xml file, where you can put things like your
> BASE_PATH parameter.
>

+1

>From the Tomcat logging above, you can see that an app is deployed to the
ROOT context.  You might want to check and see if it's really your
application that's being deployed.  Perhaps the default ROOT application is
being deployed instead.  What happens when you try to access the app?  I
believe you said that previously that you could reach Tomcat no port 8080.
Is it displaying the expected page?  or perhaps the Tomcat default page?

Dan


>
> > </Host>
> >
> > We leaning towards something wrong in the StartupCooa.java file
> > but since I never see the attempt to startup the config in the
> > catalina.out file I suspect some else might be wrong.
>
> I've never heard of StartupCooa.java.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJUV90KAAoJEBzwKT+lPKRYVRQQAIjuF6qmOOzm8Dc4NT2MPpVo
> AsZQqzRXwuH+XXl1OwQOzZlicQxGJGfwDfHM39yUoFyKhApVFmUt4umOU9f9gAQY
> 25VqkxPw35A3kPsYhw1qkON3f0dnKpwg70d8XKrgK9o47zfB3KXIqgeKkVjDbZNC
> i9xJgnY70ChRtqdSjz1ydilf57r+ya0sK7MUO3PKlyusJ0XDYREPW97+Yxc/Raps
> URwq/d9XDqdz78cFRst1WrtgevcO4Q0XQj+fI5GkOgcCquG647iTcOACcGQkAhBp
> 342O0CW1nNQ46+bcSG7lLjtJ9tFYzuTSLXRvh4otnoUlL1ZF+9omoyFMliKyCISd
> aZmjZg8d4RpGPQDU+CrFN6WCSKKQPRlo1ruEvOuktx7e2vbaUaE2IFogWLWAwNSj
> 1dzaW2HgwzJZgbbByi1EzIzVBT+BStrWMN3x5+QJ0yC++WIRu6MVRzfOxeAA4wea
> VpNXu15/61ff9IAbucVUnwPXz5MruOKCP3qLaWXorIH/Rt7haItTDqaSmsY5bGpT
> c+GYPLHq5NFxIACe4oYeY6ov94C8TxAYS1QGT7DA+j7f+vMF1OhgS6coNktImXiA
> hqxl5Jm2GzcJZC1PFWzTOCigtfteA9GZjxDjCyCS+wt10qHIc2hJXKMFGSyUQa2y
> LFOiTYJ3LtOiw7EE/LPM
> =RFAe
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

--e89a8f3b9b699a4aae0506fa1e80--