ant-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Levy-Lambert <anto...@gmx.de>
Subject Re: Exec Connection Not Closing
Date Mon, 22 Feb 2010 19:17:12 GMT
Barry Pape wrote:
> I changed the ant task to use sshexec instead of exec with ssh as the
> executable.  I still have the seemingly random hung process.  I executed
> the ant task from outside cruisecontrol and it completed successfully.
> The last time I found a hung process I grabbed a thread dump and am
> including it here.  Thanks for the suggestions and if there are any
> other ideas I'd love to hear them.
>   

Hello Barry,

Do I understand well that in your case CruiseControl is running ant
within its own process ?

If this is the case, there is a way to make CruiseControl start ant as
an external process, I think using a script attribute on the ant builder.
Try this, it might help.

Also, there might be an underlying bug, either in CruiseControl or in
Ant. Opening a ticket either in CruiseControl's JIRA or in ant's
bugzilla depending on which application you think is misbehaving might help.
If CruiseControl is starting ant within the CruiseControl process, I
would say it is primarily a CruiseControl bug.

Regards,

Antoine
> Thanks,
> Barry
>
> [cc]Feb-22 11:12:13 ScriptRunner  - 2010-02-22 11:12:13
> [cc]Feb-22 11:12:13 ScriptRunner  - Full thread dump Java HotSpot(TM)
> Server VM (14.2-b01 mixed mode):
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "Thread-3" prio=3 tid=0x08c5b400
> nid=0x12 sleeping[0xb667c000]
> [cc]Feb-22 11:12:13 ScriptRunner  -    java.lang.Thread.State:
> TIMED_WAITING (sleeping)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at java.lang.Thread.sleep(Native
> Method)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1.run(SSHExec.java:15
> 8)
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "Connect thread
> testb-pub-web01.test.aus.netspend.net session" prio=3 tid=0x08830800
> nid=0x11 runnable [0xb65da000]
> [cc]Feb-22 11:12:13 ScriptRunner  -    java.lang.Thread.State: RUNNABLE
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> java.net.SocketInputStream.socketRead0(Native Method)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> java.net.SocketInputStream.read(SocketInputStream.java:129)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> com.jcraft.jsch.IO.getByte(IO.java:82)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> com.jcraft.jsch.Session.read(Session.java:811)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> com.jcraft.jsch.Session.run(Session.java:1195)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> java.lang.Thread.run(Thread.java:619)
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "Low Memory Detector" daemon prio=3
> tid=0x08140000 nid=0xb runnable [0x00000000]
> [cc]Feb-22 11:12:13 ScriptRunner  -    java.lang.Thread.State: RUNNABLE
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "CompilerThread1" daemon prio=3
> tid=0x0813d000 nid=0xa waiting on condition [0x00000000]
> [cc]Feb-22 11:12:13 ScriptRunner  -    java.lang.Thread.State: RUNNABLE
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "CompilerThread0" daemon prio=3
> tid=0x0813a800 nid=0x9 waiting on condition [0x00000000]
> [cc]Feb-22 11:12:13 ScriptRunner  -    java.lang.Thread.State: RUNNABLE
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "Signal Dispatcher" daemon prio=3
> tid=0x08139000 nid=0x8 waiting on condition [0x00000000][cc]Feb-22
> 11:12:13 ScriptRunner  -    java.lang.Thread.State: RUNNABLE
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "Finalizer" daemon prio=3
> tid=0x08124c00 nid=0x7 in Object.wait() [0xfb0ab000]
> [cc]Feb-22 11:12:13 ScriptRunner  -    java.lang.Thread.State: WAITING
> (on object monitor)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at java.lang.Object.wait(Native
> Method)
> [cc]Feb-22 11:12:13 ScriptRunner  -     - waiting on <0xbaf16bc0> (a
> java.lang.ref.ReferenceQueue$Lock)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> [cc]Feb-22 11:12:13 ScriptRunner  -     - locked <0xbaf16bc0> (a
> java.lang.ref.ReferenceQueue$Lock)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> [cc]Feb-22 11:12:13 ScriptRunner  -
> [cc]Feb-22 11:12:13 ScriptRunner  - "Reference Handler" daemon prio=3
> tid=0x08123400 nid=0x6 in Object.wait() [0xfb0fc000]
> [cc]Feb-22 11:12:13 ScriptRunner  -    java.lang.Thread.State: WAITING
> (on object monitor)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at java.lang.Object.wait(Native
> Method)
> [cc]Feb-22 11:12:13 ScriptRunner  -     - waiting on <0xbb097080> (a
> java.lang.ref.Reference$Lock)
> [cc]Feb-22 11:12:13 ScriptRunner  -     at
> java.lang.Object.wait(Object.java:485)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> [cc]Feb-22 11:12:14 ScriptRunner  -     - locked <0xbb097080> (a
> java.lang.ref.Reference$Lock)
> [cc]Feb-22 11:12:14 ScriptRunner  -
> [cc]Feb-22 11:12:14 ScriptRunner  - "main" prio=3 tid=0x08070800 nid=0x2
> in Object.wait() [0xfe3dd000]
> [cc]Feb-22 11:12:14 ScriptRunner  -    java.lang.Thread.State: WAITING
> (on object monitor)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at java.lang.Object.wait(Native
> Method)
> [cc]Feb-22 11:12:14 ScriptRunner  -     - waiting on <0xf7f41278> (a
> org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> java.lang.Thread.join(Thread.java:1143)
> [cc]Feb-22 11:12:14 ScriptRunner  -     - locked <0xf7f41278> (a
> org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute(SSHExec.java:
> 167)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> java.lang.reflect.Method.invoke(Method.java:597)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
> 05)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Task.perform(Task.java:348)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> java.lang.reflect.Method.invoke(Method.java:597)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
> 05)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Task.perform(Task.java:348)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:3
> 91)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> java.lang.reflect.Method.invoke(Method.java:597)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
> 05)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Task.perform(Task.java:348)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> net.sf.antcontrib.logic.IfTask.execute(IfTask.java:217)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> java.lang.reflect.Method.invoke(Method.java:597)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
> 05)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.TaskAdapter.execute(TaskAdapter.java:134)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> java.lang.reflect.Method.invoke(Method.java:597)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1
> 05)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Task.perform(Task.java:348)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Target.execute(Target.java:357)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Target.performTasks(Target.java:385)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1329)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Project.executeTarget(Project.java:1298)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecut
> or.java:41)[cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Project.executeTargets(Project.java:1181)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Main.runBuild(Main.java:698)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.Main.startAnt(Main.java:199)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
> [cc]Feb-22 11:12:14 ScriptRunner  -     at
> org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
> [cc]Feb-22 11:12:14 ScriptRunner  -
> [cc]Feb-22 11:12:14 ScriptRunner  - "VM Thread" prio=3 tid=0x0811f400
> nid=0x5 runnable
> [cc]Feb-22 11:12:14 ScriptRunner  -
> [cc]Feb-22 11:12:14 ScriptRunner  - "GC task thread#0 (ParallelGC)"
> prio=3 tid=0x08076c00 nid=0x3 runnable
> [cc]Feb-22 11:12:14 ScriptRunner  -
> [cc]Feb-22 11:12:14 ScriptRunner  - "GC task thread#1 (ParallelGC)"
> prio=3 tid=0x08078400 nid=0x4 runnable
> [cc]Feb-22 11:12:14 ScriptRunner  -
> [cc]Feb-22 11:12:14 ScriptRunner  - "VM Periodic Task Thread" prio=3
> tid=0x08141800 nid=0xc waiting on condition
> [cc]Feb-22 11:12:14 ScriptRunner  -
> [cc]Feb-22 11:12:14 ScriptRunner  - JNI global references: 1235
> [cc]Feb-22 11:12:14 ScriptRunner  -
> [cc]Feb-22 11:12:14 ScriptRunner  - Heap
> [cc]Feb-22 11:12:14 ScriptRunner  -  PSYoungGen      total 67136K, used
> 55220K [0xf3e00000, 0xfb000000, 0xfb000000)
> [cc]Feb-22 11:12:14 ScriptRunner  -   eden space 64640K, 81% used
> [0xf3e00000,0xf71b52b8,0xf7d20000)
> [cc]Feb-22 11:12:14 ScriptRunner  -   from space 2496K, 91% used
> [0xf7d20000,0xf7f58010,0xf7f90000)
> [cc]Feb-22 11:12:14 ScriptRunner  -   to   space 2752K, 0% used
> [0xfad50000,0xfad50000,0xfb000000)
> [cc]Feb-22 11:12:14 ScriptRunner  -  PSOldGen        total 69632K, used
> 2971K [0xbae00000, 0xbf200000, 0xf3e00000)
> [cc]Feb-22 11:12:14 ScriptRunner  -   object space 69632K, 4% used
> [0xbae00000,0xbb0e6d80,0xbf200000)
> [cc]Feb-22 11:12:14 ScriptRunner  -  PSPermGen       total 16384K, used
> 7839K [0xb6e00000, 0xb7e00000, 0xbae00000)
> [cc]Feb-22 11:12:14 ScriptRunner  -   object space 16384K, 47% used
> [0xb6e00000,0xb75a7d08,0xb7e00000)
> [cc]Feb-22 11:12:14 ScriptRunner  -
>
>
> -----Original Message-----
> From: Antoine Levy Lambert [mailto:antoine@gmx.de] 
> Sent: Saturday, February 13, 2010 4:36 PM
> To: Ant Users List
> Subject: Re: Exec Connection Not Closing
>
> Barry Pape wrote:
>   
>> Hi Everyone,
>>
>>  
>>
>> I seem to have an issue with an exec task not closing the ssh
>>     
> connection
>   
>> it opens.  Basically we use CruiseControl and Ant to build and deploy
>> our code.  In the CruiseControl build.xml I have an ant task that
>>     
> ssh's
>   
>> to a remote server and starts a shell script.  Once the script
>>     
> completes
>   
>> CruiseControl continues along its merry way...until recently.  Now at
>> seemingly random times, one of these ssh connections will not close
>>     
> thus
>   
>> causing CruiseControl to hang.  If I kill the process, CruiseControl
>> continues like normal.
>>
>>  
>>   
>>     
> I am using CruiseControl and never saw CruiseControl hanging because a 
> build is hanging. However if CruiseControl is configured with only
> one thread, only one build can run at one time.
>   
>> The ant task is:
>>
>>                 <exec executable="ssh" logError="true">
>>
>>                         <arg line="${remote.host} ${sudo.cmd} -u
>> ${username} /usr/local/qa/deploy_build.sh ${deploy.param}" />
>>
>>                 </exec>
>>
>>   
>>     
> You could use the <sshexec/> ant task instead which is shipping with 
> ant, and requires jsch.
> Maybe the underlying problem is that sudo is now configured to request a
>
> password ? or your remote user is expired ?
> You should try to run this ssh command by hand to see what is wrong.
>   
>>  
>>
>> This had been working fine for the past year.  Now we are having
>>     
> issues.
>   
>> It started when we migrated to a new build server.  It has a more
>>     
> recent
>   
>> OS (solaris 2.10 instead of 2.9).
>>
>>  
>>
>> I tried adding the timeout attribute to the exec task, but all that
>> seemed to do was hide the stuck process.
>>
>>  
>>
>> Any thoughts or suggestions are much appreciated!
>>
>>  
>>
>> Thanks,
>>
>> Barry
>>
>>   
>>     


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org


Mime
View raw message