airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: Error while launching Gaussian experiments on gw111 server
Date Mon, 03 Aug 2015 15:28:05 GMT
Hi Dimuthu,

do you still have this issue?I just checked and gw111 has access to comet
resource.

Thanks,
Shameera.

On Sat, Aug 1, 2015 at 5:05 AM DImuthu Upeksha <dimuthu.upeksha2@gmail.com>
wrote:

> Hi,
>
> I'm getting following error when I tried to launch a Gaussian experiment
> on gw111.iu.xsede.org <airavata@gw111.iu.xsede.org> server. What could be
> the reason for that?
>
> Application interface: Gaussian_cc471eee-35e0-4d2b-8cdc-0023481dda90
> Resource host id: comet-ln1.sdsc.edu_0bb9bd78-b5e7-40cf-a5dd-fd6f8bd6b537
>
> 2015-08-01 04:54:58,754 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:08,756 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:18,756 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:28,756 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:38,757 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:48,758 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:54,691 [pool-3-thread-23] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher  -
> Publishing status to rabbitmq...
> 2015-08-01 04:55:54,700 [pool-3-thread-23] INFO
>  org.apache.airavata.api.server.handler.AiravataServerHandler  -
> Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : Created
> new experiment with experiment name dimuthu_experiment_gau4
> 2015-08-01 04:55:58,758 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:08,759 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:18,760 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:28,760 [Thread-18] INFO
>  org.apache.airavata.gfac.monitor.email.EmailBasedMonitor  - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:31,929 [pool-10-thread-17] INFO
>  org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl  -
> Validation of
> org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is
> SUCCESSFUL
> 2015-08-01 04:56:31,931 [pool-10-thread-17] INFO
>  org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl  -
> Validation of
> org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator
> is SUCCESSFUL
> 2015-08-01 04:56:32,843 [pool-18-thread-14] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher  -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:33,025 [pool-18-thread-14] INFO
>  org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry  -
> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
> TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
> 2015-08-01 04:56:33,592 [pool-18-thread-14] INFO
>  org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry  -
> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
> TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
> 2015-08-01 04:56:34,060 [pool-18-thread-14] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher  -
> Publishing to launch queue ...
> 2015-08-01 04:56:34,061 [pool-18-thread-14] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher  -
> Successfully published to launch queue ...
> 2015-08-01 04:56:34,130 [pool-13-thread-9] INFO
>  org.apache.airavata.gfac.core.utils.GFacUtils  - This is a new Job, so
> creating all the experiment docs from the scratch
> 2015-08-01 04:56:34,147 [pool-13-thread-9] INFO
>  org.apache.airavata.gfac.core.utils.GFacUtils  - Created the node:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/state
> successfully !
> 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
>  org.apache.airavata.gfac.server.GfacServerHandler  -
> -----------------------------------------------------14-----------------------------------------------------
> 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
>  org.apache.airavata.gfac.server.GfacServerHandler  -
> Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : GFac
> Received submit job request for the Experiment:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b TaskId:
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6
> 2015-08-01 04:56:34,749 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID  - There is not job created
> at this point
> 2015-08-01 04:56:34,750 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  - Gfac
> internal state changed to: ACCEPTED
> 2015-08-01 04:56:34,831 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID  - There is not job created
> at this point
> 2015-08-01 04:56:34,831 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  - Gfac
> internal state changed to: INHANDLERSINVOKING
> 2015-08-01 04:56:34,880 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler  -
> Invoking ApplicationDescriptorCheckHandler ...
> 2015-08-01 04:56:34,886 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler  -
> Recoverable data is saving to zk:
> /oasis/scratch/comet/ogce/temp_project/gta-work-dirs,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stdout,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stderr
> 2015-08-01 04:56:35,765 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException  - Error occurred...
> org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
> while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,766 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler  - Error
> occurred...
> 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException  - Error while creating
> SSHSecurityContext
> org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.handler.GFacHandlerException  - Error while
> creating SSHSecurityContext
> org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException  - Error Executing a InFlow Handler
> org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException  - Error Invoking Handlers:Error
> Executing a InFlow Handler
> org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID  - There is not job created
> at this point
> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  - Gfac
> internal state changed to: FAILED
> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  -
> Experiment Failed, So removing the ZK entry for the
> experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  -
> Zookeeper experiment Path:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:35,831 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl  - Error occured during
> updating the statuses of Experiments,tasks or Job statuses to failed,
> NullPointerException occurred because at this point there might not have
> Job Created
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,832 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator  -
> expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
> Updating workflow node status for
> tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
> 2015-08-01 04:56:36,070 [pool-19-thread-28] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher  -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:36,656 [pool-19-thread-28] INFO
>  org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator  -
> Updating the experiment status of experiment:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
> 2015-08-01 04:56:36,715 [pool-19-thread-28] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher  -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:36,718 [pool-19-thread-28] INFO
>  org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator  -
> Clean up task - Acknowledging delivery Tag - 14
> 2015-08-01 04:56:36,770 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException  - Error Invoking Handlers:Error
> Executing a InFlow Handler
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException  - Error launching the Job
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> ... 6 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl  - Error inovoking the job
> with experiment ID:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:Error
> launching the Job
> 2015-08-01 04:56:36,868 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID  - There is not job created
> at this point
> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  - Gfac
> internal state changed to: FAILED
> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  -
> Experiment Failed, So removing the ZK entry for the
> experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator  -
> Zookeeper experiment Path:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
>  org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry  -
> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
> TaskStatus(executionState:FAILED, timeOfStateChange:1438419396978).
> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher  -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
>  org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator  -
> expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
> Updating workflow node status for
> tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
> 2015-08-01 04:56:37,267 [pool-19-thread-28] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher  -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:37,889 [pool-19-thread-28] INFO
>  org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator  -
> Updating the experiment status of experiment:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
> 2015-08-01 04:56:37,947 [pool-19-thread-28] INFO
>  org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher  -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:37,949 [pool-19-thread-28] ERROR
> org.apache.airavata.common.utils.AiravataZKUtils  - Cannot find delivery
> Tag in
> path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b-deliveryTag
> for this experiment
> 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException  - Error launching the Job
> org.apache.airavata.gfac.GFacException: Error launching the Job
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> ... 5 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> ... 6 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.utils.InputHandlerWorker  -
> org.apache.airavata.gfac.GFacException: Error launching the Job
> org.apache.airavata.gfac.GFacException:
> org.apache.airavata.gfac.GFacException: Error launching the Job
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:196)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> ... 4 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> ... 5 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> ... 6 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
>
> --
> Regards
>
> W.Dimuthu Upeksha
> Undergraduate
> Department of Computer Science And Engineering
>
> University of Moratuwa, Sri Lanka
>
-- 
Shameera Rathnayaka

Mime
View raw message