Return-Path: X-Original-To: apmail-airavata-commits-archive@www.apache.org Delivered-To: apmail-airavata-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1162B17A09 for ; Thu, 14 May 2015 19:23:48 +0000 (UTC) Received: (qmail 9113 invoked by uid 500); 14 May 2015 19:23:47 -0000 Delivered-To: apmail-airavata-commits-archive@airavata.apache.org Received: (qmail 9074 invoked by uid 500); 14 May 2015 19:23:47 -0000 Mailing-List: contact commits-help@airavata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airavata.apache.org Delivered-To: mailing list commits@airavata.apache.org Received: (qmail 9064 invoked by uid 99); 14 May 2015 19:23:47 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2015 19:23:47 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id AE610DFC81; Thu, 14 May 2015 19:23:47 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: shameera@apache.org To: commits@airavata.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: airavata git commit: Added UGE Configuration class and rename SGE to UGE to resolve name conflicts. Date: Thu, 14 May 2015 19:23:47 +0000 (UTC) Repository: airavata Updated Branches: refs/heads/master acf30c013 -> 86a79b669 Added UGE Configuration class and rename SGE to UGE to resolve name conflicts. Project: http://git-wip-us.apache.org/repos/asf/airavata/repo Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/86a79b66 Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/86a79b66 Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/86a79b66 Branch: refs/heads/master Commit: 86a79b669c51bb1d464d860256057d793de38fca Parents: acf30c0 Author: shamrath Authored: Thu May 14 15:23:45 2015 -0400 Committer: shamrath Committed: Thu May 14 15:23:45 2015 -0400 ---------------------------------------------------------------------- .../server/src/main/resources/SGETemplate.xslt | 78 --------- .../server/src/main/resources/UGETemplate.xslt | 78 +++++++++ .../gsi/ssh/api/job/SGEOutputParser.java | 173 ------------------- .../gsi/ssh/api/job/UGEJobConfiguration.java | 119 +++++++++++++ .../gsi/ssh/api/job/UGEOutputParser.java | 173 +++++++++++++++++++ .../airavata/gsi/ssh/util/CommonUtils.java | 3 +- .../gsissh/src/main/resources/SGETemplate.xslt | 74 -------- .../gsissh/src/main/resources/UGETemplate.xslt | 74 ++++++++ 8 files changed, 445 insertions(+), 327 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/modules/configuration/server/src/main/resources/SGETemplate.xslt ---------------------------------------------------------------------- diff --git a/modules/configuration/server/src/main/resources/SGETemplate.xslt b/modules/configuration/server/src/main/resources/SGETemplate.xslt deleted file mode 100644 index 7005295..0000000 --- a/modules/configuration/server/src/main/resources/SGETemplate.xslt +++ /dev/null @@ -1,78 +0,0 @@ - - - - - " -#! /bin/bash -# Grid Engine batch job script built by Apache Airavata -# - -#$ -S - -#$ -V - - -#$ -q - - -#$ -m beas - -#$ -M - - - - -#$ -A - - - - -#$ -l h_rt= - - - - -#$ -N - - - - -#$ -o - - - - -#$ -e - - - - -#$ -pe way - - - - -= -export - - - - - -cd - - - - - - - - - - \ No newline at end of file http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/modules/configuration/server/src/main/resources/UGETemplate.xslt ---------------------------------------------------------------------- diff --git a/modules/configuration/server/src/main/resources/UGETemplate.xslt b/modules/configuration/server/src/main/resources/UGETemplate.xslt new file mode 100644 index 0000000..7005295 --- /dev/null +++ b/modules/configuration/server/src/main/resources/UGETemplate.xslt @@ -0,0 +1,78 @@ + + + + + " +#! /bin/bash +# Grid Engine batch job script built by Apache Airavata +# + +#$ -S + +#$ -V + + +#$ -q + + +#$ -m beas + +#$ -M + + + + +#$ -A + + + + +#$ -l h_rt= + + + + +#$ -N + + + + +#$ -o + + + + +#$ -e + + + + +#$ -pe way + + + + += +export + + + + + +cd + + + + + + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/SGEOutputParser.java ---------------------------------------------------------------------- diff --git a/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/SGEOutputParser.java b/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/SGEOutputParser.java deleted file mode 100644 index 884b7f3..0000000 --- a/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/SGEOutputParser.java +++ /dev/null @@ -1,173 +0,0 @@ -/* - * - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - * -*/ -package org.apache.airavata.gsi.ssh.api.job; - -import org.apache.airavata.gsi.ssh.api.SSHApiException; -import org.apache.airavata.gsi.ssh.impl.JobStatus; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.ArrayList; -import java.util.List; -import java.util.Map; - -public class SGEOutputParser implements OutputParser{ - private static final Logger log = LoggerFactory.getLogger(PBSOutputParser.class); - - public void parseSingleJob(JobDescriptor jobDescriptor, String rawOutput) { - log.debug(rawOutput); - String[] info = rawOutput.split("\n"); - String[] line; - for (int i = 0; i < info.length; i++) { - if (info[i].contains("=")) { - line = info[i].split("=", 2); - } else { - line = info[i].split(":", 2); - } - if (line.length >= 2) { - String header = line[0].trim(); - log.debug("Header = " + header); - String value = line[1].trim(); - log.debug("value = " + value); - - if (header.equals("Variable_List")) { - while (info[i + 1].startsWith("\t")) { - value += info[i + 1]; - i++; - } - value = value.replaceAll("\t", ""); - jobDescriptor.setVariableList(value); - } else if ("Job Id".equals(header)) { - jobDescriptor.setJobID(value); - } else if ("Job_Name".equals(header)) { - jobDescriptor.setJobName(value); - } else if ("Account_Name".equals(header)) { - jobDescriptor.setAcountString(value); - } else if ("job_state".equals(header)) { - jobDescriptor.setStatus(value); - } else if ("Job_Owner".equals(header)) { - jobDescriptor.setOwner(value); - } else if ("resources_used.cput".equals(header)) { - jobDescriptor.setUsedCPUTime(value); - } else if ("resources_used.mem".equals(header)) { - jobDescriptor.setUsedMemory(value); - } else if ("resources_used.walltime".equals(header)) { - jobDescriptor.setEllapsedTime(value); - } else if ("job_state".equals(header)) { - jobDescriptor.setStatus(value); - } else if ("queue".equals(header)) - jobDescriptor.setQueueName(value); - else if ("ctime".equals(header)) { - jobDescriptor.setCTime(value); - } else if ("qtime".equals(header)) { - jobDescriptor.setQTime(value); - } else if ("mtime".equals(header)) { - jobDescriptor.setMTime(value); - } else if ("start_time".equals(header)) { - jobDescriptor.setSTime(value); - } else if ("comp_time".equals(header)) { - jobDescriptor.setCompTime(value); - } else if ("exec_host".equals(header)) { - jobDescriptor.setExecuteNode(value); - } else if ("Output_Path".equals(header)) { - if (info[i + 1].contains("=") || info[i + 1].contains(":")) - jobDescriptor.setStandardOutFile(value); - else { - jobDescriptor.setStandardOutFile(value + info[i + 1].trim()); - i++; - } - } else if ("Error_Path".equals(header)) { - if (info[i + 1].contains("=") || info[i + 1].contains(":")) - jobDescriptor.setStandardErrorFile(value); - else { - String st = info[i + 1].trim(); - jobDescriptor.setStandardErrorFile(value + st); - i++; - } - - } else if ("submit_args".equals(header)) { - while (i + 1 < info.length) { - if (info[i + 1].startsWith("\t")) { - value += info[i + 1]; - i++; - } else - break; - } - value = value.replaceAll("\t", ""); - jobDescriptor.setSubmitArgs(value); - } - } - } - } - - public String parseJobSubmission(String rawOutput) { - log.debug(rawOutput); - if (rawOutput != null && !rawOutput.isEmpty()) { - String[] info = rawOutput.split("\n"); - String lastLine = info[info.length - 1]; - return lastLine.split(" ")[2]; // In PBS stdout is going to be directly the jobID - } else { - return ""; - } - } - - public JobStatus parseJobStatus(String jobID, String rawOutput) { - // not implemented to sun grid engine - return JobStatus.U; - } - - public void parseJobStatuses(String userName, Map statusMap, String rawOutput) { - log.debug(rawOutput); - String[] info = rawOutput.split("\n"); - int lastStop = 0; - for (String jobID : statusMap.keySet()) { - for(int i=lastStop;i columnList = new ArrayList(); - for (String s : columns) { - if (!"".equals(s)) { - columnList.add(s); - } - } - lastStop = i+1; - if ("E".equals(columnList.get(4))) { - // There is another status with the same letter E other than error status - // to avoid that we make a small tweek to the job status - columnList.set(4, "Er"); - } - statusMap.put(jobID, JobStatus.valueOf(columnList.get(4))); - break; - } - } - } - } - - @Override - public String parseJobId(String jobName, String rawOutput) throws SSHApiException { - return null; // TODO: Implement the parse logic ( with regex if possible ). - } - - -} http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEJobConfiguration.java ---------------------------------------------------------------------- diff --git a/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEJobConfiguration.java b/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEJobConfiguration.java new file mode 100644 index 0000000..8817c06 --- /dev/null +++ b/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEJobConfiguration.java @@ -0,0 +1,119 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * +*/ +package org.apache.airavata.gsi.ssh.api.job; + +import org.apache.airavata.gsi.ssh.impl.RawCommandInfo; +import org.apache.commons.io.FilenameUtils; + +import java.io.File; + +public class UGEJobConfiguration implements JobManagerConfiguration { + + private String jobDescriptionTemplateName; + + private String scriptExtension; + + private String installedPath; + + private OutputParser parser; + + public UGEJobConfiguration() { + // this can be used to construct and use setter methods to set all the params in order + } + + public UGEJobConfiguration(String jobDescriptionTemplateName, + String scriptExtension, String installedPath, OutputParser parser) { + this.jobDescriptionTemplateName = jobDescriptionTemplateName; + this.scriptExtension = scriptExtension; + this.parser = parser; + if (installedPath.endsWith("/")) { + this.installedPath = installedPath; + } else { + this.installedPath = installedPath + "/"; + } + } + + public RawCommandInfo getCancelCommand(String jobID) { + return new RawCommandInfo(this.installedPath + "qdel " + jobID); + } + + public String getJobDescriptionTemplateName() { + return jobDescriptionTemplateName; + } + + public void setJobDescriptionTemplateName(String jobDescriptionTemplateName) { + this.jobDescriptionTemplateName = jobDescriptionTemplateName; + } + + public RawCommandInfo getMonitorCommand(String jobID) { + return new RawCommandInfo(this.installedPath + "qstat -j " + jobID); + } + + public String getScriptExtension() { + return scriptExtension; + } + + public RawCommandInfo getSubmitCommand(String workingDirectory, String pbsFilePath) { + return new RawCommandInfo(this.installedPath + "qsub " + + workingDirectory + File.separator + FilenameUtils.getName(pbsFilePath)); + } + + public String getInstalledPath() { + return installedPath; + } + + public void setInstalledPath(String installedPath) { + this.installedPath = installedPath; + } + + public OutputParser getParser() { + return parser; + } + + public void setParser(OutputParser parser) { + this.parser = parser; + } + + public RawCommandInfo getUserBasedMonitorCommand(String userName) { + return new RawCommandInfo(this.installedPath + "qstat -u " + userName); + } + + @Override + public RawCommandInfo getJobIdMonitorCommand(String jobName, String userName) { + // For PBS there is no option to get jobDetails by JobName, so we search with userName + return new RawCommandInfo(this.installedPath + "qstat -u " + userName); + } + + @Override + public String getBaseCancelCommand() { + return "qdel"; + } + + @Override + public String getBaseMonitorCommand() { + return "qstat"; + } + + @Override + public String getBaseSubmitCommand() { + return "qsub "; + } +} http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEOutputParser.java ---------------------------------------------------------------------- diff --git a/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEOutputParser.java b/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEOutputParser.java new file mode 100644 index 0000000..645b975 --- /dev/null +++ b/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/api/job/UGEOutputParser.java @@ -0,0 +1,173 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * +*/ +package org.apache.airavata.gsi.ssh.api.job; + +import org.apache.airavata.gsi.ssh.api.SSHApiException; +import org.apache.airavata.gsi.ssh.impl.JobStatus; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +public class UGEOutputParser implements OutputParser{ + private static final Logger log = LoggerFactory.getLogger(PBSOutputParser.class); + + public void parseSingleJob(JobDescriptor jobDescriptor, String rawOutput) { + log.debug(rawOutput); + String[] info = rawOutput.split("\n"); + String[] line; + for (int i = 0; i < info.length; i++) { + if (info[i].contains("=")) { + line = info[i].split("=", 2); + } else { + line = info[i].split(":", 2); + } + if (line.length >= 2) { + String header = line[0].trim(); + log.debug("Header = " + header); + String value = line[1].trim(); + log.debug("value = " + value); + + if (header.equals("Variable_List")) { + while (info[i + 1].startsWith("\t")) { + value += info[i + 1]; + i++; + } + value = value.replaceAll("\t", ""); + jobDescriptor.setVariableList(value); + } else if ("Job Id".equals(header)) { + jobDescriptor.setJobID(value); + } else if ("Job_Name".equals(header)) { + jobDescriptor.setJobName(value); + } else if ("Account_Name".equals(header)) { + jobDescriptor.setAcountString(value); + } else if ("job_state".equals(header)) { + jobDescriptor.setStatus(value); + } else if ("Job_Owner".equals(header)) { + jobDescriptor.setOwner(value); + } else if ("resources_used.cput".equals(header)) { + jobDescriptor.setUsedCPUTime(value); + } else if ("resources_used.mem".equals(header)) { + jobDescriptor.setUsedMemory(value); + } else if ("resources_used.walltime".equals(header)) { + jobDescriptor.setEllapsedTime(value); + } else if ("job_state".equals(header)) { + jobDescriptor.setStatus(value); + } else if ("queue".equals(header)) + jobDescriptor.setQueueName(value); + else if ("ctime".equals(header)) { + jobDescriptor.setCTime(value); + } else if ("qtime".equals(header)) { + jobDescriptor.setQTime(value); + } else if ("mtime".equals(header)) { + jobDescriptor.setMTime(value); + } else if ("start_time".equals(header)) { + jobDescriptor.setSTime(value); + } else if ("comp_time".equals(header)) { + jobDescriptor.setCompTime(value); + } else if ("exec_host".equals(header)) { + jobDescriptor.setExecuteNode(value); + } else if ("Output_Path".equals(header)) { + if (info[i + 1].contains("=") || info[i + 1].contains(":")) + jobDescriptor.setStandardOutFile(value); + else { + jobDescriptor.setStandardOutFile(value + info[i + 1].trim()); + i++; + } + } else if ("Error_Path".equals(header)) { + if (info[i + 1].contains("=") || info[i + 1].contains(":")) + jobDescriptor.setStandardErrorFile(value); + else { + String st = info[i + 1].trim(); + jobDescriptor.setStandardErrorFile(value + st); + i++; + } + + } else if ("submit_args".equals(header)) { + while (i + 1 < info.length) { + if (info[i + 1].startsWith("\t")) { + value += info[i + 1]; + i++; + } else + break; + } + value = value.replaceAll("\t", ""); + jobDescriptor.setSubmitArgs(value); + } + } + } + } + + public String parseJobSubmission(String rawOutput) { + log.debug(rawOutput); + if (rawOutput != null && !rawOutput.isEmpty()) { + String[] info = rawOutput.split("\n"); + String lastLine = info[info.length - 1]; + return lastLine.split(" ")[2]; // In PBS stdout is going to be directly the jobID + } else { + return ""; + } + } + + public JobStatus parseJobStatus(String jobID, String rawOutput) { + // not implemented to sun grid engine + return JobStatus.U; + } + + public void parseJobStatuses(String userName, Map statusMap, String rawOutput) { + log.debug(rawOutput); + String[] info = rawOutput.split("\n"); + int lastStop = 0; + for (String jobID : statusMap.keySet()) { + for(int i=lastStop;i columnList = new ArrayList(); + for (String s : columns) { + if (!"".equals(s)) { + columnList.add(s); + } + } + lastStop = i+1; + if ("E".equals(columnList.get(4))) { + // There is another status with the same letter E other than error status + // to avoid that we make a small tweek to the job status + columnList.set(4, "Er"); + } + statusMap.put(jobID, JobStatus.valueOf(columnList.get(4))); + break; + } + } + } + } + + @Override + public String parseJobId(String jobName, String rawOutput) throws SSHApiException { + return null; // TODO: Implement the parse logic ( with regex if possible ). + } + + +} http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/util/CommonUtils.java ---------------------------------------------------------------------- diff --git a/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/util/CommonUtils.java b/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/util/CommonUtils.java index e5ab739..f1086d7 100644 --- a/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/util/CommonUtils.java +++ b/tools/gsissh/src/main/java/org/apache/airavata/gsi/ssh/util/CommonUtils.java @@ -22,7 +22,6 @@ package org.apache.airavata.gsi.ssh.util; import org.apache.airavata.gsi.ssh.api.job.*; import org.apache.airavata.gsi.ssh.impl.JobStatus; -import org.apache.airavata.gsi.ssh.impl.RawCommandInfo; public class CommonUtils { /** @@ -73,7 +72,7 @@ public class CommonUtils { } public static JobManagerConfiguration getSGEJobManager(String installedPath) { - return new PBSJobConfiguration("SGETemplate.xslt", ".pbs", installedPath, new SGEOutputParser()); + return new UGEJobConfiguration("UGETemplate.xslt", ".pbs", installedPath, new UGEOutputParser()); } public static JobManagerConfiguration getLSFJobManager(String installedPath) { http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/tools/gsissh/src/main/resources/SGETemplate.xslt ---------------------------------------------------------------------- diff --git a/tools/gsissh/src/main/resources/SGETemplate.xslt b/tools/gsissh/src/main/resources/SGETemplate.xslt deleted file mode 100644 index 5b57265..0000000 --- a/tools/gsissh/src/main/resources/SGETemplate.xslt +++ /dev/null @@ -1,74 +0,0 @@ - - - - -#! /bin/bash -# Grid Engine batch job script built by Apache Airavata -# - -#$ -S - -#$ -V - - -#$ -q - - -#$ -m beas - -#$ -A - - - - -#$ -l h_rt= - - - - -#$ -N - - - - -#$ -o - - - - -#$ -e - - - - -#$ -pe way - - - - -= -export - - - - - -cd - - - - - - - - - - - - \ No newline at end of file http://git-wip-us.apache.org/repos/asf/airavata/blob/86a79b66/tools/gsissh/src/main/resources/UGETemplate.xslt ---------------------------------------------------------------------- diff --git a/tools/gsissh/src/main/resources/UGETemplate.xslt b/tools/gsissh/src/main/resources/UGETemplate.xslt new file mode 100644 index 0000000..5b57265 --- /dev/null +++ b/tools/gsissh/src/main/resources/UGETemplate.xslt @@ -0,0 +1,74 @@ + + + + +#! /bin/bash +# Grid Engine batch job script built by Apache Airavata +# + +#$ -S + +#$ -V + + +#$ -q + + +#$ -m beas + +#$ -A + + + + +#$ -l h_rt= + + + + +#$ -N + + + + +#$ -o + + + + +#$ -e + + + + +#$ -pe way + + + + += +export + + + + + +cd + + + + + + + + + + + + \ No newline at end of file