Return-Path: X-Original-To: apmail-ignite-commits-archive@minotaur.apache.org Delivered-To: apmail-ignite-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CCFA718082 for ; Thu, 16 Jul 2015 07:15:39 +0000 (UTC) Received: (qmail 76690 invoked by uid 500); 16 Jul 2015 07:15:36 -0000 Delivered-To: apmail-ignite-commits-archive@ignite.apache.org Received: (qmail 76660 invoked by uid 500); 16 Jul 2015 07:15:36 -0000 Mailing-List: contact commits-help@ignite.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.incubator.apache.org Delivered-To: mailing list commits@ignite.incubator.apache.org Received: (qmail 76651 invoked by uid 99); 16 Jul 2015 07:15:36 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2015 07:15:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1328DD5003 for ; Thu, 16 Jul 2015 07:15:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.375 X-Spam-Level: X-Spam-Status: No, score=0.375 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-1.426, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id JCHPqrJpsBAU for ; Thu, 16 Jul 2015 07:15:28 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with SMTP id 4AB6C20F19 for ; Thu, 16 Jul 2015 07:15:28 +0000 (UTC) Received: (qmail 76287 invoked by uid 99); 16 Jul 2015 07:15:28 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2015 07:15:28 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 9193DE36EA; Thu, 16 Jul 2015 07:15:27 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: ntikhonov@apache.org To: commits@ignite.incubator.apache.org Date: Thu, 16 Jul 2015 07:15:27 -0000 Message-Id: <3b87bff0e5f7414183538d56bccea2ff@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [01/50] incubator-ignite git commit: #YARN Added simple app Repository: incubator-ignite Updated Branches: refs/heads/master 77da7288f -> 13e55b246 #YARN Added simple app Project: http://git-wip-us.apache.org/repos/asf/incubator-ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-ignite/commit/50cfa27c Tree: http://git-wip-us.apache.org/repos/asf/incubator-ignite/tree/50cfa27c Diff: http://git-wip-us.apache.org/repos/asf/incubator-ignite/diff/50cfa27c Branch: refs/heads/master Commit: 50cfa27c273e3f82687c54738b52fc23a5e7bb33 Parents: c232631 Author: nikolay_tikhonov Authored: Mon Jun 1 17:31:07 2015 +0300 Committer: nikolay_tikhonov Committed: Mon Jun 1 17:31:07 2015 +0300 ---------------------------------------------------------------------- modules/yarn/README.txt | 28 + modules/yarn/licenses/apache-2.0.txt | 202 ++++++++ modules/yarn/pom.xml | 94 ++++ .../apache/ignite/yarn/ApplicationMaster.java | 133 +++++ .../apache/ignite/yarn/ClusterProperties.java | 519 +++++++++++++++++++ .../java/org/apache/ignite/yarn/IgniteTask.java | 86 +++ .../apache/ignite/yarn/IgniteYarnClient.java | 50 ++ .../org/apache/ignite/yarn/package-info.java | 22 + .../org/apache/ignite/IgniteMesosTestSuite.java | 38 ++ .../ignite/yarn/IgniteSchedulerSelfTest.java | 27 + pom.xml | 1 + 11 files changed, 1200 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/README.txt ---------------------------------------------------------------------- diff --git a/modules/yarn/README.txt b/modules/yarn/README.txt new file mode 100644 index 0000000..75a62f8 --- /dev/null +++ b/modules/yarn/README.txt @@ -0,0 +1,28 @@ +Apache Ignite Mesos Module +------------------------ + +Apache Ignite Mesos module provides integration Apache Ignite with Apache Mesos. + +Importing Apache Ignite Mesos Module In Maven Project +------------------------------------- + +If you are using Maven to manage dependencies of your project, you can add Cloud module +dependency like this (replace '${ignite.version}' with actual Ignite version you are +interested in): + + + ... + + ... + + org.apache.ignite + ignite-mesos + ${ignite.version} + + ... + + ... + http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/licenses/apache-2.0.txt ---------------------------------------------------------------------- diff --git a/modules/yarn/licenses/apache-2.0.txt b/modules/yarn/licenses/apache-2.0.txt new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/modules/yarn/licenses/apache-2.0.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/pom.xml ---------------------------------------------------------------------- diff --git a/modules/yarn/pom.xml b/modules/yarn/pom.xml new file mode 100644 index 0000000..f51b0e8 --- /dev/null +++ b/modules/yarn/pom.xml @@ -0,0 +1,94 @@ + + + + + + + 4.0.0 + + org.apache.ignite + ignite-yarn + 1.2.0-SNAPSHOT + + + 2.7.0 + + + + + org.apache.hadoop + hadoop-yarn-client + ${hadoop.version} + + + + org.apache.hadoop + hadoop-common + ${hadoop.version} + + + + junit + junit + 4.11 + test + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.1 + + 1.7 + 1.7 + + + + + maven-assembly-plugin + 2.4.1 + + + jar-with-dependencies + + + + org.apache.ignite.yarn.IgniteYarnClient + + + + + + make-assembly + package + + single + + + + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/src/main/java/org/apache/ignite/yarn/ApplicationMaster.java ---------------------------------------------------------------------- diff --git a/modules/yarn/src/main/java/org/apache/ignite/yarn/ApplicationMaster.java b/modules/yarn/src/main/java/org/apache/ignite/yarn/ApplicationMaster.java new file mode 100644 index 0000000..f52a1de --- /dev/null +++ b/modules/yarn/src/main/java/org/apache/ignite/yarn/ApplicationMaster.java @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.yarn; + +import org.apache.hadoop.conf.*; +import org.apache.hadoop.yarn.api.*; +import org.apache.hadoop.yarn.api.protocolrecords.*; +import org.apache.hadoop.yarn.api.records.*; +import org.apache.hadoop.yarn.client.api.*; +import org.apache.hadoop.yarn.client.api.async.*; +import org.apache.hadoop.yarn.conf.*; +import org.apache.hadoop.yarn.util.*; + +import java.util.*; + +/** + * TODO + */ +public class ApplicationMaster implements AMRMClientAsync.CallbackHandler { + /** {@inheritDoc} */ + @Override public void onContainersCompleted(List statuses) { + + } + + /** {@inheritDoc} */ + @Override public void onContainersAllocated(List containers) { + + } + + /** {@inheritDoc} */ + @Override public void onShutdownRequest() { + + } + + /** {@inheritDoc} */ + @Override public void onNodesUpdated(List updatedNodes) { + + } + + /** {@inheritDoc} */ + @Override public float getProgress() { + return 0; + } + + /** {@inheritDoc} */ + @Override public void onError(Throwable e) { + + } + + /** + * @param args Arguments. + */ + public static void main(String[] args) throws Exception { + final String command = args[0]; + final int n = Integer.valueOf(args[1]); + + // Initialize clients to ResourceManager and NodeManagers + Configuration conf = new YarnConfiguration(); + + AMRMClient rmClient = AMRMClient.createAMRMClient(); + rmClient.init(conf); + rmClient.start(); + + NMClient nmClient = NMClient.createNMClient(); + nmClient.init(conf); + nmClient.start(); + + // Register with ResourceManager + rmClient.registerApplicationMaster("", 0, ""); + + // Priority for worker containers - priorities are intra-application + Priority priority = Records.newRecord(Priority.class); + priority.setPriority(0); + + // Resource requirements for worker containers + Resource capability = Records.newRecord(Resource.class); + capability.setMemory(128); + capability.setVirtualCores(1); + + // Make container requests to ResourceManager + for (int i = 0; i < n; ++i) { + AMRMClient.ContainerRequest containerAsk = + new AMRMClient.ContainerRequest(capability, null, null, priority); + + rmClient.addContainerRequest(containerAsk); + } + + // Obtain allocated containers, launch and check for responses + int responseId = 0; + int completedContainers = 0; + while (completedContainers < n) { + AllocateResponse response = rmClient.allocate(responseId++); + for (Container container : response.getAllocatedContainers()) { + // Launch container by create ContainerLaunchContext + ContainerLaunchContext ctx = + Records.newRecord(ContainerLaunchContext.class); + + ctx.setCommands( + Collections.singletonList( + command + + " 1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" + + " 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr" + )); + + nmClient.startContainer(container, ctx); + } + for (ContainerStatus status : response.getCompletedContainersStatuses()) { + ++completedContainers; + System.out.println("Completed container " + status.getContainerId()); + } + Thread.sleep(100); + } + + // Un-register with ResourceManager + rmClient.unregisterApplicationMaster( + FinalApplicationStatus.SUCCEEDED, "", ""); + } +} http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/src/main/java/org/apache/ignite/yarn/ClusterProperties.java ---------------------------------------------------------------------- diff --git a/modules/yarn/src/main/java/org/apache/ignite/yarn/ClusterProperties.java b/modules/yarn/src/main/java/org/apache/ignite/yarn/ClusterProperties.java new file mode 100644 index 0000000..0c6c26d --- /dev/null +++ b/modules/yarn/src/main/java/org/apache/ignite/yarn/ClusterProperties.java @@ -0,0 +1,519 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.yarn; + +import java.io.*; +import java.net.*; +import java.util.*; +import java.util.logging.*; +import java.util.regex.*; + +/** + * Cluster settings. + */ +public class ClusterProperties { + /** */ + private static final Logger log = Logger.getLogger(ClusterProperties.class.getSimpleName()); + + /** Unlimited. */ + public static final double UNLIMITED = Double.MAX_VALUE; + + /** */ + public static final String MESOS_MASTER_URL = "MESOS_MASTER_URL"; + + /** */ + public static final String DEFAULT_MESOS_MASTER_URL = "zk://localhost:2181/mesos"; + + /** Mesos master url. */ + private String mesosUrl = DEFAULT_MESOS_MASTER_URL; + + /** */ + public static final String IGNITE_CLUSTER_NAME = "IGNITE_CLUSTER_NAME"; + + /** */ + public static final String DEFAULT_CLUSTER_NAME = "ignite-cluster"; + + /** Mesos master url. */ + private String clusterName = DEFAULT_CLUSTER_NAME; + + /** */ + public static final String IGNITE_HTTP_SERVER_HOST = "IGNITE_HTTP_SERVER_HOST"; + + /** Http server host. */ + private String httpServerHost = null; + + /** */ + public static final String IGNITE_HTTP_SERVER_PORT = "IGNITE_HTTP_SERVER_PORT"; + + /** */ + public static final String DEFAULT_HTTP_SERVER_PORT = "48610"; + + /** Http server host. */ + private int httpServerPort = Integer.valueOf(DEFAULT_HTTP_SERVER_PORT); + + /** */ + public static final String IGNITE_TOTAL_CPU = "IGNITE_TOTAL_CPU"; + + /** CPU limit. */ + private double cpu = UNLIMITED; + + /** */ + public static final String IGNITE_RUN_CPU_PER_NODE = "IGNITE_RUN_CPU_PER_NODE"; + + /** CPU limit. */ + private double cpuPerNode = UNLIMITED; + + /** */ + public static final String IGNITE_TOTAL_MEMORY = "IGNITE_TOTAL_MEMORY"; + + /** Memory limit. */ + private double mem = UNLIMITED; + + /** */ + public static final String IGNITE_MEMORY_PER_NODE = "IGNITE_MEMORY_PER_NODE"; + + /** Memory limit. */ + private double memPerNode = UNLIMITED; + + /** */ + public static final String IGNITE_TOTAL_DISK_SPACE = "IGNITE_TOTAL_DISK_SPACE"; + + /** Disk space limit. */ + private double disk = UNLIMITED; + + /** */ + public static final String IGNITE_DISK_SPACE_PER_NODE = "IGNITE_DISK_SPACE_PER_NODE"; + + /** Disk space limit. */ + private double diskPerNode = UNLIMITED; + + /** */ + public static final String IGNITE_NODE_COUNT = "IGNITE_NODE_COUNT"; + + /** Node count limit. */ + private double nodeCnt = UNLIMITED; + + /** */ + public static final String IGNITE_MIN_CPU_PER_NODE = "IGNITE_MIN_CPU_PER_NODE"; + + /** */ + public static final double DEFAULT_RESOURCE_MIN_CPU = 1; + + /** Min memory per node. */ + private double minCpu = DEFAULT_RESOURCE_MIN_CPU; + + /** */ + public static final String IGNITE_MIN_MEMORY_PER_NODE = "IGNITE_MIN_MEMORY_PER_NODE"; + + /** */ + public static final double DEFAULT_RESOURCE_MIN_MEM = 256; + + /** Min memory per node. */ + private double minMemory = DEFAULT_RESOURCE_MIN_MEM; + + /** */ + public static final String IGNITE_VERSION = "IGNITE_VERSION"; + + /** */ + public static final String DEFAULT_IGNITE_VERSION = "latest"; + + /** Ignite version. */ + private String igniteVer = DEFAULT_IGNITE_VERSION; + + /** */ + public static final String IGNITE_PACKAGE_URL = "IGNITE_PACKAGE_URL"; + + /** Ignite package url. */ + private String ignitePackageUrl = null; + + /** */ + public static final String IGNITE_WORK_DIR = "IGNITE_WORK_DIR"; + + /** */ + public static final String DEFAULT_IGNITE_WORK_DIR = "ignite-releases/"; + + /** Ignite version. */ + private String igniteWorkDir = DEFAULT_IGNITE_WORK_DIR; + + /** */ + public static final String IGNITE_USERS_LIBS = "IGNITE_USERS_LIBS"; + + /** Path to users libs. */ + private String userLibs = null; + + /** */ + public static final String IGNITE_USERS_LIBS_URL = "IGNITE_USERS_LIBS_URL"; + + /** URL to users libs. */ + private String userLibsUrl = null; + + /** */ + public static final String IGNITE_CONFIG_XML = "IGNITE_XML_CONFIG"; + + /** Ignite config. */ + private String igniteCfg = null; + + /** */ + public static final String IGNITE_CONFIG_XML_URL = "IGNITE_CONFIG_XML_URL"; + + /** Url to ignite config. */ + private String igniteCfgUrl = null; + + /** */ + public static final String IGNITE_HOSTNAME_CONSTRAINT = "IGNITE_HOSTNAME_CONSTRAINT"; + + /** Url to ignite config. */ + private Pattern hostnameConstraint = null; + + /** */ + public ClusterProperties() { + // No-op. + } + + /** + * @return Cluster name. + */ + public String clusterName() { + return clusterName; + } + + /** + * @return CPU count limit. + */ + public double cpus() { + return cpu; + } + + /** + * Sets CPU count limit. + */ + public void cpus(double cpu) { + this.cpu = cpu; + } + + /** + * @return CPU count limit. + */ + public double cpusPerNode() { + return cpuPerNode; + } + + /** + * Sets CPU count limit. + */ + public void cpusPerNode(double cpu) { + this.cpuPerNode = cpu; + } + + /** + * @return mem limit. + */ + public double memory() { + return mem; + } + + /** + * Sets mem limit. + * + * @param mem Memory. + */ + public void memory(double mem) { + this.mem = mem; + } + + /** + * @return mem limit. + */ + public double memoryPerNode() { + return memPerNode; + } + + /** + * Sets mem limit. + * + * @param mem Memory. + */ + public void memoryPerNode(double mem) { + this.memPerNode = mem; + } + + /** + * @return disk limit. + */ + public double disk() { + return disk; + } + + /** + * @return disk limit per node. + */ + public double diskPerNode() { + return diskPerNode; + } + + /** + * @return instance count limit. + */ + public double instances() { + return nodeCnt; + } + + /** + * @return min memory per node. + */ + public double minMemoryPerNode() { + return minMemory; + } + + /** + * Sets min memory. + * + * @param minMemory Min memory. + */ + public void minMemoryPerNode(double minMemory) { + this.minMemory = minMemory; + } + + /** + * Sets hostname constraint. + * + * @param pattern Hostname pattern. + */ + public void hostnameConstraint(Pattern pattern) { + this.hostnameConstraint = pattern; + } + + /** + * @return min cpu count per node. + */ + public double minCpuPerNode() { + return minCpu; + } + + /** + * Sets min cpu count per node. + * + * @param minCpu min cpu count per node. + */ + public void minCpuPerNode(double minCpu) { + this.minCpu = minCpu; + } + + /** + * @return Ignite version. + */ + public String igniteVer() { + return igniteVer; + } + + /** + * @return Working directory. + */ + public String igniteWorkDir() { + return igniteWorkDir; + } + + /** + * @return User's libs. + */ + public String userLibs() { + return userLibs; + } + + /** + * @return Ignite configuration. + */ + public String igniteCfg() { + return igniteCfg; + } + + /** + * @return Master url. + */ + public String masterUrl() { + return mesosUrl; + } + + /** + * @return Http server host. + */ + public String httpServerHost() { + return httpServerHost; + } + + /** + * @return Http server port. + */ + public int httpServerPort() { + return httpServerPort; + } + + /** + * @return Url to ignite package. + */ + public String ignitePackageUrl() { + return ignitePackageUrl; + } + + /** + * @return Url to ignite configuration. + */ + public String igniteConfigUrl() { + return igniteCfgUrl; + } + + /** + * @return Url to users libs configuration. + */ + public String usersLibsUrl() { + return userLibsUrl; + } + + /** + * @return Host name constraint. + */ + public Pattern hostnameConstraint() { + return hostnameConstraint; + } + + /** + * @param config path to config file. + * @return Cluster configuration. + */ + public static ClusterProperties from(String config) { + try { + Properties props = null; + + if (config != null) { + props = new Properties(); + + props.load(new FileInputStream(config)); + } + + ClusterProperties prop = new ClusterProperties(); + + prop.mesosUrl = getStringProperty(MESOS_MASTER_URL, props, DEFAULT_MESOS_MASTER_URL); + + prop.httpServerHost = getStringProperty(IGNITE_HTTP_SERVER_HOST, props, getNonLoopbackAddress()); + + String port = System.getProperty("PORT0"); + + if (port != null && !port.isEmpty()) + prop.httpServerPort = Integer.valueOf(port); + else + prop.httpServerPort = Integer.valueOf(getStringProperty(IGNITE_HTTP_SERVER_PORT, props, + DEFAULT_HTTP_SERVER_PORT)); + + prop.clusterName = getStringProperty(IGNITE_CLUSTER_NAME, props, DEFAULT_CLUSTER_NAME); + + prop.userLibsUrl = getStringProperty(IGNITE_USERS_LIBS_URL, props, null); + prop.ignitePackageUrl = getStringProperty(IGNITE_PACKAGE_URL, props, null); + prop.igniteCfgUrl = getStringProperty(IGNITE_CONFIG_XML_URL, props, null); + + prop.cpu = getDoubleProperty(IGNITE_TOTAL_CPU, props, UNLIMITED); + prop.cpuPerNode = getDoubleProperty(IGNITE_RUN_CPU_PER_NODE, props, UNLIMITED); + prop.mem = getDoubleProperty(IGNITE_TOTAL_MEMORY, props, UNLIMITED); + prop.memPerNode = getDoubleProperty(IGNITE_MEMORY_PER_NODE, props, UNLIMITED); + prop.disk = getDoubleProperty(IGNITE_TOTAL_DISK_SPACE, props, UNLIMITED); + prop.diskPerNode = getDoubleProperty(IGNITE_DISK_SPACE_PER_NODE, props, 1024.0); + prop.nodeCnt = getDoubleProperty(IGNITE_NODE_COUNT, props, UNLIMITED); + prop.minCpu = getDoubleProperty(IGNITE_MIN_CPU_PER_NODE, props, DEFAULT_RESOURCE_MIN_CPU); + prop.minMemory = getDoubleProperty(IGNITE_MIN_MEMORY_PER_NODE, props, DEFAULT_RESOURCE_MIN_MEM); + + prop.igniteVer = getStringProperty(IGNITE_VERSION, props, DEFAULT_IGNITE_VERSION); + prop.igniteWorkDir = getStringProperty(IGNITE_WORK_DIR, props, DEFAULT_IGNITE_WORK_DIR); + prop.igniteCfg = getStringProperty(IGNITE_CONFIG_XML, props, null); + prop.userLibs = getStringProperty(IGNITE_USERS_LIBS, props, null); + + String pattern = getStringProperty(IGNITE_HOSTNAME_CONSTRAINT, props, null); + + if (pattern != null) { + try { + prop.hostnameConstraint = Pattern.compile(pattern); + } + catch (PatternSyntaxException e) { + log.log(Level.WARNING, "IGNITE_HOSTNAME_CONSTRAINT has invalid pattern. It will be ignore.", e); + } + } + + return prop; + } + catch (IOException e) { + throw new RuntimeException(e); + } + } + + /** + * @param name Property name. + * @param fileProps Property file. + * @return Property value. + */ + private static double getDoubleProperty(String name, Properties fileProps, Double defaultVal) { + if (fileProps != null && fileProps.containsKey(name)) + return Double.valueOf(fileProps.getProperty(name)); + + String property = System.getProperty(name); + + if (property == null) + property = System.getenv(name); + + return property == null ? defaultVal : Double.valueOf(property); + } + + /** + * @param name Property name. + * @param fileProps Property file. + * @return Property value. + */ + private static String getStringProperty(String name, Properties fileProps, String defaultVal) { + if (fileProps != null && fileProps.containsKey(name)) + return fileProps.getProperty(name); + + String property = System.getProperty(name); + + if (property == null) + property = System.getenv(name); + + return property == null ? defaultVal : property; + } + + /** + * Finds a local, non-loopback, IPv4 address + * + * @return The first non-loopback IPv4 address found, or null if no such addresses found + * @throws SocketException If there was a problem querying the network interfaces + */ + public static String getNonLoopbackAddress() throws SocketException { + Enumeration ifaces = NetworkInterface.getNetworkInterfaces(); + + while (ifaces.hasMoreElements()) { + NetworkInterface iface = ifaces.nextElement(); + + Enumeration addresses = iface.getInetAddresses(); + + while (addresses.hasMoreElements()) { + InetAddress addr = addresses.nextElement(); + + if (addr instanceof Inet4Address && !addr.isLoopbackAddress()) + return addr.getHostAddress(); + } + } + + throw new RuntimeException("Failed. Couldn't find non-loopback address"); + } +} http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteTask.java ---------------------------------------------------------------------- diff --git a/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteTask.java b/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteTask.java new file mode 100644 index 0000000..60275fd --- /dev/null +++ b/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteTask.java @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.yarn; + +/** + * Information about launched task. + */ +public class IgniteTask { + /** */ + public final String host; + + /** */ + public final double cpuCores; + + /** */ + public final double mem; + + /** */ + public final double disk; + + /** + * Ignite launched task. + * + * @param host Host. + * @param cpuCores Cpu cores count. + * @param mem Memory. + * @param disk Disk. + */ + public IgniteTask(String host, double cpuCores, double mem, double disk) { + this.host = host; + this.cpuCores = cpuCores; + this.mem = mem; + this.disk = disk; + } + + /** + * @return Host. + */ + public String host() { + return host; + } + + /** + * @return Cores count. + */ + public double cpuCores() { + return cpuCores; + } + + /** + * @return Memory. + */ + public double mem() { + return mem; + } + + /** + * @return Disk. + */ + public double disk() { + return disk; + } + + @Override + public String toString() { + return "IgniteTask " + + "host: [" + host + ']' + + ", cpuCores: [" + cpuCores + "]" + + ", mem: [" + mem + "]"; + } +} http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteYarnClient.java ---------------------------------------------------------------------- diff --git a/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteYarnClient.java b/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteYarnClient.java new file mode 100644 index 0000000..7cef50d --- /dev/null +++ b/modules/yarn/src/main/java/org/apache/ignite/yarn/IgniteYarnClient.java @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.yarn; + +import org.apache.hadoop.yarn.client.api.*; +import org.apache.hadoop.yarn.conf.*; + +import java.util.logging.*; + +/** + * Ignite yarn client. + */ +public class IgniteYarnClient { + /** */ + public static final Logger log = Logger.getLogger(IgniteYarnClient.class.getSimpleName()); + + /** + * Main methods has only one optional parameter - path to properties files. + * + * @param args Args. + */ + public static void main(String[] args) throws Exception { + ClusterProperties clusterProps = ClusterProperties.from(args.length >= 1 ? args[0] : null); + + // Create yarnClient + YarnConfiguration conf = new YarnConfiguration(); + + YarnClient yarnClient = YarnClient.createYarnClient(); + + yarnClient.init(conf); + yarnClient.start(); + + YarnClientApplication app = yarnClient.createApplication(); + } +} http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/src/main/java/org/apache/ignite/yarn/package-info.java ---------------------------------------------------------------------- diff --git a/modules/yarn/src/main/java/org/apache/ignite/yarn/package-info.java b/modules/yarn/src/main/java/org/apache/ignite/yarn/package-info.java new file mode 100644 index 0000000..c47f1e8 --- /dev/null +++ b/modules/yarn/src/main/java/org/apache/ignite/yarn/package-info.java @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * + * Contains classes to support integration with Apache Mesos. + */ +package org.apache.ignite.yarn; \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/src/test/java/org/apache/ignite/IgniteMesosTestSuite.java ---------------------------------------------------------------------- diff --git a/modules/yarn/src/test/java/org/apache/ignite/IgniteMesosTestSuite.java b/modules/yarn/src/test/java/org/apache/ignite/IgniteMesosTestSuite.java new file mode 100644 index 0000000..e6920b3 --- /dev/null +++ b/modules/yarn/src/test/java/org/apache/ignite/IgniteMesosTestSuite.java @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite; + +import junit.framework.*; +import org.apache.ignite.yarn.*; + +/** + * Apache Mesos integration tests. + */ +public class IgniteMesosTestSuite extends TestSuite { + /** + * @return Test suite. + * @throws Exception Thrown in case of the failure. + */ + public static TestSuite suite() throws Exception { + TestSuite suite = new TestSuite("Apache Mesos Integration Test Suite"); + + suite.addTest(new TestSuite(IgniteSchedulerSelfTest.class)); + + return suite; + } +} http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/modules/yarn/src/test/java/org/apache/ignite/yarn/IgniteSchedulerSelfTest.java ---------------------------------------------------------------------- diff --git a/modules/yarn/src/test/java/org/apache/ignite/yarn/IgniteSchedulerSelfTest.java b/modules/yarn/src/test/java/org/apache/ignite/yarn/IgniteSchedulerSelfTest.java new file mode 100644 index 0000000..1a03743 --- /dev/null +++ b/modules/yarn/src/test/java/org/apache/ignite/yarn/IgniteSchedulerSelfTest.java @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.yarn; + +import junit.framework.*; + +/** + * Scheduler tests. + */ +public class IgniteSchedulerSelfTest extends TestCase { + +} http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/50cfa27c/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 6f8524f..9cea078 100644 --- a/pom.xml +++ b/pom.xml @@ -70,6 +70,7 @@ modules/gce modules/cloud modules/mesos + modules/yarn