Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 47EE018FB5 for ; Thu, 18 Feb 2016 19:17:09 +0000 (UTC) Received: (qmail 38807 invoked by uid 500); 18 Feb 2016 19:17:08 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 38758 invoked by uid 500); 18 Feb 2016 19:17:08 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 38748 invoked by uid 99); 18 Feb 2016 19:17:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2016 19:17:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CBF051805D4 for ; Thu, 18 Feb 2016 19:17:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.451 X-Spam-Level: X-Spam-Status: No, score=-3.451 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.329, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=jpmchase.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id XPwe_d1Stmxz for ; Thu, 18 Feb 2016 19:17:03 +0000 (UTC) Received: from sj2.jpmchase.com (sj2.jpmchase.com [159.53.110.174]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 72DF35FAEB for ; Thu, 18 Feb 2016 19:17:02 +0000 (UTC) Received: from si4.svr.bankone.net (si4.svr.bankone.net [155.180.56.116]) by sj2.jpmchase.com (Switch-3.4.4/Switch-3.3.3mp) with ESMTP id u1IJGtSI001466 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Thu, 18 Feb 2016 19:16:55 GMT X-DKIM: OpenDKIM Filter v2.1.3 sj2.jpmchase.com u1IJGtSI001466 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jpmchase.com; s=d2048-1; t=1455823015; bh=4epfenujvbweDLp6ci4QAqXpa3E6i0EE5Z8Ccj9yrMs=; h=From:To:Subject:Date:Message-ID:Content-Type:MIME-Version; b=tq+T98BqIyic3ZsABfA6uu2VEY64mszM5la342r0603jY5+QMdjMHbax4C8YDfP2S 5CCP9lBUcjUWzUQia0xZiokcFbbW2XBnZs4GEjiirpyuIHnG3ZPA59aJs+HwrGm8Ro Revre6bgZE0oDqcQIQczrJ/Qc0OLMYEsQ9DrUsrXJGzUnRxMPR2mYcRfbk2QDDf++3 xgGNvatWI6sU7PUltYvAz4A6mqpgd5le4sUjqTkMCNFXUMYFxnTDPxInN0dWHXsyRa /BCC0+pyFUdXWmGkkcVWkch0cjhbij31xLOFYQCKT/9aesoKLoCjIUYLHHcag4pYVp Ue/oRjBMzGjIA== Received: from smgj3.svr.us.jpmchase.net (smgj3.svr.us.jpmchase.net [169.89.220.172]) by si4.svr.bankone.net (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id u1IJGdpY018785 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Thu, 18 Feb 2016 19:16:53 GMT X-AuditID: a959dcac-f79d76d0000060ca-ad-56c618a365df Received: from SCAHHB004.exchad.jpmchase.net ( [169.69.59.54]) by smgj3.svr.us.jpmchase.net (Symantec Messaging Gateway) with SMTP id 34.36.24778.3A816C65; Thu, 18 Feb 2016 19:16:52 +0000 (GMT) Received: from SBECDI004.exchad.jpmchase.net (169.83.4.240) by SCAHHB004.exchad.jpmchase.net (169.83.186.41) with Microsoft SMTP Server (TLS) id 14.3.210.3; Thu, 18 Feb 2016 14:16:50 -0500 Received: from SBECMX004.exchad.jpmchase.net ([169.254.8.190]) by SBECDI004.exchad.jpmchase.net ([169.83.4.240]) with mapi id 14.03.0210.003; Thu, 18 Feb 2016 14:16:50 -0500 From: "Kazansky, Michael" To: "user@storm.apache.org" Subject: submit topology using NimbusClient Thread-Topic: submit topology using NimbusClient Thread-Index: AdFqgOdjS+3W4Kg5Sb+eFOkjxr4ATA== Date: Thu, 18 Feb 2016 19:16:49 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.196.144.186] x-jpmc-sendertrust: JPMC Content-Type: multipart/alternative; boundary="_000_A2AA10324A422F409F7DF2D8187343BBF2CE3732SBECMX004exchad_" MIME-Version: 1.0 X-DLP-FWD: Yes X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprMJsWRmVeSWpSXmKPExsWy0tXaTHeJxLEwg5PPuSy+vHzG6sDosXDR EvYAxqgGRpvEvLz8ksSSVIWU1OJkWyWXzOLknMTM3NQiBV0Fr4Lc5IzE4lS95PxcJYXMFFsl YyWFgpzE5NTc1LwSW6XEgoLUvBQlOy4FDGADVJaZp5Cal5yfkpmXbqvkGeyva2FhaqlrqGQX kpFZrAA0M7c0L+GeQsbtq03sBd1uFbvubGNpYNxr28XIySEhYCLxduk6dghbTOLCvfVsXYxc HEIChxkl9s5bxAySEBI4wChx+3sARGIjo8TVbxvZQBJsApYSi//cASsSETCW+N5+nAXEFhbQ lvgy+TrQVA6guIHE651GEKaexJ4/NiAVLAKqElN+LWUCsXkFQiRerHoBNoUR6Ibvp9aAxZkF xCVuPZnPBHGbgMSSPeeZIWxRiZeP/7FC2EoSLz4/gIpLSNx4PJsNojdf4tWlBSwQ8wUlTs58 wgLxipbE5aU3GEHOkRDgk9jeVj2BUWwWkm2zkHTPQtINEdeRWLD7ExuErS2xbOFrZhj7zIHH TMjiCxjZVzFKFuemZxnrFZcV6ZUW62XB4jkvtWQTIyjNRN5Zs4Pxbp/tIUYBDkYlHl6O10fC hFgTy4orcw8xSnAwK4nwVn0+GibEm5JYWZValB9fVJqTWnyIcSIjMAQnMkuJJucDk2BeSbyh sZmFoYGxkaWBuZERLYWVxHmTuPaFCQmkAxNzdmpqQWoRzFFMHJxSDYwmfklTzu+ItvMNerd2 fo3SSu6WkJ16d6Z/rzrQcSBxq+K/A1MyQkxum8Uumd0/e/bpKKcsnsatVj9ym78z2KdvX9i9 4udO71zznXtuPu2cdvK8foIS1+pu7i+7P19f6/7FNyxle4vyas0zJhsY/rkaa7/btUr3Xb4s +4Q2wxW+fK9eLeEwq7dXYinOSDTUYi4qTgQAtMaC4aYDAAA= --_000_A2AA10324A422F409F7DF2D8187343BBF2CE3732SBECMX004exchad_ Content-Type: text/plain; charset="us-ascii" content-transfer-encoding: quoted-printable Hi Storm gurus, I am working on the project which should allow users to submit their topolog= y jars to Storm. We successfully using command line but now I need to write= application which allows user to submit jar through let say browser. Here is what I am trying to do Config conf =3D new Config(); TopologyBuilder topoBuilder =3D new TopologyBuilder(); conf.put(Config.NIMBUS_HOST, topologyModel.getHost()); conf.setDebug(true); Map stormConf =3D Utils.readStormConfig(); stormConf.put(Config.NIMBUS_HOST, topologyModel.getHost()); // topology submit NimbusClient nimbus =3D new NimbusClient(stormConf, topologyMode= l.getHost(), null !=3D topologyModel.getNimbusPort() ? topologyModel.getNimb= usPort() : 6627); String uploadedJarLocation =3D StormSubmitter.submitJar(stormCon= f, topologyModel.getArtifactUrl()); System.setProperty("storm.jar", uploadedJarLocation.concat(" ").= concat(topologyModel.getMainClass())); String jsonConf =3D JSONValue.toJSONString(stormConf); //StormTopology topology =3D buildToplogy(topologyModel); //SubmitOptions options =3D new SubmitOptions(); //client.jartransformer.class nimbus.getClient().submitTopology(topologyModel.getTopologyName(= ), uploadedJarLocation, jsonConf, topoBuilder.createTopology()); success =3D true; I looked at the storm python script and saw that main class name is concaten= ated to JVM_OPTS That's why I added this line System.setProperty("storm.jar", uploadedJarLocation.concat(" ").concat(topol= ogyModel.getMainClass())); But when I execute the code in test I only see empty topology without spouts= and bolts in Storm UI with only one worker and one executor. Main class in= the jar has the code to submit spouts and bolts with parallelism 3 so most= definitely main class wasn't picked up. Is there any way to implement this command python ${storm_home}/bin/storm ja= r ${dsl_apps_dir}/dist/${jarname} $mainclass in Java besides calling it (whi= ch is already done) directly? Thanks, Michael Kazansky This communication is for informational purposes only. It is not intended a= s an offer or solicitation for the purchase or sale of any financial instrum= ent or as an official confirmation of any transaction. All market prices, d= ata and other information are not warranted as to completeness or accuracy a= nd are subject to change without notice. Any comments or statements made he= rein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiar= ies and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, c= onfidential and/or exempt from disclosure under applicable law. If you are= not the intended recipient, you are hereby notified that any disclosure, co= pying, distribution, or use of the information contained herein (including a= ny reliance thereon) is STRICTLY PROHIBITED. If you received this transmiss= ion in error, please immediately contact the sender and destroy the material= in its entirety, whether in electronic or hard copy format. Although this= transmission and any attachments are believed to be free of any virus or ot= her defect that might affect any computer system into which it is received a= nd opened, it is the responsibility of the recipient to ensure that it is vi= rus free and no responsibility is accepted by JPMC for any loss or damage ar= ising in any way from its use. Please note that any electronic communicatio= n that is conducted within or through JPMC's systems is subject to intercept= ion, monitoring, review, retention and external production in accordance wit= h JPMC's policy and local laws, rules and regulations; may be stored or othe= rwise processed in countries other than the country in which you are located= ; and will be treated in accordance with JPMC policies and applicable laws a= nd regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures re= lating to European legal entities. --_000_A2AA10324A422F409F7DF2D8187343BBF2CE3732SBECMX004exchad_ Content-Type: text/html; charset="us-ascii" content-transfer-encoding: quoted-printable

Hi Storm gurus,

I am working on the project which should allow users= to submit their topology jars to Storm. We successfully using command line= but now I need to write application which allows user to submit jar through= let say browser.

Here is what I am trying to do

 

         = ;  Config conf =3D new Config();

         = ;   TopologyBuilder topoBuilder =3D new TopologyBuilder();

         = ;   conf.put(Config.NIMBUS_HOST, topologyModel.getHost());

         = ;   conf.setDebug(true);

         = ;   Map stormConf =3D Utils.readStormConfig();

         = ;   stormConf.put(Config.NIMBUS_HOST, topologyModel.getHost());

         = ;  

         = ;   // topology submit

         = ;   NimbusClient nimbus =3D new NimbusClient(stormConf, topologyMo= del.getHost(), null !=3D topologyModel.getNimbusPort() ? topologyModel.getNi= mbusPort() : 6627);

         = ;   String uploadedJarLocation =3D StormSubmitter.submitJar(stormC= onf, topologyModel.getArtifactUrl());

         = ;   System.setProperty("storm.jar", uploadedJarLocation.= concat(" ").concat(topologyModel.getMainClass()));

         = ;   String jsonConf =3D JSONValue.toJSONString(stormConf);

         = ;   //StormTopology topology =3D buildToplogy(topologyModel);=

         = ;   //SubmitOptions options =3D new SubmitOptions();

         = ;   //client.jartransformer.class

         = ;   nimbus.getClient().submitTopology(topologyModel.getTopologyNam= e(), uploadedJarLocation, jsonConf, topoBuilder.createTopology());

         = ;   success =3D true;

 

 

I looked at the storm python script and saw that main= class name is concatenated to JVM_OPTS

That’s why I added this line

 

System.setProperty("storm.jar", uploadedJar= Location.concat(" ").concat(topologyModel.getMainClass()));

 

But when I execute the code in test I only see empty= topology without spouts and bolts in Storm UI with only one worker and one= executor.  Main class in the jar has the code to submit spouts and bol= ts with parallelism 3 so most definitely main class wasn’t picked up.

Is there any way to implement this command python ${s= torm_home}/bin/storm jar ${dsl_apps_dir}/dist/${jarname} $mainclass in Java= besides calling it (which is already done) directly?

 

 

Thanks,

Michael Kazansky

 

This communication is for informational purposes only. It is not intended a= s an offer or solicitation for the purchase or sale of any financial instrum= ent or as an official confirmation of any transaction. All market prices, d= ata and other information are not warranted as to completeness or accuracy a= nd are subject to change without notice. Any comments or statements made he= rein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiar= ies and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, c= onfidential and/or exempt from disclosure under applicable law. If you are= not the intended recipient, you are hereby notified that any disclosure, co= pying, distribution, or use of the information contained herein (including a= ny reliance thereon) is STRICTLY PROHIBITED. If you received this transmiss= ion in error, please immediately contact the sender and destroy the material= in its entirety, whether in electronic or hard copy format. Although this= transmission and any attachments are believed to be free of any virus or ot= her defect that might affect any computer system into which it is received a= nd opened, it is the responsibility of the recipient to ensure that it is vi= rus free and no responsibility is accepted by JPMC for any loss or damage ar= ising in any way from its use. Please note that any electronic communicatio= n that is conducted within or through JPMC's systems is subject to intercept= ion, monitoring, review, retention and external production in accordance wi= th JPMC's policy and local laws, rules and regulations; may be stored or oth= erwise processed in countries other than the country in which you are locate= d; and will be treated in accordance with JPMC policies and applicable laws= and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures re= lating to European legal entities.

--_000_A2AA10324A422F409F7DF2D8187343BBF2CE3732SBECMX004exchad_--