Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 79CDE18762 for ; Thu, 18 Feb 2016 21:16:49 +0000 (UTC) Received: (qmail 86679 invoked by uid 500); 18 Feb 2016 21:16:47 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 86630 invoked by uid 500); 18 Feb 2016 21:16:47 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 86620 invoked by uid 99); 18 Feb 2016 21:16:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2016 21:16:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2EB04C0FD8 for ; Thu, 18 Feb 2016 21:16:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.429 X-Spam-Level: * X-Spam-Status: No, score=1.429 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id P8jANcss6B2p for ; Thu, 18 Feb 2016 21:16:45 +0000 (UTC) Received: from mail-ig0-f178.google.com (mail-ig0-f178.google.com [209.85.213.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id DC1F65FAEB for ; Thu, 18 Feb 2016 21:16:44 +0000 (UTC) Received: by mail-ig0-f178.google.com with SMTP id g6so23940857igt.1 for ; Thu, 18 Feb 2016 13:16:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=FBlzzPvikOo4Ekyopk7OwfvqA79mN9S/LV5QDHyL4EU=; b=Ye6qIHd8Op8YEMC+69/972pIbtJU5h/o3twCEPXHpIqJ3HW3BzVwJzQyhC+iqaFBUb 5yKGfqWo9Ah224sSobJHKdnMhpNY/c19yOVu3Bx4jCgWO5HXC4G6vcmHA16SZuvevB0D z6Oumnr/Tme4328qZZahb2uWb3jF2vdz9FDQGbJQECjOG6YmoL/sYtxiENyUBifLm6md ITnhAt3ptbFDVV5KdLwAvVz0ynY8ORqLtkwfBUSs1cJFxWfc9+vtQNdwztqBvUZHv5vC 0lc/Ok2+hCBtZ07klGCmctTIv8rDMTlI/xWuI2BP94SbC7ebj6CWbZYiTX/zLquANJS1 5t0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=FBlzzPvikOo4Ekyopk7OwfvqA79mN9S/LV5QDHyL4EU=; b=XDyYWjr0gKAnORlwtUTIN3nZAQBIfR9zLJOpS6fMcff/oP/Bf5iG2A/VLIS/gdyO0n qrHYZF/Q6wAFPeX1a24Boq2ZTxc/nSilVXMiF7riijD+cv+NqqQqEn2SKXpD2OqlkeeY AR1mVxaI4uRsQKJflCjm8KWnyDZfTFNO3E443BYmY3PNA9R8cRw1DIRItwz3iqjScdAn z1CUSQyk/BuKredwSI3ivQlXnAANHSLhL8dYuNj+37Q51bFWSJspRx2wheQ2z7T26vbp bpJF+1yVYuhGdPbBnOfctl1IbjLfp6FzomCyaNCITzbTDpsbp7EEqEed+xx9We6ACmEe /zYQ== X-Gm-Message-State: AG10YOQ5p6FYYXwrZK53YaMAtsWXMkm7DsbBe19O0thBZkpZBqz2pGl8CdifCFz12jhxyd8nBkILVEAXfDMM7A== MIME-Version: 1.0 X-Received: by 10.50.43.166 with SMTP id x6mr5639482igl.84.1455830197991; Thu, 18 Feb 2016 13:16:37 -0800 (PST) Received: by 10.79.108.5 with HTTP; Thu, 18 Feb 2016 13:16:37 -0800 (PST) In-Reply-To: References: Date: Thu, 18 Feb 2016 13:16:37 -0800 Message-ID: Subject: Re: submit topology using NimbusClient From: Rahul R To: user@storm.apache.org Content-Type: multipart/alternative; boundary=089e01227f36ce23a7052c11e115 --089e01227f36ce23a7052c11e115 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I had a similar use case. I just created a form to upload the jar as a file upload and execute it on nimbus hosts. ./R On Thu, Feb 18, 2016 at 11:16 AM, Kazansky, Michael < michael.kazansky@jpmchase.com> wrote: > Hi Storm gurus, > > I am working on the project which should allow users to submit their > topology jars to Storm. We successfully using command line but now I need > to write application which allows user to submit jar through let say > browser. > > Here is what I am trying to do > > > > Config conf =3D new Config(); > > TopologyBuilder topoBuilder =3D new TopologyBuilder(); > > conf.put(Config.NIMBUS_HOST, topologyModel.getHost()); > > conf.setDebug(true); > > Map stormConf =3D Utils.readStormConfig(); > > stormConf.put(Config.NIMBUS_HOST, topologyModel.getHost()); > > > > // topology submit > > NimbusClient nimbus =3D new NimbusClient(stormConf, > topologyModel.getHost(), null !=3D topologyModel.getNimbusPort() ? > topologyModel.getNimbusPort() : 6627); > > String uploadedJarLocation =3D > StormSubmitter.submitJar(stormConf, topologyModel.getArtifactUrl()); > > System.setProperty("storm.jar", uploadedJarLocation.concat(" > ").concat(topologyModel.getMainClass())); > > String jsonConf =3D JSONValue.toJSONString(stormConf); > > //StormTopology topology =3D buildToplogy(topologyModel); > > //SubmitOptions options =3D new SubmitOptions(); > > //client.jartransformer.class > > > nimbus.getClient().submitTopology(topologyModel.getTopologyName(), > uploadedJarLocation, jsonConf, topoBuilder.createTopology()); > > success =3D true; > > > > > > I looked at the storm python script and saw that main class name is > concatenated to JVM_OPTS > > That=E2=80=99s why I added this line > > > > System.setProperty("storm.jar", uploadedJarLocation.concat(" > ").concat(topologyModel.getMainClass())); > > > > But when I execute the code in test I only see empty topology without > spouts and bolts in Storm UI with only one worker and one executor. Main > class in the jar has the code to submit spouts and bolts with parallelism= 3 > so most definitely main class wasn=E2=80=99t picked up. > > Is there any way to implement this command python ${storm_home}/bin/storm > jar ${dsl_apps_dir}/dist/${jarname} $mainclass in Java besides calling it > (which is already done) directly? > > > > > > Thanks, > > Michael Kazansky > > > > This communication is for informational purposes only. It is not intended > as an offer or solicitation for the purchase or sale of any financial > instrument or as an official confirmation of any transaction. All market > prices, data and other information are not warranted as to completeness o= r > accuracy and are subject to change without notice. Any comments or > statements made herein do not necessarily reflect those of JPMorgan Chase= & > Co., its subsidiaries and affiliates (collectively, "JPMC"). This > transmission may contain information that is proprietary, privileged, > confidential and/or exempt from disclosure under applicable law. If you a= re > not the intended recipient, you are hereby notified that any disclosure, > copying, distribution, or use of the information contained herein > (including any reliance thereon) is STRICTLY PROHIBITED. If you received > this transmission in error, please immediately contact the sender and > destroy the material in its entirety, whether in electronic or hard copy > format. Although this transmission and any attachments are believed to be > free of any virus or other defect that might affect any computer system > into which it is received and opened, it is the responsibility of the > recipient to ensure that it is virus free and no responsibility is accept= ed > by JPMC for any loss or damage arising in any way from its use. Please no= te > that any electronic communication that is conducted within or through > JPMC's systems is subject to interception, monitoring, review, retention > and external production in accordance with JPMC's policy and local laws, > rules and regulations; may be stored or otherwise processed in countries > other than the country in which you are located; and will be treated in > accordance with JPMC policies and applicable laws and regulations. Please > refer to http://www.jpmorgan.com/pages/disclosures for disclosures > relating to European legal entities. > --089e01227f36ce23a7052c11e115 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I had a similar use case.=C2=A0 I just created a form= to upload the jar as a file upload and execute it on nimbus hosts.

=
./R

On Thu, Feb 18, 2016 at 11:16 AM, Kazansky, Michael <= ;michael= .kazansky@jpmchase.com> wrote:

Hi Storm gurus,

I am working on the project which should allow users= to submit their topology jars to Storm. We successfully using command line= but now I need to write application which allows user to submit jar throug= h let say browser.

Here is what I am trying to do

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Config conf =3D new Config();

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 TopologyBuilder topoBuilder =3D new TopologyBuilder();

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 conf.put(Config.NIMBUS_HOST, topologyModel.getHost());

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 conf.setDebug(true);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Map stormConf =3D Utils.readStormConfig();

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 stormConf.put(Config.NIMBUS_HOST, topologyModel.getHost());=

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0// topology submit

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 NimbusClient nimbus =3D new NimbusClient(stormConf, topolog= yModel.getHost(), null !=3D topologyModel.getNimbusPort() ? topologyModel.g= etNimbusPort() : 6627);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 String uploadedJarLocation =3D StormSubmitter.submitJar(sto= rmConf, topologyModel.getArtifactUrl());

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 System.setProperty("storm.jar", uploadedJarLocati= on.concat(" ").concat(topologyModel.getMainClass()));

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 String jsonConf =3D JSONValue.toJSONString(stormConf);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 //StormTopology topology =3D buildToplogy(topologyModel);

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 //SubmitOptions options =3D new SubmitOptions();<= /u>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 //client.jartransformer.class

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 nimbus.getClient().submitTopology(topologyModel.getTopology= Name(), uploadedJarLocation, jsonConf, topoBuilder.createTopology());

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 success =3D true;

=C2=A0

=C2=A0

I looked at the storm python script and saw that mai= n class name is concatenated to JVM_OPTS

That=E2=80=99s why I added this line <= /p>

=C2=A0

System.setProperty("storm.jar", uploadedJa= rLocation.concat(" ").concat(topologyModel.getMainClass()));

=C2=A0

But when I execute the code in test I only see empty= topology without spouts and bolts in Storm UI with only one worker and one= executor.=C2=A0 Main class in the jar has the code to submit spouts and bo= lts with parallelism 3 so most definitely main class wasn=E2=80=99t picked up.

Is there any way to implement this command python ${= storm_home}/bin/storm jar ${dsl_apps_dir}/dist/${jarname} $mainclass in Jav= a besides calling it (which is already done) directly?

=C2=A0

=C2=A0

Thanks,

Michael Kazansky

=C2=A0

This communication is for informational purposes only. It is not intended = as an offer or solicitation for the purchase or sale of any financial instr= ument or as an official confirmation of any transaction. All market prices= , data and other information are not warranted as to completeness or accura= cy and are subject to change without notice. Any comments or statements ma= de herein do not necessarily reflect those of JPMorgan Chase & Co., its= subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, = confidential and/or exempt from disclosure under applicable law. If you ar= e not the intended recipient, you are hereby notified that any disclosure, = copying, distribution, or use of the information contained herein (includin= g any reliance thereon) is STRICTLY PROHIBITED. If you received this trans= mission in error, please immediately contact the sender and destroy the mat= erial in its entirety, whether in electronic or hard copy format. Althoug= h this transmission and any attachments are believed to be free of any viru= s or other defect that might affect any computer system into which it is re= ceived and opened, it is the responsibility of the recipient to ensure that= it is virus free and no responsibility is accepted by JPMC for any loss or= damage arising in any way from its use. Please note that any electronic c= ommunication that is conducted within or through JPMC's systems is subj= ect to interception, monitoring, review, retention and external production= in accordance with JPMC's policy and local laws, rules and regulations= ; may be stored or otherwise processed in countries other than the country = in which you are located; and will be treated in accordance with JPMC polic= ies and applicable laws and regulations.=20 Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures = relating to European legal entities.


--089e01227f36ce23a7052c11e115--