Mailing-List: contact user-help@storm.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@storm.apache.org
MIME-Version: 1.0
Date: Thu, 4 Feb 2016 13:50:38 +0200
Message-ID: 
 <CAOy42CvSgvmqL+Vuz+a8Gye5BVRBSTAwoKAKZbOE1SPFydCKhg@mail.gmail.com>
Subject: Big fat jar
From: Spico Florin <spicoflorin@gmail.com>
To: "user@storm.incubator.apache.org" <user@storm.incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a113f8df8dcaee1052af0571b

--001a113f8df8dcaee1052af0571b
Content-Type: text/plain; charset=UTF-8

Hello!
  After building my project that comtaint the topology, I have a big fat
jar of 75MB. I have dependencies on HBase, OpenTSDB and Kafka. I would like
to reduce the size of the the jar due to the fact that we can a lot of
instances of the topology running (aprox 100).

I have read in two posts:
http://qnalist.com/questions/4712134/is-there-a-way-to-add-a-custom-jar-or-directory-of-jars-to-the-storm-classpath-without-copying-the-jar-s-to-storm-lib-folder
and
http://programmers.stackexchange.com/questions/238711/why-does-storm-not-supply-a-mechanism-for-supplying-topology-necessary-dependent

I found two solutions.
One is to put the commons libararies in Storm_install_dir/lib   and the
second one is to put them in folder USER_CONF_DIR defined in storm.yaml. In
both acases we should take care to keep a single version of dependent
libraries. This could be a potential issue, due to the fact that their
might be common libraries that can interfere with storm libraries (such as
zookeeper or appache-commons).
 Also, I'm running Storm from Ambari.

In the above scenario:
- what could be the best solution to keep the common dependent libraries?
- If I configure the USER_CONF_DIR in Ambari will it be propagated to all
cluster machines?

I look forward for your answers.
Thanks .
 Florin

--001a113f8df8dcaee1052af0571b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello!<div>=C2=A0 After building my project that comtaint =
the topology, I have a big fat jar of 75MB. I have dependencies on HBase, O=
penTSDB and Kafka. I would like to reduce the size of the the jar due to th=
e fact that we can a lot of instances of the topology running (aprox 100).<=
/div><div><br></div><div>I have read in two posts:</div><div><a href=3D"htt=
p://qnalist.com/questions/4712134/is-there-a-way-to-add-a-custom-jar-or-dir=
ectory-of-jars-to-the-storm-classpath-without-copying-the-jar-s-to-storm-li=
b-folder">http://qnalist.com/questions/4712134/is-there-a-way-to-add-a-cust=
om-jar-or-directory-of-jars-to-the-storm-classpath-without-copying-the-jar-=
s-to-storm-lib-folder</a></div><div>and=C2=A0</div><div><a href=3D"http://p=
rogrammers.stackexchange.com/questions/238711/why-does-storm-not-supply-a-m=
echanism-for-supplying-topology-necessary-dependent">http://programmers.sta=
ckexchange.com/questions/238711/why-does-storm-not-supply-a-mechanism-for-s=
upplying-topology-necessary-dependent</a></div><div><br></div><div>I found =
two solutions.</div><div>One is to put the commons libararies in Storm_inst=
all_dir/lib =C2=A0 and the second one is to put them in folder=C2=A0USER_CO=
NF_DIR defined in storm.yaml. In both acases we should take care to keep a =
single version of dependent libraries. This could be a potential issue, due=
 to the fact that their might be common libraries that can interfere with s=
torm libraries (such as zookeeper or appache-commons).</div><div>=C2=A0Also=
, I&#39;m running Storm from Ambari.</div><div>=C2=A0</div><div>In the abov=
e scenario:</div><div>- what could be the best solution to keep the common =
dependent libraries?</div><div>- If I configure the USER_CONF_DIR in Ambari=
 will it be propagated to all cluster machines?</div><div><br></div><div>I =
look forward for your answers.</div><div>Thanks .</div><div>=C2=A0Florin</d=
iv></div>

--001a113f8df8dcaee1052af0571b--