storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Spico Florin <spicoflo...@gmail.com>
Subject Big fat jar
Date Thu, 04 Feb 2016 11:50:38 GMT
Hello!
  After building my project that comtaint the topology, I have a big fat
jar of 75MB. I have dependencies on HBase, OpenTSDB and Kafka. I would like
to reduce the size of the the jar due to the fact that we can a lot of
instances of the topology running (aprox 100).

I have read in two posts:
http://qnalist.com/questions/4712134/is-there-a-way-to-add-a-custom-jar-or-directory-of-jars-to-the-storm-classpath-without-copying-the-jar-s-to-storm-lib-folder
and
http://programmers.stackexchange.com/questions/238711/why-does-storm-not-supply-a-mechanism-for-supplying-topology-necessary-dependent

I found two solutions.
One is to put the commons libararies in Storm_install_dir/lib   and the
second one is to put them in folder USER_CONF_DIR defined in storm.yaml. In
both acases we should take care to keep a single version of dependent
libraries. This could be a potential issue, due to the fact that their
might be common libraries that can interfere with storm libraries (such as
zookeeper or appache-commons).
 Also, I'm running Storm from Ambari.

In the above scenario:
- what could be the best solution to keep the common dependent libraries?
- If I configure the USER_CONF_DIR in Ambari will it be propagated to all
cluster machines?

I look forward for your answers.
Thanks .
 Florin

Mime
View raw message