storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian C. Huffman" <bhuff...@etinternational.com>
Subject Re: Segfault in worker when submitting topology
Date Fri, 03 Jan 2014 21:25:49 GMT
Still breaks with storm 0.9.0.1 and zmq 2.1.7 and frozen jzmq.

Just to be sure this isn't an issue - I've specified a library path in 
storm's config so that it can find the libraries for zmq and jzmq.  Is 
this ok?
java.library.path: 
"/share/storm/zeromq-2.1.7/lib:/share/storm/jzmq/lib:/usr/lib"

Thanks,
-b


On 01/03/2014 03:06 PM, Nathan Leung wrote:
> can you try storm 0.9.0.1 with zmq 2.1.7 and frozen jzmq?  assuming 
> that it doesn't work, can you try with netty as your transport?  It 
> would seem to me that something is wrong with your zmq library,
>
>
> On Fri, Jan 3, 2014 at 2:58 PM, Brian C. Huffman 
> <bhuffman@etinternational.com <mailto:bhuffman@etinternational.com>> 
> wrote:
>
>     Yes - that log was from the zmq 4 run.
>
>     Here's the hs_err_pid file from the 2.1.7 run:
>     # Problematic frame:
>     # C  [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>
>     <snip>
>
>     Stack: [0x00007f3a337d2000,0x00007f3a338d3000],
>     sp=0x00007f3a338d0f08,  free space=1019k
>
>     Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
>     C=native code)
>     C  [libzmq.so.1+0x19100]  zmq::i_inout::~i_inout()+0x10
>
>
>     Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>     j  org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>     j  org.zeromq.ZMQ$Socket.setLinger(J)V+17
>
>     Thanks,
>     Brian
>
>
>     On 01/03/2014 02:46 PM, Nathan Leung wrote:
>>     You shouldn't use a newer version of zmq, the API changed.  If
>>     you try a newer version of storm you should try it with zmq 2.1.7
>>     and the frozen jzmq.  Is the log you posted from the zmq 4 run?
>>      What does it look like with zmq 2.1.7?
>>
>>
>>     On Fri, Jan 3, 2014 at 1:56 PM, Brian C. Huffman
>>     <bhuffman@etinternational.com
>>     <mailto:bhuffman@etinternational.com>> wrote:
>>
>>         All,
>>
>>         I'm having trouble getting storm set up and submitting a
>>         topology. I've tried two different installs of storm and
>>         associated dependencies:  One using the older storm (0.8.1),
>>         zeromq (2.1.7) and jzmq (frozen for storm).  And then another
>>         using the newest of each: storm(0.9.0.1), zeromq (4.0.3), and
>>         latest jzmq master.
>>
>>         Then I'm using the storm-starter package with the
>>         WordCountTopology
>>
>>         Storm *appears* to be running fine.  I have one nimbus node
>>         and 3 supervisors.  And the UI looks good.
>>
>>         The topology submits fine:
>>         [storm@thor4 bin]$ ./storm jar
>>         /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>         storm.starter.WordCountTopology wc5
>>         Running: java -client -Dstorm.options=
>>         -Dstorm.home=/share/storm/storm-0.9.0.1
>>         -Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
>>         -Dstorm.conf.file= -cp
>>         /share/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/math.numeric-
>>         <http://0.9.0.1/lib/math.numeric->tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/share/storm/storm-0.9.0.1/lib/core.incubator-0
>>         <http://0.9.0.1/lib/core.incubator-0>.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/objenesis-1.2.jar
>>         <http://0.9.0.1/lib/objenesis-1.2.jar>:/share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-0.9.0.1/lib/json-simple-1.1
>>         <http://0.9.0.1/lib/json-simple-1.1>.jar:/share/storm/storm-0.9.0.1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5-
>>         <http://0.9.0.1/lib/servlet-api-2.5->20081211.jar:/share/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/share/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/
>>         <http://0.9.0.1/lib/httpcore-4.1.jar:/>share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
>>         -Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>         storm.starter.WordCountTopology wc5
>>         375  [main] INFO  backtype.storm.StormSubmitter - Jar not
>>         uploaded to master yet. Submitting jar...
>>         389  [main] INFO  backtype.storm.StormSubmitter - Uploading
>>         topology jar
>>         /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>         to assigned location:
>>         /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>>         410  [main] INFO  backtype.storm.StormSubmitter -
>>         Successfully uploaded topology jar to assigned location:
>>         /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>>         411  [main] INFO  backtype.storm.StormSubmitter - Submitting
>>         topology wc5 in distributed mode with conf
>>         {"topology.workers":3,"topology.debug":true}
>>         846  [main] INFO  backtype.storm.StormSubmitter - Finished
>>         submitting topology: wc5
>>
>>
>>         However, it's segfaulting.  I see many hs_err_pid files with
>>         the following content:
>>         # Problematic frame:
>>         # C  [libzmq.so.3+0x28510]  zmq::signaler_t::signaler_t()+0x30
>>
>>         <snip>
>>
>>         Stack: [0x00007f945ac1c000,0x00007f945ad1d000],
>>         sp=0x00007f945ad1aed8,  free space=1019k
>>         Native frames: (J=compiled Java code, j=interpreted, Vv=VM
>>         code, C=native code)
>>         C  [libzmq.so.3+0x28510]  zmq::signaler_t::signaler_t()+0x30
>>
>>         Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>>         j  org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>>         j  org.zeromq.ZMQ$Socket.setLinger(J)V+17
>>
>>         I've spent a few hours troubleshooting but just can't find a
>>         solution.  Can any one make suggestions on what might be wrong?
>>
>>         Thanks,
>>         Brian
>>
>>
>>
>>
>
>


Mime
View raw message