flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Trevor Grant <trevor.d.gr...@gmail.com>
Subject Re: Setting up zeppelin with flink
Date Thu, 25 Aug 2016 16:01:58 GMT
I'm glad you were able to work it out!

Your setup is somewhat unique, and as Zeppelin is the result of multiple
drive-by commits, interesting and unexpected things happen in the tail
cases.

Could you please report your problem and solution on the Zeppelin user
list?  What you've discovered may in fact be a bug or a regression caused
by some of the recent Spark 2.0/scala 2.11 mess (I see you installed
Zeppelin 0.6.1).  Suffice to say, I don't think this is a Flink issue.


Finally, out of curiosity- what jars did you copy to the interpreter/flink
directory to get this to work?  I'd like to check the Zeppelin/flink/pom.xml

Happy to be a sounding board if nothing else ;)

tg


Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org

*"Fortunate is he, who is able to know the causes of things."  -Virgil*


On Thu, Aug 25, 2016 at 8:57 AM, Frank Dekervel <kervel@gmail.com> wrote:

> Hello,
>
> Sorry for the spam, but i got it working after copying all scala libraries
> from another interpreter to the interpreter/flink directory. so i think the
> error is the scala libraries are missing from the binary release in the
> zeppelin/interpreters/flink/ directory. For now i'm adding the copy
> commands to the dockerfile, but I'm sure this is not the proper way to fix
> it, but i don't know maven enough to understand why the scala libs are
> missing for the flink interpreter but not for the ignite interpreter.
>
> I'm also unable to figure out why a local interpreter worked fine given
> the missing libraries ...
>
> greetings,
> Frank
>
>
> On Thu, Aug 25, 2016 at 3:08 PM, Frank Dekervel <kervel@gmail.com> wrote:
>
>> Hello,
>>
>> For reference, below is the dockerfile i used to build the zeppelin image
>> (basically just openjdk 8 with the latest binary release of zeppelin)
>> the "docker-entrypoint.sh" script is just starting zeppelin.sh (oneliner)
>>
>> FROM openjdk:alpine
>>
>> RUN apk add --no-cache bash snappy
>>
>> ARG ZEPPELIN_VERSION=0.6.1
>>
>> ARG INSTALL_PATH=/opt
>> ENV APP_HOME $INSTALL_PATH/zeppelin
>> ENV PATH $PATH:$APP_HOME/bin
>>
>> RUN set -x && \
>>   mkdir -p $INSTALL_PATH && \
>>   apk --update add --virtual build-dependencies curl && \
>>   curl -s $(curl -s https://www.apache.org/dyn/clo
>> ser.cgi\?preferred\=true)/zeppelin/zeppelin-0.6.1/zeppelin-$
>> ZEPPELIN_VERSION-bin-all.tgz | \
>>   tar xvz -C $INSTALL_PATH && \
>>   ln -s $INSTALL_PATH/zeppelin-$ZEPPELIN_VERSION-bin-all $APP_HOME && \
>>   addgroup -S zeppelin && adduser -D -S -H -G zeppelin -h $APP_HOME
>> zeppelin && \
>>   chown -R zeppelin:zeppelin $INSTALL_PATH && \
>>   chown -h zeppelin:zeppelin $APP_HOME && \
>>   apk del build-dependencies && \
>>   rm -rf /var/cache/apk/*
>>
>> # Configure container
>> USER zeppelin
>> ADD docker-entrypoint.sh $APP_HOME/bin/
>> ENTRYPOINT ["docker-entrypoint.sh"]
>> CMD ["sh", "-c"]
>>
>> greetings,
>> Frank
>>
>>
>> On Thu, Aug 25, 2016 at 2:58 PM, Frank Dekervel <kervel@gmail.com> wrote:
>>
>>> Hello Trevor,
>>>
>>> Thanks for your suggestion. The log does not explain a lot: on the flink
>>> side i don't see anything at all, on the zeppelin side i see this:
>>> Your suggestion sounds plausible, as i always start zeppelin, and then
>>> change the configuration from local to remote.. however, port 6123 locally
>>> doesn't seem to be open
>>>
>>> ==> zeppelin--94490c51d71e.log <==
>>>  INFO [2016-08-25 12:53:24,168] ({qtp846063400-48}
>>> InterpreterFactory.java[createInterpretersForNote]:576) - Create
>>> interpreter instance flink for note 2BW8NMCKW
>>>  INFO [2016-08-25 12:53:24,168] ({qtp846063400-48}
>>> InterpreterFactory.java[createInterpretersForNote]:606) - Interpreter
>>> org.apache.zeppelin.flink.FlinkInterpreter 795344042 created
>>>  INFO [2016-08-25 12:53:24,169] ({pool-1-thread-3}
>>> SchedulerFactory.java[jobStarted]:131) - Job
>>> paragraph_1471964818018_1833520437 started by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpretershar
>>> ed_session513606587
>>>  INFO [2016-08-25 12:53:24,170] ({pool-1-thread-3}
>>> Paragraph.java[jobRun]:252) - run paragraph 20160823-150658_99117457 using
>>> null org.apache.zeppelin.interpreter.LazyOpenInterpreter@2f67fcaa
>>>  INFO [2016-08-25 12:53:24,170] ({pool-1-thread-3}
>>> RemoteInterpreterProcess.java[reference]:148) - Run interpreter process
>>> [/opt/zeppelin/bin/interpreter.sh, -d, /opt/zeppelin/interpreter/flink,
>>> -p, 45769, -l, /opt/zeppelin/local-repo/2BVEQGGEN]
>>>  INFO [2016-08-25 12:53:24,672] ({pool-1-thread-3}
>>> RemoteInterpreter.java[init]:170) - Create remote interpreter
>>> org.apache.zeppelin.flink.FlinkInterpreter
>>>
>>> after doing %flink, i see this in ps auxw:
>>>
>>>  /usr/lib/jvm/java-1.8-openjdk/bin/java -Dfile.encoding=UTF-8
>>> -Dlog4j.configuration=file:///opt/zeppelin/conf/log4j.properties
>>> -Dzeppelin.log.file=/opt/zeppelin/logs/zeppelin-interpreter-flink--94490c51d71e.log
>>> -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/opt/zeppelin/interpreter/fl
>>> ink/*::/opt/zeppelin/lib/zeppelin-interpreter-0.6.1.jar
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 45769
>>>
>>> /usr/lib/jvm/java-1.8-openjdk/bin/java -Dfile.encoding=UTF-8 -Xms1024m
>>> -Xmx1024m -XX:MaxPermSize=512m -Dlog4j.configuration=file:///
>>> opt/zeppelin/conf/log4j.properties -Dzeppelin.log.file=/opt/zeppe
>>> lin/logs/zeppelin--94490c51d71e.log -cp ::/opt/zeppelin/lib/*:/opt/zeppelin/*::/opt/zeppelin/conf
>>> org.apache.zeppelin.server.ZeppelinServer
>>>
>>> the stdout of zeppelin flink process after doing a stackdump looks like
>>> this (note the exception, i didn't notice it before)
>>>
>>> zeppelin_1     | Exception in thread "pool-1-thread-2"
>>> java.lang.NoClassDefFoundError: scala/collection/Seq
>>> zeppelin_1     |        at java.lang.Class.forName0(Native Method)
>>> zeppelin_1     |        at java.lang.Class.forName(Class.java:264)
>>> zeppelin_1     |        at org.apache.zeppelin.interprete
>>> r.remote.RemoteInterpreterServer.createInterpreter(RemoteInt
>>> erpreterServer.java:148)
>>> zeppelin_1     |        at org.apache.zeppelin.interprete
>>> r.thrift.RemoteInterpreterService$Processor$createInterprete
>>> r.getResult(RemoteInterpreterService.java:1409)
>>> zeppelin_1     |        at org.apache.zeppelin.interprete
>>> r.thrift.RemoteInterpreterService$Processor$createInterprete
>>> r.getResult(RemoteInterpreterService.java:1394)
>>> zeppelin_1     |        at org.apache.thrift.ProcessFunct
>>> ion.process(ProcessFunction.java:39)
>>> zeppelin_1     |        at org.apache.thrift.TBaseProcess
>>> or.process(TBaseProcessor.java:39)
>>> zeppelin_1     |        at org.apache.thrift.server.TThre
>>> adPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> zeppelin_1     |        at java.lang.Thread.run(Thread.java:745)
>>> zeppelin_1     | Caused by: java.lang.ClassNotFoundException:
>>> scala.collection.Seq
>>> zeppelin_1     |        at java.net.URLClassLoader.findCl
>>> ass(URLClassLoader.java:381)
>>> zeppelin_1     |        at java.lang.ClassLoader.loadClas
>>> s(ClassLoader.java:424)
>>> zeppelin_1     |        at sun.misc.Launcher$AppClassLoad
>>> er.loadClass(Launcher.java:331)
>>> zeppelin_1     |        at java.lang.ClassLoader.loadClas
>>> s(ClassLoader.java:357)
>>> zeppelin_1     |        ... 11 more
>>> zeppelin_1     | 2016-08-25 12:56:03
>>> zeppelin_1     | Full thread dump OpenJDK 64-Bit Server VM (25.92-b14
>>> mixed mode):
>>> zeppelin_1     |
>>> zeppelin_1     | "pool-1-thread-5" #15 prio=5 os_prio=0
>>> tid=0x00005567976e8000 nid=0x108 waiting on condition [0x00007fa83ca8d000]
>>> zeppelin_1     |    java.lang.Thread.State: WAITING (parking)
>>> zeppelin_1     |        at sun.misc.Unsafe.park(Native Method)
>>> zeppelin_1     |        - parking to wait for  <0x00000000ebc3dae0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> zeppelin_1     |        at java.util.concurrent.locks.Loc
>>> kSupport.park(LockSupport.java:175)
>>> zeppelin_1     |        at java.util.concurrent.Synchrono
>>> usQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
>>> zeppelin_1     |        at java.util.concurrent.Synchrono
>>> usQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> zeppelin_1     |        at java.util.concurrent.Synchrono
>>> usQueue.take(SynchronousQueue.java:924)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> zeppelin_1     |        at java.lang.Thread.run(Thread.java:745)
>>> zeppelin_1     |
>>> zeppelin_1     | "pool-1-thread-4" #14 prio=5 os_prio=0
>>> tid=0x0000556797874800 nid=0x107 in Object.wait() [0x00007fa83cb8d000]
>>> zeppelin_1     |    java.lang.Thread.State: TIMED_WAITING (on object
>>> monitor)
>>> zeppelin_1     |        at java.lang.Object.wait(Native Method)
>>> zeppelin_1     |        - waiting on <0x00000000eba208c0> (a
>>> java.util.LinkedList)
>>> zeppelin_1     |        at org.apache.zeppelin.interprete
>>> r.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterp
>>> reterEventClient.java:200)
>>> zeppelin_1     |        - locked <0x00000000eba208c0> (a
>>> java.util.LinkedList)
>>> zeppelin_1     |        at org.apache.zeppelin.interprete
>>> r.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterS
>>> erver.java:543)
>>> zeppelin_1     |        at org.apache.zeppelin.interprete
>>> r.thrift.RemoteInterpreterService$Processor$getEvent.getResu
>>> lt(RemoteInterpreterService.java:1610)
>>> zeppelin_1     |        at org.apache.zeppelin.interprete
>>> r.thrift.RemoteInterpreterService$Processor$getEvent.getResu
>>> lt(RemoteInterpreterService.java:1595)
>>> zeppelin_1     |        at org.apache.thrift.ProcessFunct
>>> ion.process(ProcessFunction.java:39)
>>> zeppelin_1     |        at org.apache.thrift.TBaseProcess
>>> or.process(TBaseProcessor.java:39)
>>> zeppelin_1     |        at org.apache.thrift.server.TThre
>>> adPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> zeppelin_1     |        at java.lang.Thread.run(Thread.java:745)
>>> zeppelin_1     |
>>> zeppelin_1     | "pool-1-thread-3" #13 prio=5 os_prio=0
>>> tid=0x0000556797871800 nid=0x106 runnable [0x00007fa83cc8e000]
>>> zeppelin_1     |    java.lang.Thread.State: RUNNABLE
>>> zeppelin_1     |        at java.net.SocketInputStream.socketRead0(Native
>>> Method)
>>> zeppelin_1     |        at java.net.SocketInputStream.soc
>>> ketRead(SocketInputStream.java:116)
>>> zeppelin_1     |        at java.net.SocketInputStream.rea
>>> d(SocketInputStream.java:170)
>>> zeppelin_1     |        at java.net.SocketInputStream.rea
>>> d(SocketInputStream.java:141)
>>> zeppelin_1     |        at java.io.BufferedInputStream.fi
>>> ll(BufferedInputStream.java:246)
>>> zeppelin_1     |        at java.io.BufferedInputStream.re
>>> ad1(BufferedInputStream.java:286)
>>> zeppelin_1     |        at java.io.BufferedInputStream.re
>>> ad(BufferedInputStream.java:345)
>>> zeppelin_1     |        - locked <0x00000000ebfb7be0> (a
>>> java.io.BufferedInputStream)
>>> zeppelin_1     |        at org.apache.thrift.transport.TI
>>> OStreamTransport.read(TIOStreamTransport.java:127)
>>> zeppelin_1     |        at org.apache.thrift.transport.TT
>>> ransport.readAll(TTransport.java:86)
>>> zeppelin_1     |        at org.apache.thrift.protocol.TBi
>>> naryProtocol.readAll(TBinaryProtocol.java:429)
>>> zeppelin_1     |        at org.apache.thrift.protocol.TBi
>>> naryProtocol.readI32(TBinaryProtocol.java:318)
>>> zeppelin_1     |        at org.apache.thrift.protocol.TBi
>>> naryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>>> zeppelin_1     |        at org.apache.thrift.TBaseProcess
>>> or.process(TBaseProcessor.java:27)
>>> zeppelin_1     |        at org.apache.thrift.server.TThre
>>> adPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> zeppelin_1     |        at java.util.concurrent.ThreadPoo
>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> zeppelin_1     |        at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> the local port 6123 is closed (it is open on the jobmanager tho)
>>>
>>> bash-4.3$ telnet localhost 6123
>>> telnet: can't connect to remote host (127.0.0.1): Connection refused
>>>
>>> thanks!
>>> greetings,
>>> Frank
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 24, 2016 at 5:15 PM, Trevor Grant <trevor.d.grant@gmail.com>
>>> wrote:
>>>
>>>> Frank,
>>>>
>>>> can you post the zeppelin flink log please?
>>>>
>>>> You can probably find it in zeppelin_dir/logs/*flink*.log
>>>>
>>>> You've got a few moving pieces here.  I've never run zeppelin against
>>>> Flink in a docker container.   But I think the Zeppelin-Flink log is the
>>>> first place to look.
>>>>
>>>> You say you can't get Zeppelin to work in local mode either right? Just
>>>> curious, is Zeppelin running in a docker too?
>>>>
>>>> Thanks,
>>>> tg
>>>>
>>>>
>>>> Trevor Grant
>>>> Data Scientist
>>>> https://github.com/rawkintrevo
>>>> http://stackexchange.com/users/3002022/rawkintrevo
>>>> http://trevorgrant.org
>>>>
>>>> *"Fortunate is he, who is able to know the causes of things."  -Virgil*
>>>>
>>>>
>>>> On Wed, Aug 24, 2016 at 6:50 AM, Maximilian Michels <mxm@apache.org>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> There are some people familiar with the Zeppelin integration. CCing
>>>>> Till and Trevor. Otherwise, you could also send this to the Zeppelin
>>>>> community.
>>>>>
>>>>> Cheers,
>>>>> Max
>>>>>
>>>>> On Wed, Aug 24, 2016 at 12:58 PM, Frank Dekervel <kervel@gmail.com>
>>>>> wrote:
>>>>> > Hello,
>>>>> >
>>>>> > for reference:
>>>>> >
>>>>> > i already found out that "connect to existing process" was my error
>>>>> here: it
>>>>> > means connecting to an existing zeppelin interpreter, not an
>>>>> existing flink
>>>>> > cluster. After fixing my error, i'm now in the same situation as
>>>>> described
>>>>> > here:
>>>>> >
>>>>> > https://stackoverflow.com/questions/38688277/flink-zeppelin-
>>>>> not-responding
>>>>> >
>>>>> > i guess it's more a zeppelin problem than a flink problem tho, as
i
>>>>> see both
>>>>> > interpreter JVM and main zeppelin JVM waiting on thrift input (so
it
>>>>> seems
>>>>> > they are waiting for each other)
>>>>> >
>>>>> > greetings,
>>>>> > Frank
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Tue, Aug 23, 2016 at 2:09 PM, Frank Dekervel <kervel@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Hello,
>>>>> >>
>>>>> >> I try to set up apache zeppelin with a flink cluster (one
>>>>> jobmanager, one
>>>>> >> task manager).
>>>>> >>
>>>>> >> What i did was using the dockerfiles in flink-contrib/docker-flink
>>>>> + the
>>>>> >> latest binary release of apache zeppelin with all interpreters:
>>>>> >>
>>>>> >>
>>>>> >> https://github.com/apache/flink/blob/master/flink-contrib/do
>>>>> cker-flink/Dockerfile
>>>>> >> (i changed the flink version to 1.0.3 to match zeppelin's flink
>>>>> version)
>>>>> >>
>>>>> >> I built another docker image around the latest binary release
of
>>>>> zeppelin
>>>>> >> (with all interpreters), and i reconfigure the flink interpreter:
>>>>> >>
>>>>> >> connect to existing process
>>>>> >> host: jobmanager, port: 6123
>>>>> >> i removed all other properties
>>>>> >>
>>>>> >> when i try to submit a flink job, i get an error state and the
>>>>> following
>>>>> >> exception appears in the log (nothing appears in the jobmanager
log)
>>>>> >>
>>>>> >> ERROR [2016-08-23 11:44:57,932] ({Thread-16}
>>>>> >> JobProgressPoller.java[run]:54) - Can not get or update progress
>>>>> >> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> >> org.apache.thrift.transport.TTransportException
>>>>> >>         at
>>>>> >> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.get
>>>>> Progress(RemoteInterpreter.java:373)
>>>>> >>         at
>>>>> >> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgr
>>>>> ess(LazyOpenInterpreter.java:111)
>>>>> >>         at
>>>>> >> org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:237)
>>>>> >>         at
>>>>> >> org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgr
>>>>> essPoller.java:51)
>>>>> >> Caused by: org.apache.thrift.transport.TTransportException
>>>>> >>         at
>>>>> >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStrea
>>>>> mTransport.java:132)
>>>>> >>         at
>>>>> >> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>>>>> >>         at
>>>>> >> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryPr
>>>>> otocol.java:429)
>>>>> >>         at
>>>>> >> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryPr
>>>>> otocol.java:318)
>>>>> >>         at
>>>>> >> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(
>>>>> TBinaryProtocol.java:219)
>>>>> >>         at
>>>>> >> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.
>>>>> java:69)
>>>>> >>         at
>>>>> >> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ
>>>>> ice$Client.recv_getProgress(RemoteInterpreterService.java:296)
>>>>> >>         at
>>>>> >> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ
>>>>> ice$Client.getProgress(RemoteInterpreterService.java:281)
>>>>> >>         at
>>>>> >> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.get
>>>>> Progress(RemoteInterpreter.java:370)
>>>>> >>         ... 3 more
>>>>> >>
>>>>> >> Flink in local mode works fine on zeppelin.
>>>>> >> Could somebody point me to what i'm doing wrong ?
>>>>> >>
>>>>> >> Thanks a lot!
>>>>> >> Frank
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message