livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meisam Fathi <meisam.fa...@gmail.com>
Subject Re: How to submit a streaming query that runs forever using Livy?
Date Wed, 06 Dec 2017 17:03:44 GMT
Please find some of the answers inlline.

>     SparkSession sparkSession = ctx.sparkSession();
>
>     ctx.sc().getConf().setAppName("MyJob"); // *This app name is not getting set when
I go http://localhost:8998/sessions <http://localhost:8998/sessions>*
>
>     By this time the spark session is already created. You should set the
configs before starting SparkContext.


>
> *                                       // I can see my query but Appid is always set
to null*
>
> AppId should be given to Livy by YARN (if you are running on YARN). It may
take a while to get response if YARN is busy. If you are not getting an
appID at all, then your application was not submitted correctly. You may
want to check your cluster manager UI for more information


>
> *    System.out.println("READING STREAM");*
>
> This will be executed on the driver node. If you are running Spark in
standalone mode or in client mode, the driver node is the same node that
runs Spark. If you are running Spark in cluster mode, the driver is a
random node on the cluster assigned by YARN.


>
>
>     df.printSchema(); // *Where does these print statements go ?*
>
>
>
Same as above.


> awaitTermination(); // *This thing blocks forever and I don't want to set a timeout.
*
>
>
> *                                      // so what should I do to fire and forget a streaming
job ? *
>
> I believe you can call .start()

>         livy.submit(new StreamingJob()).get(); // *This will block forever*
>
> Livy.submit.get returns a value only if the job succeeds. You may want to
use
onJobFailed(JobHandle<T> job, Throwable cause) as well to handle errors and
get a better idea why the job is not returning.



>         System.out.println("SUBMITTED JAR!");  // *The control will never get here so
I can't submit another job.*\
>
> See above.

Thanks,
Meisam

Mime
View raw message