livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Bozarth" <ajboz...@us.ibm.com>
Subject Re: How to cancel the running streaming job using livy?
Date Fri, 26 Jan 2018 00:10:02 GMT

I don't personally use Spark Streaming so I don't know how important the
awaitAnyTermination() call is, have you tested your code without it to see
how it works? Also I believe you are correct that if you got rid of it you
would have access to the id needed to cancel.

                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   
                                                                                     
                                                                                     
                                                                                     
 E-mail: ajbozart@us.ibm.com                                                         
 GitHub: github.com/ajbozarth                                                        
                                                                   505 Howard Street 
                                                             San Francisco, CA 94105 
                                                                       United States 
                                                                                     








From:	kant kodali <kanth909@gmail.com>
To:	user@livy.incubator.apache.org
Date:	01/24/2018 04:51 PM
Subject:	Re: How to cancel the running streaming job using livy?



Do I need to awaitAnyTermination() ? How come it works with or without in
livy case? In spark-submit case I am sure streaming jobs wont run forever
with awaitAnyTermination().

The problem here is if I post another livy job to cancel then I would need
an id. And I cannot return an id if my code has awaitTermination() like
this.

public Void call(JobContext ctx) throws Exception {
   SparkSession sparkSession = ctx.sparkSession();
   Dataset<Row> df = sparkSession.readStream().format("kafka").load();
   df.createOrReplaceTempView("table");
   Dataset<Row> resultSet1 = sparkSession.sql("select * from table");
   resultSet1.writeStream().format("console").start();  //Streaming query1
started
   Dataset<Row> resultSet2 = sparkSession.sql("select count(*) from
table");
   resultSet2.writeStream().format("console").start(); //Streaming query2
started
   sparkSession.streams().awaitAnyTermination(); // DO I NEED THIS ?
   return null;

}

On Wed, Jan 24, 2018 at 4:00 PM, Alex Bozarth <ajbozart@us.ibm.com> wrote:
  In response to Marcelo's "big if", from reading the code
  stopping/canceling/deleting Livy sessions/jobs will always "kill" any
  running Spark jobs, leaving them in a failed state. It should allow you
  to individually cancel those jobs using the cancel command from earlier
  in this mail thread, but it wont fix the final state issue you had (since
  they'll still be listed as failed). As for more long term solutions to
  this problem, from what I can tell this is a general limitation of Spark
  Streaming currently as there is no way to stop endless stream jobs
  through Spark directly.




                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   


                                                                                    
                                                                                    
                                                                                    
 E-mail: ajbozart@us.ibm.com                                                        
 GitHub: github.com/ajbozarth                                                       
                                                                  505 Howard Street 
                                                            San Francisco, CA 94105 
                                                                      United States 
                                                                                    





  Inactive hide details for Marcelo Vanzin ---01/24/2018 03:30:24 PM---No.
  Livy doesn't keep track of everything your code does -Marcelo Vanzin
  ---01/24/2018 03:30:24 PM---No. Livy doesn't keep track of everything
  your code does - that's up to you to do.

  From: Marcelo Vanzin <vanzin@cloudera.com>
  To: user@livy.incubator.apache.org
  Cc: Alex Bozarth <ajbozart@us.ibm.com>
  Date: 01/24/2018 03:30 PM

  Subject: Re: How to cancel the running streaming job using livy?



  No. Livy doesn't keep track of everything your code does - that's up to
  you to do.

  If (big if here, don't remember the code) you submit separate jobs for
  each streaming query, then maybe canceling that Livy job will cancel any
  Spark jobs started by it.

  But that makes a lot of assumptions about how the Livy code works and how
  streaming queries work internally w.r.t Spark jobs.


  On Wed, Jan 24, 2018 at 3:25 PM, kant kodali <kanth909@gmail.com> wrote:
        Sure I guess I can do that..Is that the only way? Is there any REST
        call I can make to SPARK maybe to cancel any of the Streaming
        Query? Sorry if this is too naive.

        On Wed, Jan 24, 2018 at 2:46 PM, Marcelo Vanzin <
        vanzin@cloudera.com> wrote:
        Then that has nothing to do with Livy.

        You need to store a reference to your StreamingQuery (returned by
        start()) somewhere, and if you want to stop it, call its "stop()"
        method by submitting a new Livy job that does it.

        On Wed, Jan 24, 2018 at 2:42 PM, kant kodali <kanth909@gmail.com>
        wrote:
              Ok Let me paste some code to try and avoid the confusion. In
              the below code I am running two streaming queries. Now here
              are my two simple questions.

              1) Does each Streaming Query below spawn one job or multiple
              jobs?
              2) What should I do if I need to kill everything related to
              streaming query1 but not streaming query2?


              public Void call(JobContext ctx) throws Exception {
              SparkSession sparkSession = ctx.sparkSession();
              Dataset<Row> df = sparkSession.readStream().format("kafka"
              ).load();
              df.createOrReplaceTempView("table");
                 Dataset<Row> resultSet1 = sparkSession.sql("select * from
              table");
              resultSet1.writeStream().format("console").start
              (); //Streaming query1 started
              Dataset<Row> resultSet2 = sparkSession.sql("select count(*)
              from table"); //Streaming query2 started
              sparkSession.streams().awaitAnyTermination();
              return null;

              }

              Thanks!

              On Wed, Jan 24, 2018 at 1:47 PM, Marcelo Vanzin <
              vanzin@cloudera.com> wrote:
              I'm a little confused about what is meant as a job here,
              after all this discussion...

              For "interactive sessions", stopping a session means stopping
              the SparkContext. So the final state of any running jobs in
              that session should be the same as if you stopped the
              SparkContext without explicitly stopping the jobs in a
              normal, non-Livy application.

              For batches, stopping a batch means killing the Spark
              application, so all bets are off as to what happens there.


              On Wed, Jan 24, 2018 at 1:08 PM, Alex Bozarth <
              ajbozart@us.ibm.com> wrote:
                    You are correct that you are using the term Job
                    incorrectly (at least according to how Spark/Livy uses
                    it). Each spark-submit is a a single Spark Application
                    and can include many jobs (which are broken down
                    themselves into stages and tasks). In Livy using
                    sessions would be like using spark-shell rather than
                    spark-submit, you probably want to use batches instead
                    (which utilize spark-submit), then you would use that
                    delete command as mentioned earlier. As for the result
                    being listed as FAILED and not CANCELLED, that is as
                    intended. When a Livy Session is stopped (deleted) is
                    sends a command to all the running jobs (in your case
                    each of you apps only have one "Job") to set as failed.

                    @Marcelo you wrote the code that does this, do you
                    remember why you had Jobs killed instead of cancelled
                    when a Livy session is stopped? Otherwise we may be
                    able to open a JIRA and change this, but I am unsure of
                    any potential consequences.
                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   


                                                                                    
                                                                                    
                                                                                    
 E-mail: ajbozart@us.ibm.com                                                        
 GitHub: github.com/ajbozarth                                                       
                                                                  505 Howard Street 
                                                            San Francisco, CA 94105 
                                                                      United States 
                                                                                    





                    Inactive hide details for kant kodali ---01/23/2018
                    11:44:26 PM---I tried  POST to sessions/{session
                    id}/jobs/{job id}/cancel akant kodali ---01/23/2018
                    11:44:26 PM---I tried POST to sessions/{session
                    id}/jobs/{job id}/cancel and that doesn't seem to
                    cancel either.

                    From: kant kodali <kanth909@gmail.com>
                    To: user@livy.incubator.apache.org
                    Date: 01/23/2018 11:44 PM

                    Subject: Re: How to cancel the running streaming job
                    using livy?



                    I tried  POST to sessions/{session id}/jobs/{job
                    id}/cancel and that doesn't seem to cancel either. I
                    think first of all the word "job" is used in so many
                    context that it might be misleading.

                    Imagine for a second I don't have livy and I just use
                    spark-submit command line to spawn . say I do that
                    following

                    spark-submit hello1.jar // streaming job1 (runs
                    forever)
                    spark-submit hello2.jar //streaming job2 (runs forever)

                    The number of jobs I spawned is two and now I want to
                    be able to cancel one of them..These jobs reads data
                    from kafka and will be split into stages and task now
                    sometimes these tasks are also called jobs according to
                    SPARK UI for some reason. And looks like live may be is
                    cancelling those with the above end point.

                    It would be great help if someone could try from their
                    end and see if they are able to cancel the jobs?

                    Thanks!

                    On Fri, Jan 19, 2018 at 4:03 PM, Alex Bozarth <
                    ajbozart@us.ibm.com> wrote:
                                Ah, that's why I couldn't find cancel in
                                JobHandle, but it was implemented in all
                                it's implementations, which all implement
                                it as would be expected.
                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   

                                                                                    
                                                                                    
                                                                                    
 E-mail: ajbozart@us.ibm.com                                                        
 GitHub: github.com/ajbozarth                                                       
                                                                  505 Howard Street 
                                                            San Francisco, CA 94105 
                                                                      United States 
                                                                                    





                                Inactive hide details for Marcelo Vanzin
                                ---01/19/2018 03:55:43 PM---A JobHandle
                                (which you get by submitting a Job) is a
                                FuturMarcelo Vanzin ---01/19/2018 03:55:43
                                PM---A JobHandle (which you get by
                                submitting a Job) is a Future, and Futures
                                have a "cancel()" method.

                                From: Marcelo Vanzin <vanzin@cloudera.com>
                                To: user@livy.incubator.apache.org
                                Date: 01/19/2018 03:55 PM

                                Subject: Re: How to cancel the running
                                streaming job using livy?



                                A JobHandle (which you get by submitting a
                                Job) is a Future, and Futures have a
                                "cancel()" method.

                                I don't remember the details about how
                                "cancel()" is implemented in Livy, though.

                                On Fri, Jan 19, 2018 at 3:52 PM, Alex
                                Bozarth <ajbozart@us.ibm.com> wrote:
                                                        Ok so I looked into
                                                        this a bit more. I
                                                        misunderstood you a
                                                        bit before, the
                                                        delete call is for
                                                        ending livy
                                                        sessions using the
                                                        rest API, not jobs
                                                        and not via the
                                                        Java API. As for
                                                        the Job state that
                                                        makes sense, if you
                                                        end the session the
                                                        session kills all
                                                        currently running
                                                        jobs. What you want
                                                        to to send cancel
                                                        requests to the
                                                        jobs the session is
                                                        running. From my
                                                        research I found
                                                        that there is a way
                                                        to do this via the
                                                        REST API, but it
                                                        isn't documented
                                                        for some reason.
                                                        Doing a POST
                                                        to /{session
                                                        id}/jobs/{job
                                                        id}/cancel will
                                                        cancel a job. As
                                                        for the Java API,
                                                        the feature isn't
                                                        part of the Java
                                                        interface, but most
                                                        implementations of
                                                        it add it, such as
                                                        the Scala API which
                                                        ScalaJobHandle
                                                        class on sumbit
                                                        which has a cancel
                                                        function. I'm not
                                                        sure how you're
                                                        submitting you
                                                        jobs, but there
                                                        should be a cancel
                                                        function available
                                                        to you somewhere
                                                        depending on the
                                                        client you're
                                                        using. From this
                                                        discussion I've
                                                        realized our
                                                        current
                                                        documentation is
                                                        even more lacking
                                                        that I had thought.
                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   

                                                                                    
                                                                                    
                                                                                    
 E-mail: ajbozart@us.ibm.com                                                        
 GitHub: github.com/ajbozarth                                                       
                                                                  505 Howard Street 
                                                            San Francisco, CA 94105 
                                                                      United States 
                                                                                    





                                                        Inactive hide
                                                        details for kant
                                                        kodali
                                                        ---01/18/2018
                                                        06:09:59 PM---Also
                                                        just tried the
                                                        below and got the
                                                        state. It ended up
                                                        in "kant kodali
                                                        ---01/18/2018
                                                        06:09:59 PM---Also
                                                        just tried the
                                                        below and got the
                                                        state. It ended up
                                                        in "FAILED" stated
                                                        when I expected it
                                                        to be

                                                        From: kant kodali <
                                                        kanth909@gmail.com>
                                                        To:
                                                        user@livy.incubator.apache.org

                                                        Date: 01/18/2018
                                                        06:09 PM
                                                        Subject: Re: How to
                                                        cancel the running
                                                        streaming job using
                                                        livy?




                                                        Also just tried the
                                                        below and got the
                                                        state. It ended up
                                                        in "FAILED" stated
                                                        when I expected it
                                                        to be in
                                                        "CANCELLED" state.
                                                        Also from the docs
                                                        it is not clear if
                                                        it kills the
                                                        session or the job?
                                                        if it kills the
                                                        session I can't
                                                        spawn any other
                                                        Job. Sorry
                                                        cancelling job had
                                                        been a bit
                                                        confusing for me.
                                                        DELETE /sessions/0



                                                        On Thu, Jan 18,
                                                        2018 at 5:55 PM,
                                                        kant kodali <
                                                        kanth909@gmail.com>
                                                        wrote:
                             oh this raises couple questions.

                             1) Is there a programmatic way to cancel a
                             job?

                             2) is  there any programmatic way to get
                             session id? If not, how do I get a sessionId
                             when I spawn multiple jobs or multiple
                             sessions?


                             On Thu, Jan 18, 2018 at 5:39 PM, Alex Bozarth
                             <ajbozart@us.ibm.com> wrote:
                             You make a DELETE call as detailed here:
                             http://livy.apache.org/docs/latest/rest-api.html#response
                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   

                                                                                    
                                                                                    
                                                                                    
 E-mail: ajbozart@us.ibm.com                                                        
 GitHub: github.com/ajbozarth                                                       
                                                                  505 Howard Street 
                                                            San Francisco, CA 94105 
                                                                      United States 
                                                                                    





                             Inactive hide details for kant kodali
                             ---01/18/2018 05:34:07 PM---Hi All, I was able
                             to submit a streaming job to livy however kant
                             kodali ---01/18/2018 05:34:07 PM---Hi All, I
                             was able to submit a streaming job to livy
                             however I wasn't able to find

                             From: kant kodali <kanth909@gmail.com>
                             To: user@livy.incubator.apache.org
                             Date: 01/18/2018 05:34 PM
                             Subject: How to cancel the running streaming
                             job using livy?



                             Hi All,

                             I was able to submit a streaming job to livy
                             however I wasn't able to find any way to
                             cancel the running the job? Please let me
                             know.

                             Thanks!




                                --
                                Marcelo






              --
              Marcelo



        --
        Marcelo



  --
  Marcelo











Mime
View raw message