Mailing-List: contact user-help@livy.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@livy.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAByMnGt95FED-qgVaMvvS_uDCddJdco0WaW8N1TrJCQK1U633A@mail.gmail.com>
References: <CA+iiNx9ynjyu0gspZ5zEetOgRKPCfSFZmz=SkVc2eXR7_FQWyA@mail.gmail.com>
 <CAByMnGt95FED-qgVaMvvS_uDCddJdco0WaW8N1TrJCQK1U633A@mail.gmail.com>
From: Stefan Miklosovic <miklosovic@gmail.com>
Date: Fri, 1 Dec 2017 16:42:52 +0100
Message-ID: <CAO3A4oKq86ZgNUTci9N6c+AY77iMzk8PSwAjDwDVb=hVh6K8cQ@mail.gmail.com>
Subject: Re: How to submit the job via REST ?
To: user@livy.incubator.apache.org
Content-Type: text/plain; charset="UTF-8"
archived-at: Fri, 01 Dec 2017 15:43:04 -0000

As a former Red Hatter, I am using ShrinkWrap and ShrinkWrap Maven
resolver for it.

It basically looks like this:

    public static final class JarBuilder {

        public File buildJobJar(File destinationJar) {

            final JavaArchive javaArchive =
ShrinkWrap.create(MavenImporter.class)
                .loadPomFromFile("pom.xml")
                .importBuildOutput()
                .as(JavaArchive.class)
                .addClass(MyLivyJob.class);

            logger.info(javaArchive.toString(true));

            javaArchive.as(ZipExporter.class).exportTo(destinationJar, true);

            return destinationJar;
        }


        // for every other dependency in the runtime which is not
bundled into the build jar
        // you would need to make something similar as below

        public File buildHttpClientJar() {
            return Maven.configureResolver()
                .withMavenCentralRepo(true)
                .resolve(HTTP_CLIENT_COORDINATES)
                .withoutTransitivity()
                .asSingleFile();
        }

        public File buildJacksonDatatyeJodaJar() {
            return Maven.configureResolver()
                .withMavenCentralRepo(true)
                .resolve(JACKSON_DATATYPE_JODA_COORDINATES)
                .withoutTransitivity()
                .asSingleFile();
        }
    }

After that, you just use uploadJar on livy client instance. After
that, you can submit your job via submit method on Livy client.

Notice that I made a jar and I added that job class into that. If you
dont add it there and you upload it, it will not know how to
deserialize it because that compiled class would not be there.

On Fri, Dec 1, 2017 at 3:54 PM, Meisam Fathi <meisam.fathi@gmail.com> wrote:
> You should compile and package PiJar before running this code snippet. It
> does not need to be a separate app/project. You can put the PiJob code right
> next to the code snippet to run it. MVN/sbt/gradle can create the jar for
> you and I assume there is a way to call them programmatically, but that is
> not needed. You can use the path to the jar file as piJar.
>
> I hope this answers your question.
>
> Thanks,
> Meisam
>
> import org.apache.spark.api.java.function.*;
>
> import org.apache.livy.*;
>
> public class PiJob implements Job<Double>, Function<Integer, Integer>,
>         Function2<Integer, Integer, Integer> {
>
>   private final int samples;
>
>   public PiJob(int samples) {
>     this.samples = samples;
>   }
>
>   @Override
>   public Double call(JobContext ctx) throws Exception {
>     List<Integer> sampleList = new ArrayList<Integer>();
>     for (int i = 0; i < samples; i++) {
>       sampleList.add(i + 1);
>     }
>
>     return 4.0d * ctx.sc().parallelize(sampleList).map(this).reduce(this) /
> samples;
>   }
>
>   @Override
>   public Integer call(Integer v1) {
>     double x = Math.random();
>     double y = Math.random();
>     return (x*x + y*y < 1) ? 1 : 0;
>   }
>
>   @Override
>   public Integer call(Integer v1, Integer v2) {
>     return v1 + v2;
>   }
>
> }
>
>
> On Fri, Dec 1, 2017 at 1:09 AM kant kodali <kanth909@gmail.com> wrote:
>>
>> Hi All,
>>
>> I am looking at the following snippet of code and I wonder where and how
>> do I create piJar ? can I create programmatically if so how? is there a
>> complete hello world example somewhere where I can follow steps and see how
>> this works?
>>
>> Concerning line
>>
>> client.uploadJar(new File(piJar)).get();
>>
>>
>>
>> Code snippet
>>
>> LivyClient client = new LivyClientBuilder()
>>   .setURI(new URI(livyUrl))
>>   .build();
>>
>> try {
>>   System.err.printf("Uploading %s to the Spark context...\n", piJar);
>>   client.uploadJar(new File(piJar)).get();
>>
>>   System.err.printf("Running PiJob with %d samples...\n", samples);
>>   double pi = client.submit(new PiJob(samples)).get();
>>
>>   System.out.println("Pi is roughly: " + pi);
>> } finally {
>>   client.stop(true);
>> }


-- 
Stefan Miklosovic