predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankil Doshi <forpan...@gmail.com>
Subject Re: Predition.io setup with Azure
Date Mon, 01 May 2017 20:35:14 GMT
Hi Donald,

As per my understanding, Azure doesn't support job submission through YARN.
It doesn't expose all YARN configs (public facing IP). They recommend
"Livy" to submit spark jobs to HDInsight cluster [0]. Any recommendations
around integrating it with Livy ?

Thanks,
Pankil

[0]:
https://blogs.msdn.microsoft.com/azuredatalake/2017/01/06/spark-job-submission-on-hdinsight-101/

On Fri, Apr 28, 2017 at 4:24 PM, Donald Szeto <donald@apache.org> wrote:

> Hi Pankil,
>
> Looks like HDInsight runs Spark on YARN. You would need to point PIO to
> the right Hadoop config files, and specify the correct master URL (yarn)
> when you run "pio train".
>
> Regards,
> Donald
>
> On Fri, Apr 28, 2017 at 12:12 PM Pankil Doshi <forpankil@gmail.com> wrote:
>
>> Hi,
>>
>> Has anyone tried running prediction.io with Azure specifically using
>> Azure's HDInsight cluster for spark ?
>>
>> I understand it has capability of pointing to remote spark cluster. But
>> trying to figure out what is involved in making it work on HDInsight
>> cluster? Can we leverage livy server to submit spark job?
>>
>> Any pointers would be really helpful.
>>
>> Thanks,
>> Pankil
>>
>

Mime
View raw message