predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bansari Shah <bansari.ja...@gmail.com>
Subject Re: Use of common argument scratch-uri and remote deployment
Date Wed, 05 Oct 2016 04:37:48 GMT
Thank you for your guidance.

We have to build and train on development machine which can be standalone
or 3 node cluster and deploy on production environment which is completely
different cluster. In this case does scratch-uri will work or we have to
follow another process.

Please suggest me.

Thank you
Regards,
Bansari

On Wed, Oct 5, 2016 at 1:30 AM, Donald Szeto <donald@apache.org> wrote:

> Hi Bansari,
>
> The --scratch-uri switch is only useful with "pio train/deploy" using YARN
> cluster mode, which is your case. It tells PredictionIO where to copy
> PredictionIO JARs and engine.json for YARN cluster mode to work properly.
>
> 1. Make sure HADOOP_CONF_DIR is set properly in conf/pio-env.sh.
> 2. Provide an HDFS URL to --scratch-uri. You need to have write access to
> this location.
>
> Regards,
> Donald
>
> On Tue, Oct 4, 2016 at 11:21 AM, Pat Ferrel <pat@occamsmachete.com> wrote:
>
>> No idea about 'scratch-uri’ but once you build a model if you have
>> specified (in pio-env.sh) that pio use hdfs for the model storage it will
>> already be available to any machine that has access to hdfs. It somewhat
>> depends on the template, the Universal Recommender uses Elasticsearch for
>> model storage so any machine with access to ES will have the model.
>>
>>
>> On Oct 4, 2016, at 10:09 AM, Bansari Shah <bansari.jan93@gmail.com>
>> wrote:
>>
>> Hi Donald,
>>
>> I am running my spark cluster of 3 node with YARN and spark driver is
>> within cluster.
>>
>> Thanks
>> Regards,
>> Bansari
>>
>> On Tue, Oct 4, 2016 at 9:59 PM, Donald Szeto <donald@apache.org> wrote:
>>
>>> Hi Bansari,
>>>
>>> How are you running your Spark cluster? Standalone, YARN, or Mesos? Are
>>> you running the Spark driver on the client or within the cluster?
>>>
>>> Regards,
>>> Donald
>>>
>>> On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <bansari.jan93@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> Can you please guide me how to use 'scratch-uri' argument in case of
>>>>  transferring all necessary files to remote location.
>>>>
>>>> And can you please suggest me any way for deploying model on remote
>>>> location which is trained and build on other machine.
>>>>
>>>> Please consider it.
>>>>
>>>> Thank you,
>>>>
>>>> Regards,
>>>> Bansari
>>>>
>>>
>>>
>>
>>
>

Mime
View raw message