predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Szeto <don...@apache.org>
Subject Re: PIO train error on Spark/Hbase remote cluster
Date Fri, 31 Mar 2017 16:22:16 GMT
Can you show the relevant parts from pio.log, please? If you don't care
about existing log messages, the easiest way would be to delete pio.log
from where you run the pio command and start fresh.

On Fri, Mar 31, 2017 at 8:46 AM, Malay Tripathi <malaytripathi3@gmail.com>
wrote:

> I think it's Yarn based, setup through Ambari.
>
>
> On Mar 31, 2017, at 6:29 AM, Donald Szeto <donald@apache.org> wrote:
>
> Hi Malay,
>
> Is your Spark cluster a standalone deployment or based on YARN?
>
> Regards,
> Donald
>
> On Thu, Mar 30, 2017 at 11:48 PM Malay Tripathi <malaytripathi3@gmail.com>
> wrote:
>
>> Hello,
>>
>> I am running pio train *on an edge node* of distributed 8 node spark
>> cluster & 3 node Hbase.
>> When I run "pio train" the job runs but it runs on local spark & not
>> submitted to cluster.
>> If I do "pio train *--master spark://localhost:7077" *or "pio train *--master
>> yarn-cluster" *I get below error -
>>
>> * File does not exist:
>> hdfs://mdc2vra179.federated.fds:8020/user/da_mcom_milan/.sparkStaging/application_1489598450058_0024/template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar*
>>
>> *java.io.FileNotFoundException: File does not exist:
>> hdfs://mdc2vra179.federated.fds:8020/user/da_mcom_milan/.sparkStaging/application_1489598450058_0024/template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar*
>>
>> *at
>> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1427)*
>>
>> *at
>> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)*
>>
>>
>> mdc2vra179 is my Hbase cluster node, also running Namenode. Not sure why
>> my spark expecting a jar file on hbase/Namenode.
>> *$PIO_HOME/conf/pio-env.sh-*
>>
>> SPARK_HOME=$PIO_HOME/vendors/spark
>>
>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase/conf
>>
>> PIO_FS_BASEDIR=$HOME/.pio_store
>>
>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>>
>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>
>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>>
>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>
>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>>
>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>
>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>>
>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>
>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-prod
>>
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176
>>
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>
>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>
>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>>
>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase
>>
>>
>> Thanks,
>>
>> Malay
>>
>

Mime
View raw message