predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhimanyu Nagrath <abhimanyunagr...@gmail.com>
Subject Re: Not able to train data
Date Thu, 26 Oct 2017 06:59:48 GMT
Hi Vaghawan,

yes, the spark master connection string is correct I am getting executor
fails to connect to spark master after 4-5 hrs.


Regards,
Abhimanyu

On Thu, Oct 26, 2017 at 12:17 PM, Sachin Kamkar <sachinkamkar@gmail.com>
wrote:

> It should be correct, as the user got the exception after 3-4 hours of
> starting. So looks like something else broke. OOM?
>
> With Regards,
>
>      Sachin
> ⚜KTBFFH⚜
>
> On Thu, Oct 26, 2017 at 12:15 PM, Vaghawan Ojha <vaghawan781@gmail.com>
> wrote:
>
>> "Executor failed to connect with master ", are you sure the --master
>> spark://*.*.*.*:7077 is correct?
>>
>> Like the one you copied from the spark master's web ui? sometimes having
>> that wrong fails to connect with the spark master.
>>
>> Thanks
>>
>> On Thu, Oct 26, 2017 at 12:02 PM, Abhimanyu Nagrath <
>> abhimanyunagrath@gmail.com> wrote:
>>
>>> I am new to predictionIO . I am using template
>>> https://github.com/EmergentOrder/template-scala-probabilisti
>>> c-classifier-batch-lbfgs.
>>>
>>> My training dataset count is 1184603 having approx 6500 features. I am
>>> using ec2 r4.8xlarge system (240 GB RAM, 32 Cores, 200 GB Swap).
>>>
>>>
>>> I tried two ways for training
>>>
>>>  1. Command '
>>>
>>> > pio train -- --driver-memory 120G --executor-memory 100G -- conf
>>> > spark.network.timeout=10000000
>>>
>>> '
>>>   Its throwing exception after 3-4 hours.
>>>
>>>
>>>     Exception in thread "main" org.apache.spark.SparkException: Job
>>> aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most
>>> recent failure: Lost task 0.0 in stage 1.0 (TID 15, localhost, executor
>>> driver): ExecutorLostFailure (executor driver exited caused by one of the
>>> running tasks) Reason: Executor heartbeat timed out after 181529 ms
>>>     Driver stacktrace:
>>>             at org.apache.spark.scheduler.DAGScheduler.org
>>> $apache$spark$scheduler$DAGScheduler$$failJobAn
>>> dIndependentStages(DAGScheduler.scala:1435)
>>>             at org.apache.spark.scheduler.DAG
>>> Scheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
>>>             at org.apache.spark.scheduler.DAG
>>> Scheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
>>>             at scala.collection.mutable.Resiz
>>> ableArray$class.foreach(ResizableArray.scala:59)
>>>             at scala.collection.mutable.Array
>>> Buffer.foreach(ArrayBuffer.scala:48)
>>>             at org.apache.spark.scheduler.DAG
>>> Scheduler.abortStage(DAGScheduler.scala:1422)
>>>             at org.apache.spark.scheduler.DAG
>>> Scheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
>>>             at org.apache.spark.scheduler.DAG
>>> Scheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
>>>             at scala.Option.foreach(Option.scala:257)
>>>             at org.apache.spark.scheduler.DAG
>>> Scheduler.handleTaskSetFailed(DAGScheduler.scala:802)
>>>             at org.apache.spark.scheduler.DAG
>>> SchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
>>>             at org.apache.spark.scheduler.DAG
>>> SchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
>>>             at org.apache.spark.scheduler.DAG
>>> SchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
>>>             at org.apache.spark.util.EventLoo
>>> p$$anon$1.run(EventLoop.scala:48)
>>>             at org.apache.spark.scheduler.DAG
>>> Scheduler.runJob(DAGScheduler.scala:628)
>>>             at org.apache.spark.SparkContext.
>>> runJob(SparkContext.scala:1918)
>>>             at org.apache.spark.SparkContext.
>>> runJob(SparkContext.scala:1931)
>>>             at org.apache.spark.SparkContext.
>>> runJob(SparkContext.scala:1944)
>>>             at org.apache.spark.rdd.RDD$$anon
>>> fun$take$1.apply(RDD.scala:1353)
>>>             at org.apache.spark.rdd.RDDOperat
>>> ionScope$.withScope(RDDOperationScope.scala:151)
>>>             at org.apache.spark.rdd.RDDOperat
>>> ionScope$.withScope(RDDOperationScope.scala:112)
>>>             at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>>>             at org.apache.spark.rdd.RDD.take(RDD.scala:1326)
>>>             at org.example.classification.Log
>>> isticRegressionWithLBFGSAlgorithm.train(LogisticRegressionWi
>>> thLBFGSAlgorithm.scala:28)
>>>             at org.example.classification.Log
>>> isticRegressionWithLBFGSAlgorithm.train(LogisticRegressionWi
>>> thLBFGSAlgorithm.scala:21)
>>>             at org.apache.predictionio.contro
>>> ller.P2LAlgorithm.trainBase(P2LAlgorithm.scala:49)
>>>             at org.apache.predictionio.contro
>>> ller.Engine$$anonfun$18.apply(Engine.scala:692)
>>>             at org.apache.predictionio.contro
>>> ller.Engine$$anonfun$18.apply(Engine.scala:692)
>>>             at scala.collection.TraversableLi
>>> ke$$anonfun$map$1.apply(TraversableLike.scala:234)
>>>             at scala.collection.TraversableLi
>>> ke$$anonfun$map$1.apply(TraversableLike.scala:234)
>>>             at scala.collection.immutable.List.foreach(List.scala:381)
>>>             at scala.collection.TraversableLi
>>> ke$class.map(TraversableLike.scala:234)
>>>             at scala.collection.immutable.List.map(List.scala:285)
>>>             at org.apache.predictionio.contro
>>> ller.Engine$.train(Engine.scala:692)
>>>             at org.apache.predictionio.contro
>>> ller.Engine.train(Engine.scala:177)
>>>             at org.apache.predictionio.workfl
>>> ow.CoreWorkflow$.runTrain(CoreWorkflow.scala:67)
>>>             at org.apache.predictionio.workfl
>>> ow.CreateWorkflow$.main(CreateWorkflow.scala:250)
>>>             at org.apache.predictionio.workfl
>>> ow.CreateWorkflow.main(CreateWorkflow.scala)
>>>             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>>             at sun.reflect.NativeMethodAccess
>>> orImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>             at sun.reflect.DelegatingMethodAc
>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>             at java.lang.reflect.Method.invoke(Method.java:498)
>>>             at org.apache.spark.deploy.SparkS
>>> ubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSub
>>> mit.scala:738)
>>>             at org.apache.spark.deploy.SparkS
>>> ubmit$.doRunMain$1(SparkSubmit.scala:187)
>>>             at org.apache.spark.deploy.SparkS
>>> ubmit$.submit(SparkSubmit.scala:212)
>>>             at org.apache.spark.deploy.SparkS
>>> ubmit$.main(SparkSubmit.scala:126)
>>>             at org.apache.spark.deploy.SparkS
>>> ubmit.main(SparkSubmit.scala)
>>>
>>> 2. I started spark standalone cluster with 1 master and 3 workers and
>>> executed the command
>>>
>>> > pio train -- --master spark://*.*.*.*:7077 --driver-memory 50G
>>> > --executor-memory 50G
>>>
>>> And after some times getting the error . Executor failed to connect with
>>> master and training gets stopped.
>>>
>>> I have changed the feature count from 6500 - > 500 and still the
>>> condition is same. So can anyone suggest me am I missing something
>>>
>>> and In between training getting continuous warnings like :
>>> [
>>>
>>> > WARN] [ScannerCallable] Ignore, probably already closed
>>>
>>>
>>> Regards,
>>> Abhimanyu
>>>
>>>
>>
>

Mime
View raw message