predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhimanyu Nagrath <abhimanyunagr...@gmail.com>
Subject Re: Not able to train data
Date Thu, 26 Oct 2017 07:07:12 GMT
Hi Vaghawan ,

I am using v0.11.0-incubating with (ES - v5.2.1 , Hbase - 1.2.6 , Spark -
2.1.0).

Regards,
Abhimanyu

On Thu, Oct 26, 2017 at 12:31 PM, Vaghawan Ojha <vaghawan781@gmail.com>
wrote:

> Hi Abhimanyu,
>
> Ok, which version of pio is this? Because the template looks old to me.
>
> On Thu, Oct 26, 2017 at 12:44 PM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
>> Hi Vaghawan,
>>
>> yes, the spark master connection string is correct I am getting executor
>> fails to connect to spark master after 4-5 hrs.
>>
>>
>> Regards,
>> Abhimanyu
>>
>> On Thu, Oct 26, 2017 at 12:17 PM, Sachin Kamkar <sachinkamkar@gmail.com>
>> wrote:
>>
>>> It should be correct, as the user got the exception after 3-4 hours of
>>> starting. So looks like something else broke. OOM?
>>>
>>> With Regards,
>>>
>>>      Sachin
>>> ⚜KTBFFH⚜
>>>
>>> On Thu, Oct 26, 2017 at 12:15 PM, Vaghawan Ojha <vaghawan781@gmail.com>
>>> wrote:
>>>
>>>> "Executor failed to connect with master ", are you sure the --master
>>>> spark://*.*.*.*:7077 is correct?
>>>>
>>>> Like the one you copied from the spark master's web ui? sometimes
>>>> having that wrong fails to connect with the spark master.
>>>>
>>>> Thanks
>>>>
>>>> On Thu, Oct 26, 2017 at 12:02 PM, Abhimanyu Nagrath <
>>>> abhimanyunagrath@gmail.com> wrote:
>>>>
>>>>> I am new to predictionIO . I am using template
>>>>> https://github.com/EmergentOrder/template-scala-probabilisti
>>>>> c-classifier-batch-lbfgs.
>>>>>
>>>>> My training dataset count is 1184603 having approx 6500 features. I am
>>>>> using ec2 r4.8xlarge system (240 GB RAM, 32 Cores, 200 GB Swap).
>>>>>
>>>>>
>>>>> I tried two ways for training
>>>>>
>>>>>  1. Command '
>>>>>
>>>>> > pio train -- --driver-memory 120G --executor-memory 100G -- conf
>>>>> > spark.network.timeout=10000000
>>>>>
>>>>> '
>>>>>   Its throwing exception after 3-4 hours.
>>>>>
>>>>>
>>>>>     Exception in thread "main" org.apache.spark.SparkException: Job
>>>>> aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most
>>>>> recent failure: Lost task 0.0 in stage 1.0 (TID 15, localhost, executor
>>>>> driver): ExecutorLostFailure (executor driver exited caused by one of
the
>>>>> running tasks) Reason: Executor heartbeat timed out after 181529 ms
>>>>>     Driver stacktrace:
>>>>>             at org.apache.spark.scheduler.DAGScheduler.org
>>>>> $apache$spark$scheduler$DAGScheduler$$failJobAn
>>>>> dIndependentStages(DAGScheduler.scala:1435)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> Scheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> Scheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
>>>>>             at scala.collection.mutable.Resiz
>>>>> ableArray$class.foreach(ResizableArray.scala:59)
>>>>>             at scala.collection.mutable.Array
>>>>> Buffer.foreach(ArrayBuffer.scala:48)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> Scheduler.abortStage(DAGScheduler.scala:1422)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> Scheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> Scheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
>>>>>             at scala.Option.foreach(Option.scala:257)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> Scheduler.handleTaskSetFailed(DAGScheduler.scala:802)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> SchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> SchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> SchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
>>>>>             at org.apache.spark.util.EventLoo
>>>>> p$$anon$1.run(EventLoop.scala:48)
>>>>>             at org.apache.spark.scheduler.DAG
>>>>> Scheduler.runJob(DAGScheduler.scala:628)
>>>>>             at org.apache.spark.SparkContext.
>>>>> runJob(SparkContext.scala:1918)
>>>>>             at org.apache.spark.SparkContext.
>>>>> runJob(SparkContext.scala:1931)
>>>>>             at org.apache.spark.SparkContext.
>>>>> runJob(SparkContext.scala:1944)
>>>>>             at org.apache.spark.rdd.RDD$$anon
>>>>> fun$take$1.apply(RDD.scala:1353)
>>>>>             at org.apache.spark.rdd.RDDOperat
>>>>> ionScope$.withScope(RDDOperationScope.scala:151)
>>>>>             at org.apache.spark.rdd.RDDOperat
>>>>> ionScope$.withScope(RDDOperationScope.scala:112)
>>>>>             at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>>>>>             at org.apache.spark.rdd.RDD.take(RDD.scala:1326)
>>>>>             at org.example.classification.Log
>>>>> isticRegressionWithLBFGSAlgorithm.train(LogisticRegressionWi
>>>>> thLBFGSAlgorithm.scala:28)
>>>>>             at org.example.classification.Log
>>>>> isticRegressionWithLBFGSAlgorithm.train(LogisticRegressionWi
>>>>> thLBFGSAlgorithm.scala:21)
>>>>>             at org.apache.predictionio.contro
>>>>> ller.P2LAlgorithm.trainBase(P2LAlgorithm.scala:49)
>>>>>             at org.apache.predictionio.contro
>>>>> ller.Engine$$anonfun$18.apply(Engine.scala:692)
>>>>>             at org.apache.predictionio.contro
>>>>> ller.Engine$$anonfun$18.apply(Engine.scala:692)
>>>>>             at scala.collection.TraversableLi
>>>>> ke$$anonfun$map$1.apply(TraversableLike.scala:234)
>>>>>             at scala.collection.TraversableLi
>>>>> ke$$anonfun$map$1.apply(TraversableLike.scala:234)
>>>>>             at scala.collection.immutable.List.foreach(List.scala:381)
>>>>>             at scala.collection.TraversableLi
>>>>> ke$class.map(TraversableLike.scala:234)
>>>>>             at scala.collection.immutable.List.map(List.scala:285)
>>>>>             at org.apache.predictionio.contro
>>>>> ller.Engine$.train(Engine.scala:692)
>>>>>             at org.apache.predictionio.contro
>>>>> ller.Engine.train(Engine.scala:177)
>>>>>             at org.apache.predictionio.workfl
>>>>> ow.CoreWorkflow$.runTrain(CoreWorkflow.scala:67)
>>>>>             at org.apache.predictionio.workfl
>>>>> ow.CreateWorkflow$.main(CreateWorkflow.scala:250)
>>>>>             at org.apache.predictionio.workfl
>>>>> ow.CreateWorkflow.main(CreateWorkflow.scala)
>>>>>             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>>             at sun.reflect.NativeMethodAccess
>>>>> orImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>             at sun.reflect.DelegatingMethodAc
>>>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>             at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>             at org.apache.spark.deploy.SparkS
>>>>> ubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSub
>>>>> mit.scala:738)
>>>>>             at org.apache.spark.deploy.SparkS
>>>>> ubmit$.doRunMain$1(SparkSubmit.scala:187)
>>>>>             at org.apache.spark.deploy.SparkS
>>>>> ubmit$.submit(SparkSubmit.scala:212)
>>>>>             at org.apache.spark.deploy.SparkS
>>>>> ubmit$.main(SparkSubmit.scala:126)
>>>>>             at org.apache.spark.deploy.SparkS
>>>>> ubmit.main(SparkSubmit.scala)
>>>>>
>>>>> 2. I started spark standalone cluster with 1 master and 3 workers and
>>>>> executed the command
>>>>>
>>>>> > pio train -- --master spark://*.*.*.*:7077 --driver-memory 50G
>>>>> > --executor-memory 50G
>>>>>
>>>>> And after some times getting the error . Executor failed to connect
>>>>> with master and training gets stopped.
>>>>>
>>>>> I have changed the feature count from 6500 - > 500 and still the
>>>>> condition is same. So can anyone suggest me am I missing something
>>>>>
>>>>> and In between training getting continuous warnings like :
>>>>> [
>>>>>
>>>>> > WARN] [ScannerCallable] Ignore, probably already closed
>>>>>
>>>>>
>>>>> Regards,
>>>>> Abhimanyu
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message