spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Asked to remove non-existent executor exception
Date Sun, 26 Jul 2015 22:36:49 GMT
If I read the code correctly, that error message came
from CoarseGrainedSchedulerBackend.

There may be existing / future error messages, other than the one cited
below, which are useful. Maybe change the log level of this message  to
DEBUG ?

Cheers

On Sun, Jul 26, 2015 at 3:28 PM, Mridul Muralidharan <mridul@gmail.com>
wrote:

> Simply customize your log4j confit instead of modifying code if you don't
> want messages from that class.
>
>
> Regards
> Mridul
>
> On Sunday, July 26, 2015, Sea <261810726@qq.com> wrote:
>
>> This exception is so ugly!!!  The screen is full of these information
>> when the program runs a long time,  and they will not fail the job.
>>
>> I comment it in the source code. I think this information is useless
>> because the executor is already removed and I don't know what does the
>> executor id mean.
>>
>> Should we remove this information forever?
>>
>>
>>
>>  15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove
>> non-existent executor 2...
>>
>> 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove
>> non-existent executor 2...
>>
>>
>>
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>>  *发件人:* "Ted Yu";<yuzhihong@gmail.com>;
>> *发送时间:* 2015年7月26日(星期天) 晚上10:51
>> *收件人:* "Pa Rö"<paul.roewer1990@googlemail.com>;
>> *抄送:* "user"<user@spark.apache.org>;
>> *主题:* Re: Asked to remove non-existent executor exception
>>
>> You can list the files in tmpfs in reverse chronological order and remove
>> the oldest until you have enough space.
>>
>> Cheers
>>
>> On Sun, Jul 26, 2015 at 12:43 AM, Pa Rö <paul.roewer1990@googlemail.com>
>> wrote:
>>
>>> i has seen that the "tempfs" is full, how i can clear this?
>>>
>>> 2015-07-23 13:41 GMT+02:00 Pa Rö <paul.roewer1990@googlemail.com>:
>>>
>>>>   hello spark community,
>>>>
>>>> i have build an application with geomesa, accumulo and spark.
>>>> if it run on spark local mode, it is working, but on spark
>>>> cluster not. in short it says: No space left on device. Asked to remove
>>>> non-existent executor XY.
>>>> I´m confused, because there were many GB´s of free space. do i need to
>>>> change my configuration or what else can i do? thanks in advance.
>>>>
>>>> here is the complete exception:
>>>>
>>>> og4j:WARN No appenders could be found for logger
>>>> (org.apache.accumulo.fate.zookeeper.ZooSession).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Using Spark's default log4j profile:
>>>> org/apache/spark/log4j-defaults.properties
>>>> 15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0
>>>> 15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop
>>>> library for your platform... using builtin-java classes where applicable
>>>> 15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel
>>>> 15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel
>>>> 15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication
>>>> disabled; ui acls disabled; users with view permissions: Set(marcel); users
>>>> with modify permissions: Set(marcel)
>>>> 15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started
>>>> 15/07/23 13:26:40 INFO Remoting: Starting remoting
>>>> 15/07/23 13:26:40 INFO Remoting: Remoting started; listening on
>>>> addresses :[akka.tcp://sparkDriver@node1-scads02:52478]
>>>> 15/07/23 13:26:40 INFO Utils: Successfully started service
>>>> 'sparkDriver' on port 52478.
>>>> 15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker
>>>> 15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster
>>>> 15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at
>>>> /tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b
>>>> 15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity
>>>> 1916.2 MB
>>>> 15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is
>>>> /tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2
>>>> 15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server
>>>> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT
>>>> 15/07/23 13:26:40 INFO AbstractConnector: Started
>>>> SocketConnector@0.0.0.0:56499
>>>> 15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file
>>>> server' on port 56499.
>>>> 15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator
>>>> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT
>>>> 15/07/23 13:26:40 INFO AbstractConnector: Started
>>>> SelectChannelConnector@0.0.0.0:4040
>>>> 15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on
>>>> port 4040.
>>>> 15/07/23 13:26:40 INFO SparkUI: Started SparkUI at
>>>> http://node1-scads02:4040
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master
>>>> akka.tcp://sparkMaster@node1-scads02:7077/user/Master...
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark
>>>> cluster with app ID app-20150723132640-0000
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
>>>> app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078
>>>> (node3-scads06:7078) with 8 cores
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
>>>> app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores,
>>>> 512.0 MB RAM
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
>>>> app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078
>>>> (node2-scads05:7078) with 8 cores
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
>>>> app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores,
>>>> 512.0 MB RAM
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/0 is now RUNNING
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/1 is now RUNNING
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/0 is now LOADING
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/1 is now LOADING
>>>> 15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on
>>>> 45786
>>>> 15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register
>>>> BlockManager
>>>> 15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block
>>>> manager node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>,
>>>> node1-scads02, 45786)
>>>> 15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left
>>>> on device)
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor
>>>> app-20150723132640-0000/0 removed: java.io.IOException: No space left on
>>>> device
>>>> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove
>>>> non-existent executor 0
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
>>>> app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078
>>>> (node3-scads06:7078) with 8 cores
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
>>>> app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores,
>>>> 512.0 MB RAM
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left
>>>> on device)
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor
>>>> app-20150723132640-0000/1 removed: java.io.IOException: No space left on
>>>> device
>>>> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove
>>>> non-existent executor 1
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
>>>> app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078
>>>> (node2-scads05:7078) with 8 cores
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
>>>> app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores,
>>>> 512.0 MB RAM
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/2 is now LOADING
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/3 is now LOADING
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/2 is now RUNNING
>>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is
>>>> ready for scheduling beginning after reached minRegisteredResourcesRatio:
>>>> 0.0
>>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/3 is now RUNNING
>>>> 15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated:
>>>> app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left
>>>> on device)
>>>> 15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor
>>>> app-20150723132640-0000/2 removed: java.io.IOException: No space left on
>>>> device
>>>> 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove
>>>> non-existent executor 2...
>>>>
>>>
>>>
>>

Mime
View raw message