spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ash (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1895) Run tests on windows
Date Mon, 15 Sep 2014 17:24:35 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Ash updated SPARK-1895:
------------------------------
    Description: 
    bin\pyspark python\pyspark\rdd.py

Sometimes tests complete without error >_<.
Last tests fail log:

{noformat}
....
14/05/21 18:31:40 INFO Executor: Running task ID 321
14/05/21 18:31:40 INFO Executor: Running task ID 324
14/05/21 18:31:40 INFO Executor: Running task ID 322
14/05/21 18:31:40 INFO Executor: Running task ID 323
14/05/21 18:31:40 INFO PythonRDD: Times: total = 241, boot = 240, init = 1, finish = 0
14/05/21 18:31:40 INFO Executor: Serialized size of result for 324 is 607
14/05/21 18:31:40 INFO Executor: Sending result for 324 directly to driver
14/05/21 18:31:40 INFO Executor: Finished task ID 324
14/05/21 18:31:40 INFO TaskSetManager: Finished TID 324 in 248 ms on localhost (progress: 1/4)
14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 3)
14/05/21 18:31:40 INFO PythonRDD: Times: total = 518, boot = 516, init = 2, finish = 0
14/05/21 18:31:40 INFO Executor: Serialized size of result for 323 is 607
14/05/21 18:31:40 INFO Executor: Sending result for 323 directly to driver
14/05/21 18:31:40 INFO Executor: Finished task ID 323
14/05/21 18:31:40 INFO TaskSetManager: Finished TID 323 in 528 ms on localhost (progress: 2/4)
14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 2)
14/05/21 18:31:41 INFO PythonRDD: Times: total = 776, boot = 774, init = 2, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 322 is 607
14/05/21 18:31:41 INFO Executor: Sending result for 322 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 322
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 322 in 785 ms on localhost (progress: 3/4)
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 1)
14/05/21 18:31:41 INFO PythonRDD: Times: total = 1043, boot = 1042, init = 1, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 321 is 607
14/05/21 18:31:41 INFO Executor: Sending result for 321 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 321
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 321 in 1049 ms on localhost (progress: 4/4)
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 0)
14/05/21 18:31:41 INFO TaskSchedulerImpl: Removed TaskSet 80.0, whose tasks have all completed, from pool
14/05/21 18:31:41 INFO DAGScheduler: Stage 80 (top at <doctest __main__.RDD.top[0]>:1) finished in 1,051 s
14/05/21 18:31:41 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[0]>:1, took 1.053832912 s
14/05/21 18:31:41 INFO SparkContext: Starting job: top at <doctest __main__.RDD.top[1]>:1
14/05/21 18:31:41 INFO DAGScheduler: Got job 63 (top at <doctest __main__.RDD.top[1]>:1) with 4 output partitions (allowLocal=false)
14/05/21 18:31:41 INFO DAGScheduler: Final stage: Stage 81 (top at <doctest __main__.RDD.top[1]>:1)
14/05/21 18:31:41 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:41 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:41 INFO DAGScheduler: Submitting Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1), which has no missing parents
14/05/21 18:31:41 INFO DAGScheduler: Submitting 4 missing tasks from Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1)
14/05/21 18:31:41 INFO TaskSchedulerImpl: Adding task set 81.0 with 4 tasks
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:0 as TID 325 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:0 as 2594 bytes in 0 ms
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:1 as TID 326 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:1 as 2594 bytes in 0 ms
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:2 as TID 327 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:2 as 2594 bytes in 0 ms
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:3 as TID 328 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:3 as 2609 bytes in 1 ms
14/05/21 18:31:41 INFO Executor: Running task ID 326
14/05/21 18:31:41 INFO Executor: Running task ID 328
14/05/21 18:31:41 INFO Executor: Running task ID 327
14/05/21 18:31:41 INFO Executor: Running task ID 325
14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_3 not found, computing it
14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(152) called with curMem=1120, maxMem=311387750
14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_3 stored as values to memory (estimated size 152.0 B, free 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_3 in memory on stribog-pc:37187 (size: 152.0 B, free: 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_3
14/05/21 18:31:41 INFO PythonRDD: Times: total = 245, boot = 241, init = 4, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 328 is 610
14/05/21 18:31:41 INFO Executor: Sending result for 328 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 328
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 328 in 251 ms on localhost (progress: 1/4)
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 3)
14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_0 not found, computing it
14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1272, maxMem=311387750
14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_0 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_0 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_0
14/05/21 18:31:41 INFO PythonRDD: Times: total = 484, boot = 481, init = 3, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 325 is 607
14/05/21 18:31:41 INFO Executor: Sending result for 325 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 325
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 0)
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 325 in 493 ms on localhost (progress: 2/4)
14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_2 not found, computing it
14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1400, maxMem=311387750
14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_2 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_2 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_2
14/05/21 18:31:42 INFO PythonRDD: Times: total = 728, boot = 724, init = 4, finish = 0
14/05/21 18:31:42 INFO Executor: Serialized size of result for 327 is 607
14/05/21 18:31:42 INFO Executor: Sending result for 327 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 327
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 327 in 735 ms on localhost (progress: 3/4)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 2)
14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_1 not found, computing it
14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1528, maxMem=311387750
14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_1 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_1 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_1
14/05/21 18:31:42 INFO PythonRDD: Times: total = 1487, boot = 1484, init = 3, finish = 0
14/05/21 18:31:42 INFO Executor: Serialized size of result for 326 is 607
14/05/21 18:31:42 INFO Executor: Sending result for 326 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 326
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 326 in 1493 ms on localhost (progress: 4/4)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 81.0, whose tasks have all completed, from pool
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 1)
14/05/21 18:31:42 INFO DAGScheduler: Stage 81 (top at <doctest __main__.RDD.top[1]>:1) finished in 1,496 s
14/05/21 18:31:42 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[1]>:1, took 1.498387373 s
14/05/21 18:31:42 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.union[1]>:1
14/05/21 18:31:42 INFO DAGScheduler: Got job 64 (collect at <doctest __main__.RDD.union[1]>:1) with 8 output partitions (allowLocal=false)
14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1)
14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 82 (UnionRDD[215] at union at null:-2), which has no missing parents
14/05/21 18:31:42 INFO DAGScheduler: Submitting 8 missing tasks from Stage 82 (UnionRDD[215] at union at null:-2)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 82.0 with 8 tasks
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:0 as TID 329 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:0 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:1 as TID 330 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:1 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:2 as TID 331 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:2 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:3 as TID 332 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:3 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 330
14/05/21 18:31:42 INFO Executor: Running task ID 331
14/05/21 18:31:42 INFO Executor: Running task ID 332
14/05/21 18:31:42 INFO Executor: Serialized size of result for 331 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 331 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 331
14/05/21 18:31:42 INFO Executor: Serialized size of result for 332 is 537
14/05/21 18:31:42 INFO Executor: Running task ID 329
14/05/21 18:31:42 INFO Executor: Sending result for 332 directly to driver
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:4 as TID 333 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 330 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 330 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 330
14/05/21 18:31:42 INFO Executor: Finished task ID 332
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:4 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 333
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 331 in 3 ms on localhost (progress: 1/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 2)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 332 in 2 ms on localhost (progress: 2/8)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 329 is 537
14/05/21 18:31:42 INFO Executor: Serialized size of result for 333 is 537
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 3)
14/05/21 18:31:42 INFO Executor: Sending result for 333 directly to driver
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:5 as TID 334 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO Executor: Sending result for 329 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 333
14/05/21 18:31:42 INFO Executor: Finished task ID 329
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:5 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 334
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:6 as TID 335 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:6 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:7 as TID 336 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO Executor: Running task ID 335
14/05/21 18:31:42 INFO Executor: Serialized size of result for 334 is 537
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:7 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Sending result for 334 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 334
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 1)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 330 in 5 ms on localhost (progress: 3/8)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 335 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 335 directly to driver
14/05/21 18:31:42 INFO Executor: Running task ID 336
14/05/21 18:31:42 INFO Executor: Finished task ID 335
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 5)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 334 in 2 ms on localhost (progress: 4/8)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 333 in 5 ms on localhost (progress: 5/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 4)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 336 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 336 directly to driver
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 335 in 3 ms on localhost (progress: 6/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 6)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 329 in 7 ms on localhost (progress: 7/8)
14/05/21 18:31:42 INFO Executor: Finished task ID 336
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 336 in 3 ms on localhost (progress: 8/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 0)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 82.0, whose tasks have all completed, from pool
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 7)
14/05/21 18:31:42 INFO DAGScheduler: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1) finished in 0,008 s
14/05/21 18:31:42 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.union[1]>:1, took 0.010936769 s
14/05/21 18:31:42 INFO SparkContext: Starting job: variance at <doctest __main__.RDD.variance[0]>:1
14/05/21 18:31:42 INFO DAGScheduler: Got job 65 (variance at <doctest __main__.RDD.variance[0]>:1) with 4 output partitions (allowLocal=false)
14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1)
14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1), which has no missing parents
14/05/21 18:31:42 INFO DAGScheduler: Submitting 4 missing tasks from Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 83.0 with 4 tasks
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:0 as TID 337 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:0 as 2486 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:1 as TID 338 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:1 as 2514 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:2 as TID 339 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:2 as 2514 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:3 as TID 340 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:3 as 2514 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 338
14/05/21 18:31:42 INFO Executor: Running task ID 340
14/05/21 18:31:42 INFO Executor: Running task ID 339
14/05/21 18:31:42 INFO Executor: Running task ID 337
14/05/21 18:31:43 INFO PythonRDD: Times: total = 265, boot = 264, init = 1, finish = 0
14/05/21 18:31:43 INFO Executor: Serialized size of result for 340 is 682
14/05/21 18:31:43 INFO Executor: Sending result for 340 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 340
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 340 in 278 ms on localhost (progress: 1/4)
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 3)
14/05/21 18:31:43 INFO PythonRDD: Times: total = 519, boot = 517, init = 2, finish = 0
14/05/21 18:31:43 INFO Executor: Serialized size of result for 337 is 681
14/05/21 18:31:43 INFO Executor: Sending result for 337 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 337
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 337 in 527 ms on localhost (progress: 2/4)
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 0)
14/05/21 18:31:43 INFO PythonRDD: Times: total = 779, boot = 778, init = 1, finish = 0
14/05/21 18:31:43 INFO Executor: Serialized size of result for 338 is 682
14/05/21 18:31:43 INFO Executor: Sending result for 338 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 338
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 1)
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 338 in 786 ms on localhost (progress: 3/4)
14/05/21 18:31:43 INFO PythonRDD: Times: total = 1025, boot = 1023, init = 1, finish = 1
14/05/21 18:31:43 INFO Executor: Serialized size of result for 339 is 682
14/05/21 18:31:43 INFO Executor: Sending result for 339 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 339
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 2)
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 339 in 1030 ms on localhost (progress: 4/4)
14/05/21 18:31:43 INFO TaskSchedulerImpl: Removed TaskSet 83.0, whose tasks have all completed, from pool
14/05/21 18:31:43 INFO DAGScheduler: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1) finished in 1,032 s
14/05/21 18:31:43 INFO SparkContext: Job finished: variance at <doctest __main__.RDD.variance[0]>:1, took 1.034616593 s
14/05/21 18:31:44 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.zip[2]>:1
14/05/21 18:31:44 INFO DAGScheduler: Got job 66 (collect at <doctest __main__.RDD.zip[2]>:1) with 4 output partitions (allowLocal=false)
14/05/21 18:31:44 INFO DAGScheduler: Final stage: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1)
14/05/21 18:31:44 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:44 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:44 INFO DAGScheduler: Submitting Stage 84 (ZippedRDD[220] at zip at null:-2), which has no missing parents
14/05/21 18:31:44 INFO DAGScheduler: Submitting 4 missing tasks from Stage 84 (ZippedRDD[220] at zip at null:-2)
14/05/21 18:31:44 INFO TaskSchedulerImpl: Adding task set 84.0 with 4 tasks
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:0 as TID 341 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:0 as 1581 bytes in 1 ms
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:1 as TID 342 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:1 as 1581 bytes in 0 ms
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:2 as TID 343 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:2 as 1581 bytes in 0 ms
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:3 as TID 344 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:3 as 1612 bytes in 0 ms
14/05/21 18:31:44 INFO Executor: Running task ID 341
14/05/21 18:31:44 INFO Executor: Running task ID 343
14/05/21 18:31:44 INFO Executor: Running task ID 344
14/05/21 18:31:44 INFO Executor: Running task ID 342
14/05/21 18:31:44 INFO Executor: Serialized size of result for 343 is 630
14/05/21 18:31:44 INFO Executor: Serialized size of result for 344 is 667
14/05/21 18:31:44 INFO Executor: Serialized size of result for 341 is 630
14/05/21 18:31:44 INFO Executor: Sending result for 344 directly to driver
14/05/21 18:31:44 INFO Executor: Sending result for 343 directly to driver
14/05/21 18:31:44 INFO Executor: Finished task ID 343
14/05/21 18:31:44 INFO Executor: Finished task ID 344
14/05/21 18:31:44 INFO Executor: Sending result for 341 directly to driver
14/05/21 18:31:44 INFO Executor: Serialized size of result for 342 is 630
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 344 in 3 ms on localhost (progress: 1/4)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 3)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 2)
14/05/21 18:31:44 INFO Executor: Finished task ID 341
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 343 in 4 ms on localhost (progress: 2/4)
14/05/21 18:31:44 INFO Executor: Sending result for 342 directly to driver
14/05/21 18:31:44 INFO Executor: Finished task ID 342
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 341 in 5 ms on localhost (progress: 3/4)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 0)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 1)
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 342 in 5 ms on localhost (progress: 4/4)
14/05/21 18:31:44 INFO DAGScheduler: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1) finished in 0,006 s
14/05/21 18:31:44 INFO TaskSchedulerImpl: Removed TaskSet 84.0, whose tasks have all completed, from pool
14/05/21 18:31:44 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.zip[2]>:1, took 0.010312512 s
14/05/21 18:31:45 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
14/05/21 18:31:45 INFO ConnectionManager: Selector thread was interrupted!
14/05/21 18:31:45 INFO ConnectionManager: ConnectionManager stopped
14/05/21 18:31:45 INFO MemoryStore: MemoryStore cleared
14/05/21 18:31:45 INFO BlockManager: BlockManager stopped
14/05/21 18:31:45 INFO BlockManagerMasterActor: Stopping BlockManagerMaster
14/05/21 18:31:45 INFO BlockManagerMaster: BlockManagerMaster stopped
14/05/21 18:31:45 INFO SparkContext: Successfully stopped SparkContext
14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
14/05/21 18:31:45 INFO Remoting: Remoting shut down
14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
Exception in thread "delete Spark temp dir C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6a2ced75"
java.io.IOException: Failed to delete: C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6
a2ced75\tmpbdykfu
        at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:483)
        at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:479)
        at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:478)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
        at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:478)
        at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:212)
{noformat}

  was:

    bin\pyspark python\pyspark\rdd.py

Sometimes tests complete without error >_<.
Last tests fail log:

....
14/05/21 18:31:40 INFO Executor: Running task ID 321
14/05/21 18:31:40 INFO Executor: Running task ID 324
14/05/21 18:31:40 INFO Executor: Running task ID 322
14/05/21 18:31:40 INFO Executor: Running task ID 323
14/05/21 18:31:40 INFO PythonRDD: Times: total = 241, boot = 240, init = 1, finish = 0
14/05/21 18:31:40 INFO Executor: Serialized size of result for 324 is 607
14/05/21 18:31:40 INFO Executor: Sending result for 324 directly to driver
14/05/21 18:31:40 INFO Executor: Finished task ID 324
14/05/21 18:31:40 INFO TaskSetManager: Finished TID 324 in 248 ms on localhost (progress: 1/4)
14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 3)
14/05/21 18:31:40 INFO PythonRDD: Times: total = 518, boot = 516, init = 2, finish = 0
14/05/21 18:31:40 INFO Executor: Serialized size of result for 323 is 607
14/05/21 18:31:40 INFO Executor: Sending result for 323 directly to driver
14/05/21 18:31:40 INFO Executor: Finished task ID 323
14/05/21 18:31:40 INFO TaskSetManager: Finished TID 323 in 528 ms on localhost (progress: 2/4)
14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 2)
14/05/21 18:31:41 INFO PythonRDD: Times: total = 776, boot = 774, init = 2, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 322 is 607
14/05/21 18:31:41 INFO Executor: Sending result for 322 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 322
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 322 in 785 ms on localhost (progress: 3/4)
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 1)
14/05/21 18:31:41 INFO PythonRDD: Times: total = 1043, boot = 1042, init = 1, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 321 is 607
14/05/21 18:31:41 INFO Executor: Sending result for 321 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 321
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 321 in 1049 ms on localhost (progress: 4/4)
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 0)
14/05/21 18:31:41 INFO TaskSchedulerImpl: Removed TaskSet 80.0, whose tasks have all completed, from pool
14/05/21 18:31:41 INFO DAGScheduler: Stage 80 (top at <doctest __main__.RDD.top[0]>:1) finished in 1,051 s
14/05/21 18:31:41 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[0]>:1, took 1.053832912 s
14/05/21 18:31:41 INFO SparkContext: Starting job: top at <doctest __main__.RDD.top[1]>:1
14/05/21 18:31:41 INFO DAGScheduler: Got job 63 (top at <doctest __main__.RDD.top[1]>:1) with 4 output partitions (allowLocal=false)
14/05/21 18:31:41 INFO DAGScheduler: Final stage: Stage 81 (top at <doctest __main__.RDD.top[1]>:1)
14/05/21 18:31:41 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:41 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:41 INFO DAGScheduler: Submitting Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1), which has no missing parents
14/05/21 18:31:41 INFO DAGScheduler: Submitting 4 missing tasks from Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1)
14/05/21 18:31:41 INFO TaskSchedulerImpl: Adding task set 81.0 with 4 tasks
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:0 as TID 325 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:0 as 2594 bytes in 0 ms
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:1 as TID 326 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:1 as 2594 bytes in 0 ms
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:2 as TID 327 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:2 as 2594 bytes in 0 ms
14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:3 as TID 328 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:3 as 2609 bytes in 1 ms
14/05/21 18:31:41 INFO Executor: Running task ID 326
14/05/21 18:31:41 INFO Executor: Running task ID 328
14/05/21 18:31:41 INFO Executor: Running task ID 327
14/05/21 18:31:41 INFO Executor: Running task ID 325
14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_3 not found, computing it
14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(152) called with curMem=1120, maxMem=311387750
14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_3 stored as values to memory (estimated size 152.0 B, free 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_3 in memory on stribog-pc:37187 (size: 152.0 B, free: 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_3
14/05/21 18:31:41 INFO PythonRDD: Times: total = 245, boot = 241, init = 4, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 328 is 610
14/05/21 18:31:41 INFO Executor: Sending result for 328 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 328
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 328 in 251 ms on localhost (progress: 1/4)
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 3)
14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_0 not found, computing it
14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1272, maxMem=311387750
14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_0 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_0 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_0
14/05/21 18:31:41 INFO PythonRDD: Times: total = 484, boot = 481, init = 3, finish = 0
14/05/21 18:31:41 INFO Executor: Serialized size of result for 325 is 607
14/05/21 18:31:41 INFO Executor: Sending result for 325 directly to driver
14/05/21 18:31:41 INFO Executor: Finished task ID 325
14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 0)
14/05/21 18:31:41 INFO TaskSetManager: Finished TID 325 in 493 ms on localhost (progress: 2/4)
14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_2 not found, computing it
14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1400, maxMem=311387750
14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_2 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_2 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_2
14/05/21 18:31:42 INFO PythonRDD: Times: total = 728, boot = 724, init = 4, finish = 0
14/05/21 18:31:42 INFO Executor: Serialized size of result for 327 is 607
14/05/21 18:31:42 INFO Executor: Sending result for 327 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 327
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 327 in 735 ms on localhost (progress: 3/4)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 2)
14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_1 not found, computing it
14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1528, maxMem=311387750
14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_1 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_1 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_1
14/05/21 18:31:42 INFO PythonRDD: Times: total = 1487, boot = 1484, init = 3, finish = 0
14/05/21 18:31:42 INFO Executor: Serialized size of result for 326 is 607
14/05/21 18:31:42 INFO Executor: Sending result for 326 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 326
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 326 in 1493 ms on localhost (progress: 4/4)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 81.0, whose tasks have all completed, from pool
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 1)
14/05/21 18:31:42 INFO DAGScheduler: Stage 81 (top at <doctest __main__.RDD.top[1]>:1) finished in 1,496 s
14/05/21 18:31:42 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[1]>:1, took 1.498387373 s
14/05/21 18:31:42 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.union[1]>:1
14/05/21 18:31:42 INFO DAGScheduler: Got job 64 (collect at <doctest __main__.RDD.union[1]>:1) with 8 output partitions (allowLocal=false)
14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1)
14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 82 (UnionRDD[215] at union at null:-2), which has no missing parents
14/05/21 18:31:42 INFO DAGScheduler: Submitting 8 missing tasks from Stage 82 (UnionRDD[215] at union at null:-2)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 82.0 with 8 tasks
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:0 as TID 329 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:0 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:1 as TID 330 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:1 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:2 as TID 331 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:2 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:3 as TID 332 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:3 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 330
14/05/21 18:31:42 INFO Executor: Running task ID 331
14/05/21 18:31:42 INFO Executor: Running task ID 332
14/05/21 18:31:42 INFO Executor: Serialized size of result for 331 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 331 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 331
14/05/21 18:31:42 INFO Executor: Serialized size of result for 332 is 537
14/05/21 18:31:42 INFO Executor: Running task ID 329
14/05/21 18:31:42 INFO Executor: Sending result for 332 directly to driver
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:4 as TID 333 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 330 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 330 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 330
14/05/21 18:31:42 INFO Executor: Finished task ID 332
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:4 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 333
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 331 in 3 ms on localhost (progress: 1/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 2)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 332 in 2 ms on localhost (progress: 2/8)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 329 is 537
14/05/21 18:31:42 INFO Executor: Serialized size of result for 333 is 537
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 3)
14/05/21 18:31:42 INFO Executor: Sending result for 333 directly to driver
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:5 as TID 334 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO Executor: Sending result for 329 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 333
14/05/21 18:31:42 INFO Executor: Finished task ID 329
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:5 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 334
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:6 as TID 335 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:6 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:7 as TID 336 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO Executor: Running task ID 335
14/05/21 18:31:42 INFO Executor: Serialized size of result for 334 is 537
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:7 as 2208 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Sending result for 334 directly to driver
14/05/21 18:31:42 INFO Executor: Finished task ID 334
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 1)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 330 in 5 ms on localhost (progress: 3/8)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 335 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 335 directly to driver
14/05/21 18:31:42 INFO Executor: Running task ID 336
14/05/21 18:31:42 INFO Executor: Finished task ID 335
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 5)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 334 in 2 ms on localhost (progress: 4/8)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 333 in 5 ms on localhost (progress: 5/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 4)
14/05/21 18:31:42 INFO Executor: Serialized size of result for 336 is 537
14/05/21 18:31:42 INFO Executor: Sending result for 336 directly to driver
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 335 in 3 ms on localhost (progress: 6/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 6)
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 329 in 7 ms on localhost (progress: 7/8)
14/05/21 18:31:42 INFO Executor: Finished task ID 336
14/05/21 18:31:42 INFO TaskSetManager: Finished TID 336 in 3 ms on localhost (progress: 8/8)
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 0)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 82.0, whose tasks have all completed, from pool
14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 7)
14/05/21 18:31:42 INFO DAGScheduler: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1) finished in 0,008 s
14/05/21 18:31:42 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.union[1]>:1, took 0.010936769 s
14/05/21 18:31:42 INFO SparkContext: Starting job: variance at <doctest __main__.RDD.variance[0]>:1
14/05/21 18:31:42 INFO DAGScheduler: Got job 65 (variance at <doctest __main__.RDD.variance[0]>:1) with 4 output partitions (allowLocal=false)
14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1)
14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1), which has no missing parents
14/05/21 18:31:42 INFO DAGScheduler: Submitting 4 missing tasks from Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1)
14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 83.0 with 4 tasks
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:0 as TID 337 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:0 as 2486 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:1 as TID 338 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:1 as 2514 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:2 as TID 339 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:2 as 2514 bytes in 0 ms
14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:3 as TID 340 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:3 as 2514 bytes in 0 ms
14/05/21 18:31:42 INFO Executor: Running task ID 338
14/05/21 18:31:42 INFO Executor: Running task ID 340
14/05/21 18:31:42 INFO Executor: Running task ID 339
14/05/21 18:31:42 INFO Executor: Running task ID 337
14/05/21 18:31:43 INFO PythonRDD: Times: total = 265, boot = 264, init = 1, finish = 0
14/05/21 18:31:43 INFO Executor: Serialized size of result for 340 is 682
14/05/21 18:31:43 INFO Executor: Sending result for 340 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 340
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 340 in 278 ms on localhost (progress: 1/4)
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 3)
14/05/21 18:31:43 INFO PythonRDD: Times: total = 519, boot = 517, init = 2, finish = 0
14/05/21 18:31:43 INFO Executor: Serialized size of result for 337 is 681
14/05/21 18:31:43 INFO Executor: Sending result for 337 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 337
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 337 in 527 ms on localhost (progress: 2/4)
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 0)
14/05/21 18:31:43 INFO PythonRDD: Times: total = 779, boot = 778, init = 1, finish = 0
14/05/21 18:31:43 INFO Executor: Serialized size of result for 338 is 682
14/05/21 18:31:43 INFO Executor: Sending result for 338 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 338
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 1)
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 338 in 786 ms on localhost (progress: 3/4)
14/05/21 18:31:43 INFO PythonRDD: Times: total = 1025, boot = 1023, init = 1, finish = 1
14/05/21 18:31:43 INFO Executor: Serialized size of result for 339 is 682
14/05/21 18:31:43 INFO Executor: Sending result for 339 directly to driver
14/05/21 18:31:43 INFO Executor: Finished task ID 339
14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 2)
14/05/21 18:31:43 INFO TaskSetManager: Finished TID 339 in 1030 ms on localhost (progress: 4/4)
14/05/21 18:31:43 INFO TaskSchedulerImpl: Removed TaskSet 83.0, whose tasks have all completed, from pool
14/05/21 18:31:43 INFO DAGScheduler: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1) finished in 1,032 s
14/05/21 18:31:43 INFO SparkContext: Job finished: variance at <doctest __main__.RDD.variance[0]>:1, took 1.034616593 s
14/05/21 18:31:44 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.zip[2]>:1
14/05/21 18:31:44 INFO DAGScheduler: Got job 66 (collect at <doctest __main__.RDD.zip[2]>:1) with 4 output partitions (allowLocal=false)
14/05/21 18:31:44 INFO DAGScheduler: Final stage: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1)
14/05/21 18:31:44 INFO DAGScheduler: Parents of final stage: List()
14/05/21 18:31:44 INFO DAGScheduler: Missing parents: List()
14/05/21 18:31:44 INFO DAGScheduler: Submitting Stage 84 (ZippedRDD[220] at zip at null:-2), which has no missing parents
14/05/21 18:31:44 INFO DAGScheduler: Submitting 4 missing tasks from Stage 84 (ZippedRDD[220] at zip at null:-2)
14/05/21 18:31:44 INFO TaskSchedulerImpl: Adding task set 84.0 with 4 tasks
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:0 as TID 341 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:0 as 1581 bytes in 1 ms
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:1 as TID 342 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:1 as 1581 bytes in 0 ms
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:2 as TID 343 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:2 as 1581 bytes in 0 ms
14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:3 as TID 344 on executor localhost: localhost (PROCESS_LOCAL)
14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:3 as 1612 bytes in 0 ms
14/05/21 18:31:44 INFO Executor: Running task ID 341
14/05/21 18:31:44 INFO Executor: Running task ID 343
14/05/21 18:31:44 INFO Executor: Running task ID 344
14/05/21 18:31:44 INFO Executor: Running task ID 342
14/05/21 18:31:44 INFO Executor: Serialized size of result for 343 is 630
14/05/21 18:31:44 INFO Executor: Serialized size of result for 344 is 667
14/05/21 18:31:44 INFO Executor: Serialized size of result for 341 is 630
14/05/21 18:31:44 INFO Executor: Sending result for 344 directly to driver
14/05/21 18:31:44 INFO Executor: Sending result for 343 directly to driver
14/05/21 18:31:44 INFO Executor: Finished task ID 343
14/05/21 18:31:44 INFO Executor: Finished task ID 344
14/05/21 18:31:44 INFO Executor: Sending result for 341 directly to driver
14/05/21 18:31:44 INFO Executor: Serialized size of result for 342 is 630
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 344 in 3 ms on localhost (progress: 1/4)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 3)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 2)
14/05/21 18:31:44 INFO Executor: Finished task ID 341
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 343 in 4 ms on localhost (progress: 2/4)
14/05/21 18:31:44 INFO Executor: Sending result for 342 directly to driver
14/05/21 18:31:44 INFO Executor: Finished task ID 342
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 341 in 5 ms on localhost (progress: 3/4)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 0)
14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 1)
14/05/21 18:31:44 INFO TaskSetManager: Finished TID 342 in 5 ms on localhost (progress: 4/4)
14/05/21 18:31:44 INFO DAGScheduler: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1) finished in 0,006 s
14/05/21 18:31:44 INFO TaskSchedulerImpl: Removed TaskSet 84.0, whose tasks have all completed, from pool
14/05/21 18:31:44 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.zip[2]>:1, took 0.010312512 s
14/05/21 18:31:45 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
14/05/21 18:31:45 INFO ConnectionManager: Selector thread was interrupted!
14/05/21 18:31:45 INFO ConnectionManager: ConnectionManager stopped
14/05/21 18:31:45 INFO MemoryStore: MemoryStore cleared
14/05/21 18:31:45 INFO BlockManager: BlockManager stopped
14/05/21 18:31:45 INFO BlockManagerMasterActor: Stopping BlockManagerMaster
14/05/21 18:31:45 INFO BlockManagerMaster: BlockManagerMaster stopped
14/05/21 18:31:45 INFO SparkContext: Successfully stopped SparkContext
14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
14/05/21 18:31:45 INFO Remoting: Remoting shut down
14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
Exception in thread "delete Spark temp dir C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6a2ced75"
java.io.IOException: Failed to delete: C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6
a2ced75\tmpbdykfu
        at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:483)
        at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:479)
        at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:478)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
        at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:478)
        at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:212)



> Run tests on windows
> --------------------
>
>                 Key: SPARK-1895
>                 URL: https://issues.apache.org/jira/browse/SPARK-1895
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, Windows
>    Affects Versions: 0.9.1
>         Environment: spark-0.9.1-bin-hadoop1
>            Reporter: stribog
>            Priority: Trivial
>
>     bin\pyspark python\pyspark\rdd.py
> Sometimes tests complete without error >_<.
> Last tests fail log:
> {noformat}
> ....
> 14/05/21 18:31:40 INFO Executor: Running task ID 321
> 14/05/21 18:31:40 INFO Executor: Running task ID 324
> 14/05/21 18:31:40 INFO Executor: Running task ID 322
> 14/05/21 18:31:40 INFO Executor: Running task ID 323
> 14/05/21 18:31:40 INFO PythonRDD: Times: total = 241, boot = 240, init = 1, finish = 0
> 14/05/21 18:31:40 INFO Executor: Serialized size of result for 324 is 607
> 14/05/21 18:31:40 INFO Executor: Sending result for 324 directly to driver
> 14/05/21 18:31:40 INFO Executor: Finished task ID 324
> 14/05/21 18:31:40 INFO TaskSetManager: Finished TID 324 in 248 ms on localhost (progress: 1/4)
> 14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 3)
> 14/05/21 18:31:40 INFO PythonRDD: Times: total = 518, boot = 516, init = 2, finish = 0
> 14/05/21 18:31:40 INFO Executor: Serialized size of result for 323 is 607
> 14/05/21 18:31:40 INFO Executor: Sending result for 323 directly to driver
> 14/05/21 18:31:40 INFO Executor: Finished task ID 323
> 14/05/21 18:31:40 INFO TaskSetManager: Finished TID 323 in 528 ms on localhost (progress: 2/4)
> 14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 2)
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 776, boot = 774, init = 2, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 322 is 607
> 14/05/21 18:31:41 INFO Executor: Sending result for 322 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 322
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 322 in 785 ms on localhost (progress: 3/4)
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 1)
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 1043, boot = 1042, init = 1, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 321 is 607
> 14/05/21 18:31:41 INFO Executor: Sending result for 321 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 321
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 321 in 1049 ms on localhost (progress: 4/4)
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 0)
> 14/05/21 18:31:41 INFO TaskSchedulerImpl: Removed TaskSet 80.0, whose tasks have all completed, from pool
> 14/05/21 18:31:41 INFO DAGScheduler: Stage 80 (top at <doctest __main__.RDD.top[0]>:1) finished in 1,051 s
> 14/05/21 18:31:41 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[0]>:1, took 1.053832912 s
> 14/05/21 18:31:41 INFO SparkContext: Starting job: top at <doctest __main__.RDD.top[1]>:1
> 14/05/21 18:31:41 INFO DAGScheduler: Got job 63 (top at <doctest __main__.RDD.top[1]>:1) with 4 output partitions (allowLocal=false)
> 14/05/21 18:31:41 INFO DAGScheduler: Final stage: Stage 81 (top at <doctest __main__.RDD.top[1]>:1)
> 14/05/21 18:31:41 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:41 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:41 INFO DAGScheduler: Submitting Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1), which has no missing parents
> 14/05/21 18:31:41 INFO DAGScheduler: Submitting 4 missing tasks from Stage 81 (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1)
> 14/05/21 18:31:41 INFO TaskSchedulerImpl: Adding task set 81.0 with 4 tasks
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:0 as TID 325 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:0 as 2594 bytes in 0 ms
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:1 as TID 326 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:1 as 2594 bytes in 0 ms
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:2 as TID 327 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:2 as 2594 bytes in 0 ms
> 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:3 as TID 328 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:3 as 2609 bytes in 1 ms
> 14/05/21 18:31:41 INFO Executor: Running task ID 326
> 14/05/21 18:31:41 INFO Executor: Running task ID 328
> 14/05/21 18:31:41 INFO Executor: Running task ID 327
> 14/05/21 18:31:41 INFO Executor: Running task ID 325
> 14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_3 not found, computing it
> 14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(152) called with curMem=1120, maxMem=311387750
> 14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_3 stored as values to memory (estimated size 152.0 B, free 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_3 in memory on stribog-pc:37187 (size: 152.0 B, free: 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_3
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 245, boot = 241, init = 4, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 328 is 610
> 14/05/21 18:31:41 INFO Executor: Sending result for 328 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 328
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 328 in 251 ms on localhost (progress: 1/4)
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 3)
> 14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_0 not found, computing it
> 14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1272, maxMem=311387750
> 14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_0 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_0 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
> 14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_0
> 14/05/21 18:31:41 INFO PythonRDD: Times: total = 484, boot = 481, init = 3, finish = 0
> 14/05/21 18:31:41 INFO Executor: Serialized size of result for 325 is 607
> 14/05/21 18:31:41 INFO Executor: Sending result for 325 directly to driver
> 14/05/21 18:31:41 INFO Executor: Finished task ID 325
> 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 0)
> 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 325 in 493 ms on localhost (progress: 2/4)
> 14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_2 not found, computing it
> 14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1400, maxMem=311387750
> 14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_2 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_2 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_2
> 14/05/21 18:31:42 INFO PythonRDD: Times: total = 728, boot = 724, init = 4, finish = 0
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 327 is 607
> 14/05/21 18:31:42 INFO Executor: Sending result for 327 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 327
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 327 in 735 ms on localhost (progress: 3/4)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 2)
> 14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_1 not found, computing it
> 14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with curMem=1528, maxMem=311387750
> 14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_1 stored as values to memory (estimated size 128.0 B, free 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added rdd_212_1 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB)
> 14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_1
> 14/05/21 18:31:42 INFO PythonRDD: Times: total = 1487, boot = 1484, init = 3, finish = 0
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 326 is 607
> 14/05/21 18:31:42 INFO Executor: Sending result for 326 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 326
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 326 in 1493 ms on localhost (progress: 4/4)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 81.0, whose tasks have all completed, from pool
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 1)
> 14/05/21 18:31:42 INFO DAGScheduler: Stage 81 (top at <doctest __main__.RDD.top[1]>:1) finished in 1,496 s
> 14/05/21 18:31:42 INFO SparkContext: Job finished: top at <doctest __main__.RDD.top[1]>:1, took 1.498387373 s
> 14/05/21 18:31:42 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.union[1]>:1
> 14/05/21 18:31:42 INFO DAGScheduler: Got job 64 (collect at <doctest __main__.RDD.union[1]>:1) with 8 output partitions (allowLocal=false)
> 14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1)
> 14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 82 (UnionRDD[215] at union at null:-2), which has no missing parents
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting 8 missing tasks from Stage 82 (UnionRDD[215] at union at null:-2)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 82.0 with 8 tasks
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:0 as TID 329 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:0 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:1 as TID 330 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:1 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:2 as TID 331 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:2 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:3 as TID 332 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:3 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 330
> 14/05/21 18:31:42 INFO Executor: Running task ID 331
> 14/05/21 18:31:42 INFO Executor: Running task ID 332
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 331 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 331 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 331
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 332 is 537
> 14/05/21 18:31:42 INFO Executor: Running task ID 329
> 14/05/21 18:31:42 INFO Executor: Sending result for 332 directly to driver
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:4 as TID 333 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 330 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 330 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 330
> 14/05/21 18:31:42 INFO Executor: Finished task ID 332
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:4 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 333
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 331 in 3 ms on localhost (progress: 1/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 2)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 332 in 2 ms on localhost (progress: 2/8)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 329 is 537
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 333 is 537
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 3)
> 14/05/21 18:31:42 INFO Executor: Sending result for 333 directly to driver
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:5 as TID 334 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO Executor: Sending result for 329 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 333
> 14/05/21 18:31:42 INFO Executor: Finished task ID 329
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:5 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 334
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:6 as TID 335 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:6 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:7 as TID 336 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO Executor: Running task ID 335
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 334 is 537
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:7 as 2208 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Sending result for 334 directly to driver
> 14/05/21 18:31:42 INFO Executor: Finished task ID 334
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 1)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 330 in 5 ms on localhost (progress: 3/8)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 335 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 335 directly to driver
> 14/05/21 18:31:42 INFO Executor: Running task ID 336
> 14/05/21 18:31:42 INFO Executor: Finished task ID 335
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 5)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 334 in 2 ms on localhost (progress: 4/8)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 333 in 5 ms on localhost (progress: 5/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 4)
> 14/05/21 18:31:42 INFO Executor: Serialized size of result for 336 is 537
> 14/05/21 18:31:42 INFO Executor: Sending result for 336 directly to driver
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 335 in 3 ms on localhost (progress: 6/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 6)
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 329 in 7 ms on localhost (progress: 7/8)
> 14/05/21 18:31:42 INFO Executor: Finished task ID 336
> 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 336 in 3 ms on localhost (progress: 8/8)
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 0)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 82.0, whose tasks have all completed, from pool
> 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 7)
> 14/05/21 18:31:42 INFO DAGScheduler: Stage 82 (collect at <doctest __main__.RDD.union[1]>:1) finished in 0,008 s
> 14/05/21 18:31:42 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.union[1]>:1, took 0.010936769 s
> 14/05/21 18:31:42 INFO SparkContext: Starting job: variance at <doctest __main__.RDD.variance[0]>:1
> 14/05/21 18:31:42 INFO DAGScheduler: Got job 65 (variance at <doctest __main__.RDD.variance[0]>:1) with 4 output partitions (allowLocal=false)
> 14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1)
> 14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1), which has no missing parents
> 14/05/21 18:31:42 INFO DAGScheduler: Submitting 4 missing tasks from Stage 83 (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1)
> 14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 83.0 with 4 tasks
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:0 as TID 337 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:0 as 2486 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:1 as TID 338 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:1 as 2514 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:2 as TID 339 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:2 as 2514 bytes in 0 ms
> 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:3 as TID 340 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:3 as 2514 bytes in 0 ms
> 14/05/21 18:31:42 INFO Executor: Running task ID 338
> 14/05/21 18:31:42 INFO Executor: Running task ID 340
> 14/05/21 18:31:42 INFO Executor: Running task ID 339
> 14/05/21 18:31:42 INFO Executor: Running task ID 337
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 265, boot = 264, init = 1, finish = 0
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 340 is 682
> 14/05/21 18:31:43 INFO Executor: Sending result for 340 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 340
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 340 in 278 ms on localhost (progress: 1/4)
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 3)
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 519, boot = 517, init = 2, finish = 0
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 337 is 681
> 14/05/21 18:31:43 INFO Executor: Sending result for 337 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 337
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 337 in 527 ms on localhost (progress: 2/4)
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 0)
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 779, boot = 778, init = 1, finish = 0
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 338 is 682
> 14/05/21 18:31:43 INFO Executor: Sending result for 338 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 338
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 1)
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 338 in 786 ms on localhost (progress: 3/4)
> 14/05/21 18:31:43 INFO PythonRDD: Times: total = 1025, boot = 1023, init = 1, finish = 1
> 14/05/21 18:31:43 INFO Executor: Serialized size of result for 339 is 682
> 14/05/21 18:31:43 INFO Executor: Sending result for 339 directly to driver
> 14/05/21 18:31:43 INFO Executor: Finished task ID 339
> 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 2)
> 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 339 in 1030 ms on localhost (progress: 4/4)
> 14/05/21 18:31:43 INFO TaskSchedulerImpl: Removed TaskSet 83.0, whose tasks have all completed, from pool
> 14/05/21 18:31:43 INFO DAGScheduler: Stage 83 (variance at <doctest __main__.RDD.variance[0]>:1) finished in 1,032 s
> 14/05/21 18:31:43 INFO SparkContext: Job finished: variance at <doctest __main__.RDD.variance[0]>:1, took 1.034616593 s
> 14/05/21 18:31:44 INFO SparkContext: Starting job: collect at <doctest __main__.RDD.zip[2]>:1
> 14/05/21 18:31:44 INFO DAGScheduler: Got job 66 (collect at <doctest __main__.RDD.zip[2]>:1) with 4 output partitions (allowLocal=false)
> 14/05/21 18:31:44 INFO DAGScheduler: Final stage: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1)
> 14/05/21 18:31:44 INFO DAGScheduler: Parents of final stage: List()
> 14/05/21 18:31:44 INFO DAGScheduler: Missing parents: List()
> 14/05/21 18:31:44 INFO DAGScheduler: Submitting Stage 84 (ZippedRDD[220] at zip at null:-2), which has no missing parents
> 14/05/21 18:31:44 INFO DAGScheduler: Submitting 4 missing tasks from Stage 84 (ZippedRDD[220] at zip at null:-2)
> 14/05/21 18:31:44 INFO TaskSchedulerImpl: Adding task set 84.0 with 4 tasks
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:0 as TID 341 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:0 as 1581 bytes in 1 ms
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:1 as TID 342 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:1 as 1581 bytes in 0 ms
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:2 as TID 343 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:2 as 1581 bytes in 0 ms
> 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:3 as TID 344 on executor localhost: localhost (PROCESS_LOCAL)
> 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:3 as 1612 bytes in 0 ms
> 14/05/21 18:31:44 INFO Executor: Running task ID 341
> 14/05/21 18:31:44 INFO Executor: Running task ID 343
> 14/05/21 18:31:44 INFO Executor: Running task ID 344
> 14/05/21 18:31:44 INFO Executor: Running task ID 342
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 343 is 630
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 344 is 667
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 341 is 630
> 14/05/21 18:31:44 INFO Executor: Sending result for 344 directly to driver
> 14/05/21 18:31:44 INFO Executor: Sending result for 343 directly to driver
> 14/05/21 18:31:44 INFO Executor: Finished task ID 343
> 14/05/21 18:31:44 INFO Executor: Finished task ID 344
> 14/05/21 18:31:44 INFO Executor: Sending result for 341 directly to driver
> 14/05/21 18:31:44 INFO Executor: Serialized size of result for 342 is 630
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 344 in 3 ms on localhost (progress: 1/4)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 3)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 2)
> 14/05/21 18:31:44 INFO Executor: Finished task ID 341
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 343 in 4 ms on localhost (progress: 2/4)
> 14/05/21 18:31:44 INFO Executor: Sending result for 342 directly to driver
> 14/05/21 18:31:44 INFO Executor: Finished task ID 342
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 341 in 5 ms on localhost (progress: 3/4)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 0)
> 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 1)
> 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 342 in 5 ms on localhost (progress: 4/4)
> 14/05/21 18:31:44 INFO DAGScheduler: Stage 84 (collect at <doctest __main__.RDD.zip[2]>:1) finished in 0,006 s
> 14/05/21 18:31:44 INFO TaskSchedulerImpl: Removed TaskSet 84.0, whose tasks have all completed, from pool
> 14/05/21 18:31:44 INFO SparkContext: Job finished: collect at <doctest __main__.RDD.zip[2]>:1, took 0.010312512 s
> 14/05/21 18:31:45 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
> 14/05/21 18:31:45 INFO ConnectionManager: Selector thread was interrupted!
> 14/05/21 18:31:45 INFO ConnectionManager: ConnectionManager stopped
> 14/05/21 18:31:45 INFO MemoryStore: MemoryStore cleared
> 14/05/21 18:31:45 INFO BlockManager: BlockManager stopped
> 14/05/21 18:31:45 INFO BlockManagerMasterActor: Stopping BlockManagerMaster
> 14/05/21 18:31:45 INFO BlockManagerMaster: BlockManagerMaster stopped
> 14/05/21 18:31:45 INFO SparkContext: Successfully stopped SparkContext
> 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
> 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
> 14/05/21 18:31:45 INFO Remoting: Remoting shut down
> 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
> Exception in thread "delete Spark temp dir C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6a2ced75"
> java.io.IOException: Failed to delete: C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6
> a2ced75\tmpbdykfu
>         at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:483)
>         at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:479)
>         at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:478)
>         at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>         at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
>         at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:478)
>         at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:212)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message