Thank you Rohith for the reply and I wasn’t exactly sure what to make of those log entries.  I had assumed that from Yarn’s perspective, yes the container was setup and broken down completely.  I’ll send this to the Spark user mailing list since it does make better sense to be in that forum.

 

Regards,
Keith

 

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Wednesday, May 20, 2015 2:06 AM
To: user@hadoop.apache.org
Subject: RE: SparkPi fails with Job aborted due to stage failure: Task serialization failed:

 

Hi,

 

From the ResourceManager log, it is very clear that Job has succeeded. There is no problem running Spark Applications.

 

2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=Application Finished - Succeeded      TARGET=RMAppManager     RESULT=SUCCESS     APPID=application_1432064564266_0003

2015-05-19 19:53:20,389 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1432064564266_0003,name=Spark Pi,user=testuser,queue=default,state=FINISHED,trackingUrl=https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003/A,appMasterHost=ip-10-10-128-10.ec2.internal,startTime=1432065172758,finishTime=1432065198967,finalStatus=SUCCEEDED

 

But I do not have idea what does spark is doing internally.  Would you mind sending mail to spark user mailing lists.

 

 

Thanks & Regards

Rohith Sharma K S

 

From: Nance, Keith [mailto:knance@smartronix.com]
Sent: 20 May 2015 02:05
To: user@hadoop.apache.org
Subject: SparkPi fails with Job aborted due to stage failure: Task serialization failed:

 

All, unable to find any reference to my issue with spark.  Any ideas?  Thanks for any and all help

Attached are logs from the Spark job (SparkPi) results, Userlog, Nodemanager, and Resourcemanager.

###: SPARK JOB RESULTS :###

###########################

[testuser@ip-10-10-127-10 spark]$ ./bin/spark-submit --verbose --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 1 --executor-cores 1 lib/spark-examples*.jar 10

/home/testuser/spark/conf/spark-env.sh: line 55: -Dspark.history.kerberos.principal=spark/ip-10-10-127-10.ec2.internal@MALARD.LOCAL: No such file or directory

Using properties file: /home/testuser/spark/conf/spark-defaults.conf

Adding default property: spark.broadcast.port=8004

Adding default property: spark.executor.extraJavaOptions=-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

Adding default property: spark.akka.threads=1

Adding default property: spark.ui.port=4040

Adding default property: spark.driver.port=8001

Adding default property: spark.akka.heartbeat.interval=100

Adding default property: spark.yarn.access.namenodes=hdfs://10.10.10.10:8020

Adding default property: spark.executor.port=8002

Adding default property: spark.logConf=true

Adding default property: spark.yarn.am.extraJavaOptions=-Djava.net.preferIPv4Stack=true

Adding default property: spark.worker.ui.port=8081

Adding default property: spark.replClassServer.port=8006

Adding default property: spark.eventlog.dir=hdfs://10.10.10.10:8020/user/testuser/spark/eventlog

Adding default property: spark.driver.extraJavaOptions=-Djava.net.preferIPv4Stack=true

Adding default property: spark.blockManager.port=8007

Adding default property: spark.yarn.am.waitTime=200000

Adding default property: spark.master=yarn-client

Adding default property: spark.yarn.preserve.staging.files=true

Adding default property: spark.fileserver.port=8003

Adding default property: spark.authenticate=true

Adding default property: spark.yarn.am.port=8008

Adding default property: spark.authenticate.secret=fubar

Adding default property: spark.master.ui.port=8080

Adding default property: spark.history.ui.port=18080

Adding default property: spark.eventlog.enabled=true

Parsed arguments:

  master                  yarn-client

  deployMode              null

  executorMemory          null

  executorCores           1

  totalExecutorCores      null

  propertiesFile          /home/testuser/spark/conf/spark-defaults.conf

  driverMemory            null

  driverCores             null

  driverExtraClassPath    null

  driverExtraLibraryPath  null

 driverExtraJavaOptions  -Djava.net.preferIPv4Stack=true

  supervise               false

  queue                   null

  numExecutors            1

  files                   null

  pyFiles                 null

  archives                null

  mainClass               org.apache.spark.examples.SparkPi

  primaryResource         file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar

  name                    org.apache.spark.examples.SparkPi

  childArgs               [10]

  jars                    null

  packages                null

  repositories            null

  verbose                 true

 

Spark properties used, including those specified through

--conf and those from the properties file /home/testuser/spark/conf/spark-defaults.conf:

  spark.blockManager.port -> 8007

  spark.yarn.access.namenodes -> hdfs://10.10.10.10:8020

  spark.broadcast.port -> 8004

  spark.authenticate.secret -> fubar

  spark.authenticate -> true

  spark.eventlog.dir -> hdfs://10.10.10.10:8020/user/testuser/spark/eventlog

  spark.logConf -> true

  spark.replClassServer.port -> 8006

  spark.history.ui.port -> 18080

  spark.fileserver.port -> 8003

  spark.ui.port -> 4040

  spark.executor.extraJavaOptions -> -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

  spark.driver.extraJavaOptions -> -Djava.net.preferIPv4Stack=true

  spark.yarn.am.waitTime -> 200000

  spark.yarn.am.extraJavaOptions -> -Djava.net.preferIPv4Stack=true

  spark.worker.ui.port -> 8081

  spark.driver.port -> 8001

  spark.master -> yarn-client

  spark.yarn.preserve.staging.files -> true

  spark.yarn.am.port -> 8008

  spark.akka.heartbeat.interval -> 100

  spark.executor.port -> 8002

  spark.master.ui.port -> 8080

  spark.eventlog.enabled -> true

  spark.akka.threads -> 1

 

 

Main class:

org.apache.spark.examples.SparkPi

Arguments:

10

System properties:

spark.blockManager.port -> 8007

spark.yarn.access.namenodes -> hdfs://10.10.10.10:8020

spark.broadcast.port -> 8004

spark.authenticate.secret -> fubar

spark.eventlog.dir -> hdfs://10.10.10.10:8020/user/testuser/spark/eventlog

spark.authenticate -> true

spark.executor.instances -> 1

spark.logConf -> true

spark.replClassServer.port -> 8006

spark.history.ui.port -> 18080

spark.fileserver.port -> 8003

SPARK_SUBMIT -> true

spark.ui.port -> 4040

spark.executor.extraJavaOptions -> -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

spark.app.name -> org.apache.spark.examples.SparkPi

spark.driver.extraJavaOptions -> -Djava.net.preferIPv4Stack=true

spark.jars -> file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar

spark.yarn.am.waitTime -> 200000

spark.yarn.am.extraJavaOptions -> -Djava.net.preferIPv4Stack=true

spark.master -> yarn-client

spark.worker.ui.port -> 8081

spark.driver.port -> 8001

spark.yarn.preserve.staging.files -> true

spark.yarn.am.port -> 8008

spark.akka.heartbeat.interval -> 100

spark.executor.port -> 8002

spark.executor.cores -> 1

spark.eventlog.enabled -> true

spark.master.ui.port -> 8080

spark.akka.threads -> 1

Classpath elements:

file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar

 

 

15/05/19 19:52:37 INFO spark.SparkContext: Running Spark version 1.3.1

15/05/19 19:52:37 INFO spark.SparkContext: Spark configuration:

spark.akka.heartbeat.interval=100

spark.akka.threads=1

spark.app.name=Spark Pi

spark.authenticate=true

spark.authenticate.secret=fubar

spark.blockManager.port=8007

spark.broadcast.port=8004

spark.driver.extraJavaOptions=-Djava.net.preferIPv4Stack=true

spark.driver.port=8001

spark.eventlog.dir=hdfs://10.10.10.10:8020/user/testuser/spark/eventlog

spark.eventlog.enabled=true

spark.executor.cores=1

spark.executor.extraJavaOptions=-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

spark.executor.instances=1

spark.executor.port=8002

spark.fileserver.port=8003

spark.history.ui.port=18080

spark.jars=file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar

spark.logConf=true

spark.master=yarn-client

spark.master.ui.port=8080

spark.replClassServer.port=8006

spark.ui.port=4040

spark.worker.ui.port=8081

spark.yarn.access.namenodes=hdfs://10.10.10.10:8020

spark.yarn.am.extraJavaOptions=-Djava.net.preferIPv4Stack=true

spark.yarn.am.port=8008

spark.yarn.am.waitTime=200000

spark.yarn.preserve.staging.files=true

15/05/19 19:52:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

15/05/19 19:52:39 INFO spark.SecurityManager: Changing view acls to: testuser

15/05/19 19:52:39 INFO spark.SecurityManager: Changing modify acls to: testuser

15/05/19 19:52:39 INFO spark.SecurityManager: adding secret to credentials in yarn mode

15/05/19 19:52:39 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(testuser); users with modify permissions: Set(testuser)

15/05/19 19:52:40 INFO slf4j.Slf4jLogger: Slf4jLogger started

15/05/19 19:52:41 INFO Remoting: Starting remoting

15/05/19 19:52:41 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001]

15/05/19 19:52:41 INFO util.Utils: Successfully started service 'sparkDriver' on port 8001.

15/05/19 19:52:41 INFO spark.SparkEnv: Registering MapOutputTracker

15/05/19 19:52:41 INFO spark.SparkEnv: Registering BlockManagerMaster

15/05/19 19:52:41 INFO storage.DiskBlockManager: Created local directory at /scratch/spark-17902ac9-b400-4698-97c3-069d804a29e3/blockmgr-d95b3bdf-9c4d-4b48-97a5-4983dd2ab66d

15/05/19 19:52:41 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB

15/05/19 19:52:42 INFO spark.HttpFileServer: HTTP File server directory is /scratch/spark-d2fb7948-5ab3-4f22-804c-7485d209bd3e/httpd-91938877-b371-4c0c-ba3d-bb7ee8ec4e09

15/05/19 19:52:42 INFO spark.HttpServer: Starting HTTP Server

15/05/19 19:52:42 INFO server.Server: jetty-8.y.z-SNAPSHOT

15/05/19 19:52:42 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:8003

15/05/19 19:52:42 INFO util.Utils: Successfully started service 'HTTP file server' on port 8003.

15/05/19 19:52:42 INFO spark.SparkEnv: Registering OutputCommitCoordinator

15/05/19 19:52:42 INFO server.Server: jetty-8.y.z-SNAPSHOT

15/05/19 19:52:42 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040

15/05/19 19:52:42 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.

15/05/19 19:52:42 INFO ui.SparkUI: Started SparkUI at http://ip-10-10-127-10.ec2.internal:4040

15/05/19 19:52:43 INFO spark.SparkContext: Added JAR file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar at http://10.10.127.10:8003/jars/spark-examples-1.3.1-hadoop2.6.0.jar with timestamp 1432065163724

15/05/19 19:52:44 INFO client.RMProxy: Connecting to ResourceManager at ip-10-10-127-10.ec2.internal/10.10.127.10:8032

15/05/19 19:52:45 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers

15/05/19 19:52:45 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)

15/05/19 19:52:45 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead

15/05/19 19:52:45 INFO yarn.Client: Setting up container launch context for our AM

15/05/19 19:52:45 INFO yarn.Client: Preparing resources for our AM container

15/05/19 19:52:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 5 for testuser on 10.10.10.10:8020

15/05/19 19:52:46 INFO yarn.Client: Uploading resource file:/home/testuser/spark/lib/spark-assembly-1.3.1-hadoop2.6.0.jar -> hdfs://10.10.10.10:8020/user/testuser/.sparkStaging/application_1432064564266_0003/spark-assembly-1.3.1-hadoop2.6.0.jar

15/05/19 19:52:52 INFO yarn.Client: Setting up the launch environment for our AM container

15/05/19 19:52:52 INFO spark.SecurityManager: Changing view acls to: testuser

15/05/19 19:52:52 INFO spark.SecurityManager: Changing modify acls to: testuser

15/05/19 19:52:52 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(testuser); users with modify permissions: Set(testuser)

15/05/19 19:52:52 INFO yarn.Client: Submitting application 3 to ResourceManager

15/05/19 19:52:53 INFO impl.YarnClientImpl: Submitted application application_1432064564266_0003

15/05/19 19:52:54 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:52:54 INFO yarn.Client:

         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }

         diagnostics: N/A

         ApplicationMaster host: N/A

         ApplicationMaster RPC port: -1

         queue: default

         start time: 1432065172758

         final status: UNDEFINED

         tracking URL: https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003/

         user: testuser

15/05/19 19:52:55 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:52:56 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:52:57 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:52:58 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:52:59 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:00 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:01 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:02 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:03 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:04 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:05 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:06 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:07 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:08 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:09 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:10 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:11 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:12 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:13 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:14 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:15 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:16 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)

15/05/19 19:53:17 INFO cluster.YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977/user/YarnAM#-1453228800]

15/05/19 19:53:17 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> ip-10-10-127-10.ec2.internal, PROXY_URI_BASES -> https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003), /proxy/application_1432064564266_0003

15/05/19 19:53:17 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter

15/05/19 19:53:17 INFO yarn.Client: Application report for application_1432064564266_0003 (state: RUNNING)

15/05/19 19:53:17 INFO yarn.Client:

         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }

         diagnostics: N/A

         ApplicationMaster host: ip-10-10-128-10.ec2.internal

         ApplicationMaster RPC port: 0

         queue: default

         start time: 1432065172758

         final status: UNDEFINED

         tracking URL: https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003/

         user: testuser

15/05/19 19:53:17 INFO cluster.YarnClientSchedulerBackend: Application application_1432064564266_0003 has started running.

15/05/19 19:53:17 INFO netty.NettyBlockTransferService: Server created on 8007

15/05/19 19:53:17 INFO storage.BlockManagerMaster: Trying to register BlockManager

15/05/19 19:53:17 INFO storage.BlockManagerMasterActor: Registering block manager ip-10-10-127-10.ec2.internal:8007 with 267.3 MB RAM, BlockManagerId(<driver>, ip-10-10-127-10.ec2.internal, 8007)

15/05/19 19:53:17 INFO storage.BlockManagerMaster: Registered BlockManager

15/05/19 19:53:18 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)

15/05/19 19:53:18 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:35

15/05/19 19:53:18 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:35) with 10 output partitions (allowLocal=false)

15/05/19 19:53:18 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at SparkPi.scala:35)

15/05/19 19:53:18 INFO scheduler.DAGScheduler: Parents of final stage: List()

15/05/19 19:53:18 INFO scheduler.DAGScheduler: Missing parents: List()

15/05/19 19:53:18 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:31), which has no missing parents

15/05/19 19:53:18 INFO cluster.YarnScheduler: Cancelling stage 0

15/05/19 19:53:18 INFO scheduler.DAGScheduler: Stage 0 (reduce at SparkPi.scala:35) failed in Unknown s

15/05/19 19:53:18 INFO scheduler.DAGScheduler: Job 0 failed: reduce at SparkPi.scala:35, took 0.258029 s

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.reflect.InvocationTargetException

sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

java.lang.reflect.Constructor.newInstance(Constructor.java:526)

org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:68)

org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)

org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)

org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:79)

org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)

org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)

org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)

org.apache.spark.SparkContext.broadcast(SparkContext.scala:1051)

org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:839)

org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)

org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)

org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1362)

org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)

org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

 

        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204)

        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193)

        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)

        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)

        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:847)

        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)

        at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)

        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1362)

        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)

        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

[testuser@ip-10-10-127-10 spark]$

 

 

 

###: SPARK JOB USERLOG RESULTS :###

###################################

[yarn@ip-10-10-128-10 container_1432064564266_0003_01_000001]$ cat stderr

Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/filecache/13/spark-assembly-1.3.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/var/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

15/05/19 19:53:09 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]

15/05/19 19:53:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

15/05/19 19:53:12 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1432064564266_0003_000001

15/05/19 19:53:14 INFO spark.SecurityManager: Changing view acls to: testuser

15/05/19 19:53:14 INFO spark.SecurityManager: Changing modify acls to: testuser

15/05/19 19:53:14 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(testuser); users with modify permissions: Set(testuser)

15/05/19 19:53:15 INFO slf4j.Slf4jLogger: Slf4jLogger started

15/05/19 19:53:15 INFO Remoting: Starting remoting

15/05/19 19:53:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977]

15/05/19 19:53:16 INFO util.Utils: Successfully started service 'sparkYarnAM' on port 56977.

15/05/19 19:53:16 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.

15/05/19 19:53:16 INFO yarn.ApplicationMaster: Driver now available: ip-10-10-127-10.ec2.internal:8001

15/05/19 19:53:16 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001/user/YarnScheduler

15/05/19 19:53:16 INFO yarn.ApplicationMaster: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> ip-10-10-127-10.ec2.internal, PROXY_URI_BASES -> https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003),/proxy/application_1432064564266_0003)

15/05/19 19:53:16 INFO client.RMProxy: Connecting to ResourceManager at /10.10.127.10:8030

15/05/19 19:53:16 INFO yarn.YarnRMClient: Registering the ApplicationMaster

15/05/19 19:53:17 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead

15/05/19 19:53:17 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)

15/05/19 19:53:17 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000

15/05/19 19:53:18 INFO yarn.ApplicationMaster: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977] -> [akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001]

15/05/19 19:53:18 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0

15/05/19 19:53:18 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].

15/05/19 19:53:18 INFO yarn.ApplicationMaster: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977] -> [akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001]

15/05/19 19:53:18 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED

15/05/19 19:53:18 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.

[yarn@ip-10-10-128-10 container_1432064564266_0003_01_000001]$

 

 

 

###: YARN NODEMANAGER LOG RESULTS :###

######################################

2015-05-19 19:52:53,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1432064564266_0003_000001 (auth:SIMPLE)

2015-05-19 19:52:53,746 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user appattempt_1432064564266_0003_000001: id: appattempt_1432064564266_0003_000001: no such user

 

2015-05-19 19:52:53,746 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user appattempt_1432064564266_0003_000001

2015-05-19 19:52:53,746 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1432064564266_0003_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB

2015-05-19 19:52:53,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1432064564266_0003_01_000001 by user testuser

2015-05-19 19:52:53,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1432064564266_0003

2015-05-19 19:52:53,747 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=testuser     IP=10.10.127.10 OPERATION=Start Container Request       TARGET=ContainerManageImplRESULT=SUCCESS   APPID=application_1432064564266_0003    CONTAINERID=container_1432064564266_0003_01_000001

2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from NEW to INITING

2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1432064564266_0003_01_000001 to application application_1432064564266_0003

2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from INITING to RUNNING

2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from NEW to LOCALIZING

2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1432064564266_0003

2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.10.10.10:8020/user/testuser/.sparkStaging/application_1432064564266_0003/spark-assembly-1.3.1-hadoop2.6.0.jar transitioned from INIT to DOWNLOADING

2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1432064564266_0003_01_000001

2015-05-19 19:52:53,763 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /var/hadoop/tmp/yarn/nm-local-dir/nmPrivate/container_1432064564266_0003_01_000001.tokens. Credentials list:

2015-05-19 19:52:58,614 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for testing (auth:SIMPLE)

2015-05-19 19:52:58,685 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for testing (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB

2015-05-19 19:53:08,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.10.10.10:8020/user/testuser/.sparkStaging/application_1432064564266_0003/spark-assembly-1.3.1-hadoop2.6.0.jar(->/var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/filecache/13/spark-assembly-1.3.1-hadoop2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED

2015-05-19 19:53:08,134 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from LOCALIZING to LOCALIZED

2015-05-19 19:53:08,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from LOCALIZED to RUNNING

2015-05-19 19:53:10,044 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1432064564266_0003_01_000001

2015-05-19 19:53:10,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 79.5 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used

2015-05-19 19:53:13,105 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 99.8 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used

2015-05-19 19:53:16,122 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 121.4 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used

2015-05-19 19:53:19,129 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 125.8 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used

2015-05-19 19:53:19,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container container_1432064564266_0003_01_000001 succeeded

2015-05-19 19:53:19,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS

2015-05-19 19:53:19,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1432064564266_0003_01_000001

2015-05-19 19:53:19,457 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=testuser     OPERATION=Container Finished - Succeeded        TARGET=ContainerImpl    RESULT=SUCCESS     APPID=application_1432064564266_0003    CONTAINERID=container_1432064564266_0003_01_000001

2015-05-19 19:53:19,457 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE

2015-05-19 19:53:19,457 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1432064564266_0003_01_000001 from application application_1432064564266_0003

2015-05-19 19:53:19,458 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1432064564266_0003

2015-05-19 19:53:19,458 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/appcache/application_1432064564266_0003/container_1432064564266_0003_01_000001

2015-05-19 19:53:20,428 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1432064564266_0003_000001 (auth:SIMPLE)

2015-05-19 19:53:20,432 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user appattempt_1432064564266_0003_000001

2015-05-19 19:53:20,432 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1432064564266_0003_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB

2015-05-19 19:53:20,432 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1432064564266_0003_01_000001

2015-05-19 19:53:20,432 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=testuser     IP=10.10.127.10 OPERATION=Stop Container Request        TARGET=ContainerManageImplRESULT=SUCCESS   APPID=application_1432064564266_0003    CONTAINERID=container_1432064564266_0003_01_000001

2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1432064564266_0003_01_000001]

2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP

2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1432064564266_0003

2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED

2015-05-19 19:53:20,443 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1432064564266_0003, with delay of 10800 seconds

2015-05-19 19:53:20,443 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/appcache/application_1432064564266_0003

2015-05-19 19:53:22,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1432064564266_0003_01_000001

root@ip-10-10-128-10:/var/log/hadoop>

 

 

###: YARN RESOURCE MANAGER LOGS :###

####################################

2015-05-19 19:52:45,408 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for testuser@MALARD.LOCAL (auth:KERBEROS)

2015-05-19 19:52:45,447 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for testuser@MALARD.LOCAL (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB

2015-05-19 19:52:45,494 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 3

2015-05-19 19:52:52,758 WARN org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 3 is invalid, because it is out of the range [1, 2]. Use the global max attempts instead.

2015-05-19 19:52:52,758 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 3 submitted by user testuser

2015-05-19 19:52:52,758 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser IP=10.10.127.10 OPERATION=Submit Application Request    TARGET=ClientRMService  RESULT=SUCCESS     APPID=application_1432064564266_0003

2015-05-19 19:52:52,803 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: application_1432064564266_0003 found existing hdfs token Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020, Ident: (HDFS_DELEGATION_TOKEN token 5 for testuser)

2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020, Ident: (HDFS_DELEGATION_TOKEN token 5 for testuser);exp=1432151572908], for application_1432064564266_0003

2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020, Ident: (HDFS_DELEGATION_TOKEN token 5 for testuser);exp=1432151572908 in 86399980 ms, appId = application_1432064564266_0003

2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1432064564266_0003

2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from NEW to NEW_SAVING

2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1432064564266_0003

2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from NEW_SAVING to SUBMITTED

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1432064564266_0003 user: testuser leaf-queue of parent: root #applications: 1

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1432064564266_0003 from user: testuser, in queue: default

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from SUBMITTED to ACCEPTED

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1432064564266_0003_000001

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from NEW to SUBMITTED

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1432064564266_0003 from user: testuser activated in queue: default

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1432064564266_0003 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@72fa0d32, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1

2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1432064564266_0003_000001 to scheduler from user testuser in queue default

2015-05-19 19:52:52,930 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from SUBMITTED to SCHEDULED

2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from NEW to ALLOCATED

2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Allocated Container        TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000001

2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1432064564266_0003_01_000001 of capacity <memory:1024, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which has 1 containers, <memory:1024, vCores:1> used and <memory:7168, vCores:7> available after allocation

2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1432064564266_0003_000001 container=Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 clusterResource=<memory:8192, vCores:8>

2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=1

2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.125 absoluteUsedCapacity=0.125 used=<memory:1024, vCores:1> cluster=<memory:8192, vCores:8>

2015-05-19 19:52:53,719 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : ip-10-10-128-10.ec2.internal:9032 for container : container_1432064564266_0003_01_000001

2015-05-19 19:52:53,720 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from ALLOCATED to ACQUIRED

2015-05-19 19:52:53,720 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1432064564266_0003_000001

2015-05-19 19:52:53,721 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1432064564266_0003 AttemptId: appattempt_1432064564266_0003_000001 MasterContainer: Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ]

2015-05-19 19:52:53,721 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from SCHEDULED to ALLOCATED_SAVING

2015-05-19 19:52:53,721 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from ALLOCATED_SAVING to ALLOCATED

2015-05-19 19:52:53,723 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1432064564266_0003_000001

2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ] for AM appattempt_1432064564266_0003_000001

2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1432064564266_0003_01_000001 : {{JAVA_HOME}}/bin/java,-server,-Xmx512m,-Djava.io.tmpdir={{PWD}}/tmp,'-Dspark.fileserver.uri=http://10.10.127.10:8003','-Dspark.broadcast.port=8004','-Dspark.executor.port=8002','-Dspark.driver.appUIAddress=http://ip-10-10-127-10.ec2.internal:4040','-Dspark.fileserver.port=8003','-Dspark.ui.port=4040','-Dspark.tachyonStore.folderName=spark-ea0a49a2-1643-4410-892d-690c62cb6857','-Dspark.driver.extraJavaOptions=-Djava.net.preferIPv4Stack=true','-Dspark.jars=file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar','-Dspark.app.name=Spark Pi','-Dspark.akka.threads=1','-Dspark.authenticate.secret=fubar','-Dspark.eventlog.enabled=true','-Dspark.driver.host=ip-10-10-127-10.ec2.internal','-Dspark.blockManager.port=8007','-Dspark.history.ui.port=18080','-Dspark.replClassServer.port=8006','-Dspark.worker.ui.port=8081','-Dspark.master=yarn-client','-Dspark.yarn.preserve.staging.files=true','-Dspark.executor.extraJavaOptions=-XX:+PrintGCDetails -Dkey=value -Dnumbers=\"one two three\"','-Dspark.yarn.access.namenodes=hdfs://10.10.10.10:8020','-Dspark.logConf=true','-Dspark.driver.port=8001','-Dspark.executor.id=<driver>','-Dspark.akka.heartbeat.interval=100','-Dspark.master.ui.port=8080','-Dspark.yarn.am.waitTime=200000','-Dspark.yarn.am.extraJavaOptions=-Djava.net.preferIPv4Stack=true','-Dspark.executor.instances=1','-Dspark.yarn.am.port=8008','-Dspark.eventlog.dir=hdfs://10.10.10.10:8020/user/testuser/spark/eventlog','-Dspark.executor.cores=1','-Dspark.authenticate=true','-Djava.net.preferIPv4Stack=true',-Dspark.yarn.app.container.log.dir=<LOG_DIR>,org.apache.spark.deploy.yarn.ExecutorLauncher,--arg,'ip-10-10-127-10.ec2.internal:8001',--executor-memory,1024m,--executor-cores,1,--num-executors ,1,1>,<LOG_DIR>/stdout,2>,<LOG_DIR>/stderr

2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1432064564266_0003_000001

2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1432064564266_0003_000001

2015-05-19 19:52:53,750 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ] for AM appattempt_1432064564266_0003_000001

2015-05-19 19:52:53,750 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from ALLOCATED to LAUNCHED

2015-05-19 19:52:54,733 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from ACQUIRED to RUNNING

2015-05-19 19:53:17,318 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1432064564266_0003_000001 (auth:SIMPLE)

2015-05-19 19:53:17,376 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user appattempt_1432064564266_0003_000001: id: appattempt_1432064564266_0003_000001: no such user

 

2015-05-19 19:53:17,376 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user appattempt_1432064564266_0003_000001

2015-05-19 19:53:17,376 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1432064564266_0003_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB

2015-05-19 19:53:17,377 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1432064564266_0003_000001

2015-05-19 19:53:17,377 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser IP=10.10.128.10 OPERATION=Register App Master   TARGET=ApplicationMasterService RESULT=SUCCESS     APPID=application_1432064564266_0003    APPATTEMPTID=appattempt_1432064564266_0003_000001

2015-05-19 19:53:17,377 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Setting client token master key

2015-05-19 19:53:17,378 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from LAUNCHED to RUNNING

2015-05-19 19:53:17,378 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from ACCEPTED to RUNNING

2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000002 Container Transitioned from NEW to ALLOCATED

2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Allocated Container        TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000002

2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1432064564266_0003_01_000002 of capacity <memory:2048, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which has 2 containers, <memory:3072, vCores:2> used and <memory:5120, vCores:6> available after allocation

2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1432064564266_0003_000001 container=Container: [ContainerId: container_1432064564266_0003_01_000002, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:2048, vCores:1>, Priority: 1, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=1 clusterResource=<memory:8192, vCores:8>

2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:3072, vCores:2>, usedCapacity=0.375, absoluteUsedCapacity=0.375, numApps=1, numContainers=2

2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.375 absoluteUsedCapacity=0.375 used=<memory:3072, vCores:2> cluster=<memory:8192, vCores:8>

2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1432064564266_0003_000001 with final state: FINISHING, and exit status: -1000

2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from RUNNING to FINAL_SAVING

2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1432064564266_0003 with final state: FINISHING

2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from RUNNING to FINAL_SAVING

2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1432064564266_0003

2015-05-19 19:53:18,968 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from FINAL_SAVING to FINISHING

2015-05-19 19:53:18,968 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from FINAL_SAVING to FINISHING

2015-05-19 19:53:19,076 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: application_1432064564266_0003 unregistered successfully.

2015-05-19 19:53:20,383 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from RUNNING to COMPLETED

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1432064564266_0003_01_000001 in state: COMPLETED event:FINISHED

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000001

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1432064564266_0003_01_000001 of capacity <memory:1024, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which currently has 1 containers, <memory:2048, vCores:1> used and <memory:6144, vCores:7> available, release resources=true

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:2048, vCores:1> numContainers=1 user=testuser user-resources=<memory:2048, vCores:1>

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:2048, vCores:1>, usedCapacity=0.25, absoluteUsedCapacity=0.25, numApps=1, numContainers=1 cluster=<memory:8192, vCores:8>

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.25 absoluteUsedCapacity=0.25 used=<memory:2048, vCores:1> cluster=<memory:8192, vCores:8>

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:2048, vCores:1>, usedCapacity=0.25, absoluteUsedCapacity=0.25, numApps=1, numContainers=1

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1432064564266_0003_000001 released container container_1432064564266_0003_01_000001 on node: host: ip-10-10-128-10.ec2.internal:9032 #containers=1 available=6144 used=2048 with event: FINISHED

2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1432064564266_0003_000001

2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1432064564266_0003_000001

2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from FINISHING to FINISHED

2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from FINISHING to FINISHED

2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1432064564266_0003_000001 is done. finalState=FINISHED

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000002 Container Transitioned from ALLOCATED to KILLED

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1432064564266_0003_01_000002 in state: KILLED event:KILL

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000002

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1432064564266_0003_01_000002 of capacity <memory:2048, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:0, vCores:0> numContainers=0 user=testuser user-resources=<memory:0, vCores:0>

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1432064564266_0003_01_000002, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:2048, vCores:1>, Priority: 1, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 cluster=<memory:8192, vCores:8>

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:8192, vCores:8>

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1432064564266_0003_000001 released container container_1432064564266_0003_01_000002 on node: host: ip-10-10-128-10.ec2.internal:9032 #containers=0 available=8192 used=0 with event: KILL

2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1432064564266_0003 requests cleared

2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1432064564266_0003 user: testuser queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0

2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1432064564266_0003 user: testuser leaf-queue of parent: root #applications: 0

2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=Application Finished - Succeeded      TARGET=RMAppManager     RESULT=SUCCESS     APPID=application_1432064564266_0003

2015-05-19 19:53:20,389 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1432064564266_0003,name=Spark Pi,user=testuser,queue=default,state=FINISHED,trackingUrl=https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003/A,appMasterHost=ip-10-10-128-10.ec2.internal,startTime=1432065172758,finishTime=1432065198967,finalStatus=SUCCEEDED

2015-05-19 19:53:20,389 INFO org.apache.hadoop.hdfs.DFSClient: Cancelling HDFS_DELEGATION_TOKEN token 5 for testuser on 10.10.10.10:8020

2015-05-19 19:53:20,390 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1432064564266_0003_000001

2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Null container completed...

root@ip-10-10-127-10:/var/log/hadoop>

 

Keith Nance

Sr. Software Engineer

*Email: knance@smartronix.com

(    Cell: 808-343-0071

cid:image002.jpg@01CA58DB.D44B0990

www.smartronix.com