predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shane Johnson <shanewaldenjohn...@gmail.com>
Subject Exporting data into Parquet format fails
Date Mon, 05 Jun 2017 13:43:07 GMT
Hi all,

I am trying to follow the analytics-zepplin tutorial and am having issues
exporting the data from the event server into parquet. I am using postgres
for my event server and am building locally on a mac. Has anyone run into
this error before? I've checked to make sure I can save to this directory
and have tried multiple paths.


[ERROR] [FileOutputCommitter] Mkdirs failed to create
file:/data/_temporary/0



http://predictionio.incubator.apache.org/datacollection/analytics-zeppelin/

[image: Inline image 1]

Sidnis-Air% pio export --appid 2 --output /data --format parquet

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in
[jar:file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in
[jar:file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/pio-assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

[INFO] [Runner$] Submission command:
/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit
--class org.apache.predictionio.tools.export.EventsToFile --jars
file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.11.0-incubating.jar,file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
--files
file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/conf/log4j.properties
--driver-class-path
/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/conf:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/conf:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/postgresql_jdbc.jar:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/conf
--driver-java-options -Dpio.log.dir=/Users/shane
file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/pio-assembly-0.11.0-incubating.jar
--appid 2 --output /data --format parquet --env
PIO_ENV_LOADED=1,PIO_EVENTSERVER_APP_NAME=classi,PIO_STORAGE_SOURCES_PGSQL_INDEX=enabled,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/Users/shane/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_HOME=/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist,PIO_FS_ENGINESDIR=/Users/shane/.pio_store/engines,PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost:5432/pio,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/vendors/elasticsearch,PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/Users/shane/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http,PIO_POSTGRES_OPTIONAL_SSL=true,PIO_STORAGE_SOURCES_PGSQL_CONNECTIONS=8,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_CONF_DIR=/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/conf,PIO_ORIGINAL_PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/shane/Desktop/Apps/pio/apache-predictionio-0.11.0-incubating/PredictionIO-0.11.0-incubating/bin,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_BUILDPACK_DIR=/Users/shane/Desktop/Apps/liftiq_prototype/predictionio-buildpack,PIO_STORAGE_SOURCES_PGSQL_PARTITIONS=4

[INFO] [log] Logging initialized @6467ms

[INFO] [Server] jetty-9.2.z-SNAPSHOT

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@58ba5b30
{/jobs,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4dba773d
{/jobs/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1d9bd4da
{/jobs/job,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4c58255
{/jobs/job/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@eac3a26
{/stages,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@10b1a751
{/stages/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@53cf9c99
{/stages/stage,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7b306b9f
{/stages/stage/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@142213d5
{/stages/pool,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@934b52f
{/stages/pool/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2630dbc4
{/storage,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5ea4300e
{/storage/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5a1c3cb4
{/storage/rdd,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@76ad6715
{/storage/rdd/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@56637cff
{/environment,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2938127d
{/environment/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@127a7272
{/executors,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@ff23ae7
{/executors/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@423b2b62
{/executors/threadDump,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2b7962a2
{/executors/threadDump/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6ad16c5d
{/static,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@74ea46e2
{/,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6f911326
{/api,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@f1d0004
{/jobs/job/kill,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6e4f263e
{/stages/stage/kill,null,AVAILABLE}

[INFO] [ServerConnector] Started ServerConnector@6edf29c1{HTTP/1.1}{
127.0.0.1:4040}

[INFO] [Server] Started @6723ms

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@26ca61bf
{/metrics/json,null,AVAILABLE}

[WARN] [SparkContext] Using an existing SparkContext; some configuration
may not take effect.

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@58dad04a
{/SQL,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@113ee1ce
{/SQL/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@467045c4
{/SQL/execution,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5a566922
{/SQL/execution/json,null,AVAILABLE}

[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7015ebef
{/static/sql,null,AVAILABLE}

[ERROR] [FileOutputCommitter] Mkdirs failed to create
file:/data/_temporary/0

[Stage 1:>                                                          (0 + 4)
/ 4][INFO] [CodecConfig] Compression: SNAPPY

[INFO] [CodecConfig] Compression: SNAPPY

[INFO] [CodecConfig] Compression: SNAPPY

[INFO] [CodecConfig] Compression: SNAPPY

[INFO] [CodecConfig] Compression: SNAPPY

[INFO] [CodecConfig] Compression: SNAPPY

[INFO] [CodecConfig] Compression: SNAPPY

[INFO] [CodecConfig] Compression: SNAPPY

[INFO] [ParquetOutputFormat] Parquet block size to 134217728

[INFO] [ParquetOutputFormat] Parquet block size to 134217728

[INFO] [ParquetOutputFormat] Parquet page size to 1048576

[INFO] [ParquetOutputFormat] Parquet block size to 134217728

[INFO] [ParquetOutputFormat] Parquet block size to 134217728

[INFO] [ParquetOutputFormat] Parquet page size to 1048576

[INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576

[INFO] [ParquetOutputFormat] Parquet page size to 1048576

[INFO] [ParquetOutputFormat] Dictionary is on

[INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576

[INFO] [ParquetOutputFormat] Parquet page size to 1048576

[INFO] [ParquetOutputFormat] Dictionary is on

[INFO] [ParquetOutputFormat] Validation is off

[INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576

[INFO] [ParquetOutputFormat] Validation is off

[INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576

[INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0

[INFO] [ParquetOutputFormat] Dictionary is on

[INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes

[INFO] [ParquetOutputFormat] Dictionary is on

[INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0

[INFO] [ParquetOutputFormat] Validation is off

[INFO] [ParquetOutputFormat] Validation is off

[INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0

[INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes

[INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes

[INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0

[INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes

[ERROR] [Executor] Exception in task 1.0 in stage 1.0 (TID 5)

[ERROR] [Executor] Exception in task 2.0 in stage 1.0 (TID 6)

[ERROR] [Executor] Exception in task 3.0 in stage 1.0 (TID 7)

[ERROR] [Executor] Exception in task 0.0 in stage 1.0 (TID 4)

[WARN] [TaskSetManager] Lost task 2.0 in stage 1.0 (TID 6, localhost,
executor driver): java.io.IOException: Mkdirs failed to create
file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000002_0
(exists=false,
cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:223)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:312)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:283)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

at org.apache.spark.scheduler.Task.run(Task.scala:99)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)


[ERROR] [TaskSetManager] Task 2 in stage 1.0 failed 1 times; aborting job

[WARN] [TaskSetManager] Lost task 1.0 in stage 1.0 (TID 5, localhost,
executor driver): java.io.IOException: Mkdirs failed to create
file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000001_0
(exists=false,
cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:223)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:312)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:283)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

at org.apache.spark.scheduler.Task.run(Task.scala:99)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)


[WARN] [TaskSetManager] Lost task 0.0 in stage 1.0 (TID 4, localhost,
executor driver): java.io.IOException: Mkdirs failed to create
file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000000_0
(exists=false,
cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:223)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:312)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:283)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

at org.apache.spark.scheduler.Task.run(Task.scala:99)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)


[WARN] [TaskSetManager] Lost task 3.0 in stage 1.0 (TID 7, localhost,
executor driver): java.io.IOException: Mkdirs failed to create
file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000003_0
(exists=false,
cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:223)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:312)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:283)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

at org.apache.spark.scheduler.Task.run(Task.scala:99)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)


[ERROR] [FileFormatWriter] Aborting job null.

Exception in thread "main" org.apache.spark.SparkException: Job aborted.

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:147)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)

at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:121)

at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:101)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)

at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)

at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)

at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)

at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)

at
org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:492)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:198)

at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:494)

at
org.apache.predictionio.tools.export.EventsToFile$$anonfun$main$1.apply(EventsToFile.scala:102)

at
org.apache.predictionio.tools.export.EventsToFile$$anonfun$main$1.apply(EventsToFile.scala:71)

at scala.Option.map(Option.scala:146)

at
org.apache.predictionio.tools.export.EventsToFile$.main(EventsToFile.scala:71)

at
org.apache.predictionio.tools.export.EventsToFile.main(EventsToFile.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task
2.0 in stage 1.0 (TID 6, localhost, executor driver): java.io.IOException:
Mkdirs failed to create
file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000002_0
(exists=false,
cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:223)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:312)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:283)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

at org.apache.spark.scheduler.Task.run(Task.scala:99)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)


Driver stacktrace:

at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)

at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)

at scala.Option.foreach(Option.scala:257)

at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)

at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)

at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)

at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)

at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:127)

... 34 more

Caused by: java.io.IOException: Mkdirs failed to create
file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000002_0
(exists=false,
cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-classi)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)

at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:223)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:312)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:283)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)

at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

at org.apache.spark.scheduler.Task.run(Task.scala:99)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)

[INFO] [ServerConnector] Stopped ServerConnector@6edf29c1{HTTP/1.1}{
127.0.0.1:4040}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6e4f263e
{/stages/stage/kill,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@f1d0004
{/jobs/job/kill,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6f911326
{/api,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@74ea46e2
{/,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6ad16c5d
{/static,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2b7962a2
{/executors/threadDump/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@423b2b62
{/executors/threadDump,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@ff23ae7
{/executors/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@127a7272
{/executors,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2938127d
{/environment/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@56637cff
{/environment,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@76ad6715
{/storage/rdd/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5a1c3cb4
{/storage/rdd,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5ea4300e
{/storage/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2630dbc4
{/storage,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@934b52f
{/stages/pool/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@142213d5
{/stages/pool,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@7b306b9f
{/stages/stage/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@53cf9c99
{/stages/stage,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@10b1a751
{/stages/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@eac3a26
{/stages,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@4c58255
{/jobs/job/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1d9bd4da
{/jobs/job,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@4dba773d
{/jobs/json,null,UNAVAILABLE}

[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@58ba5b30
{/jobs,null,UNAVAILABLE}


*Shane Johnson | 801.360.3350*
LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
<https://www.facebook.com/shane.johnson.71653>

Mime
View raw message