predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Szeto <don...@apache.org>
Subject Re: Exporting data into Parquet format fails
Date Mon, 05 Jun 2017 18:08:29 GMT
Hey Shane,

Does that directory have enough free space? Sometimes it could fill up
pretty quickly during execution.

Regards,
Donald

On Mon, Jun 5, 2017 at 6:43 AM, Shane Johnson <shanewaldenjohnson@gmail.com>
wrote:

> Hi all,
>
> I am trying to follow the analytics-zepplin tutorial and am having issues
> exporting the data from the event server into parquet. I am using postgres
> for my event server and am building locally on a mac. Has anyone run into
> this error before? I've checked to make sure I can save to this directory
> and have tried multiple paths.
>
>
> [ERROR] [FileOutputCommitter] Mkdirs failed to create
> file:/data/_temporary/0
>
>
>
> http://predictionio.incubator.apache.org/datacollection/
> analytics-zeppelin/
>
> [image: Inline image 1]
>
> Sidnis-Air% pio export --appid 2 --output /data --format parquet
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/Users/shane/
> Desktop/Apps/liftiq_prototype/pio-engine-classi/
> PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.11.0-
> incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/Users/shane/
> Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/lib/pio-
> assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> [INFO] [Runner$] Submission command: /Users/shane/Desktop/Apps/
> liftiq_prototype/pio-engine-classi/PredictionIO-dist/
> vendors/spark-hadoop/bin/spark-submit --class
> org.apache.predictionio.tools.export.EventsToFile --jars
> file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi/PredictionIO-dist/lib/postgresql_jdbc.jar,
> file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi/PredictionIO-dist/lib/spark/pio-data-
> elasticsearch-assembly-0.11.0-incubating.jar,file:/Users/
> shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/
> PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.11.
> 0-incubating.jar,file:/Users/shane/Desktop/Apps/liftiq_
> prototype/pio-engine-classi/PredictionIO-dist/lib/spark/
> pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/Users/
> shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/
> PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/Users/shane/Desktop/Apps/liftiq_
> prototype/pio-engine-classi/PredictionIO-dist/lib/spark/
> pio-data-localfs-assembly-0.11.0-incubating.jar --files
> file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi/PredictionIO-dist/conf/log4j.properties --driver-class-path
> /Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-
> classi/PredictionIO-dist/conf:/Users/shane/Desktop/Apps/
> liftiq_prototype/pio-engine-classi/PredictionIO-dist/conf:
> /Users/shane/Desktop/Apps/liftiq_prototype/pio-engine-
> classi/PredictionIO-dist/lib/postgresql_jdbc.jar:/Users/
> shane/Desktop/Apps/liftiq_prototype/pio-engine-classi/PredictionIO-dist/conf
> --driver-java-options -Dpio.log.dir=/Users/shane file:/Users/shane/Desktop/
> Apps/liftiq_prototype/pio-engine-classi/PredictionIO-
> dist/lib/pio-assembly-0.11.0-incubating.jar --appid 2 --output /data
> --format parquet --env PIO_ENV_LOADED=1,PIO_EVENTSERVER_APP_NAME=classi,
> PIO_STORAGE_SOURCES_PGSQL_INDEX=enabled,PIO_STORAGE_
> REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/
> Users/shane/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_
> HOSTS=localhost,PIO_HOME=/Users/shane/Desktop/Apps/
> liftiq_prototype/pio-engine-classi/PredictionIO-dist,PIO_
> FS_ENGINESDIR=/Users/shane/.pio_store/engines,PIO_STORAGE_
> SOURCES_PGSQL_URL=jdbc:postgresql://localhost:5432/
> pio,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_
> REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_STORAGE_
> REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_
> REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_
> PGSQL_PASSWORD=pio,PIO_STORAGE_SOURCES_ELASTICSEARCH_
> HOME=/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi/PredictionIO-dist/vendors/elasticsearch,
> PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/
> Users/shane/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_
> USERNAME=pio,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=
> pio_model,PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http,
> PIO_POSTGRES_OPTIONAL_SSL=true,PIO_STORAGE_SOURCES_
> PGSQL_CONNECTIONS=8,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_
> CONF_DIR=/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi/PredictionIO-dist/conf,PIO_ORIGINAL_PATH=/
> usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:
> /Users/shane/Desktop/Apps/pio/apache-predictionio-0.11.0-
> incubating/PredictionIO-0.11.0-incubating/bin,PIO_STORAGE_
> SOURCES_ELASTICSEARCH_PORTS=9200,PIO_BUILDPACK_DIR=/Users/
> shane/Desktop/Apps/liftiq_prototype/predictionio-
> buildpack,PIO_STORAGE_SOURCES_PGSQL_PARTITIONS=4
>
> [INFO] [log] Logging initialized @6467ms
>
> [INFO] [Server] jetty-9.2.z-SNAPSHOT
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 58ba5b30{/jobs,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 4dba773d{/jobs/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 1d9bd4da{/jobs/job,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 4c58255{/jobs/job/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> eac3a26{/stages,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 10b1a751{/stages/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 53cf9c99{/stages/stage,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 7b306b9f{/stages/stage/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 142213d5{/stages/pool,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 934b52f{/stages/pool/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 2630dbc4{/storage,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 5ea4300e{/storage/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 5a1c3cb4{/storage/rdd,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 76ad6715{/storage/rdd/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 56637cff{/environment,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 2938127d{/environment/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 127a7272{/executors,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> ff23ae7{/executors/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 423b2b62{/executors/threadDump,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 2b7962a2{/executors/threadDump/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 6ad16c5d{/static,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 74ea46e2{/,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 6f911326{/api,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> f1d0004{/jobs/job/kill,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 6e4f263e{/stages/stage/kill,null,AVAILABLE}
>
> [INFO] [ServerConnector] Started ServerConnector@6edf29c1{HTTP/1.1}{
> 127.0.0.1:4040}
>
> [INFO] [Server] Started @6723ms
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 26ca61bf{/metrics/json,null,AVAILABLE}
>
> [WARN] [SparkContext] Using an existing SparkContext; some configuration
> may not take effect.
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 58dad04a{/SQL,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 113ee1ce{/SQL/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 467045c4{/SQL/execution,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 5a566922{/SQL/execution/json,null,AVAILABLE}
>
> [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@
> 7015ebef{/static/sql,null,AVAILABLE}
>
> [ERROR] [FileOutputCommitter] Mkdirs failed to create
> file:/data/_temporary/0
>
> [Stage 1:>                                                          (0 +
> 4) / 4][INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [CodecConfig] Compression: SNAPPY
>
> [INFO] [ParquetOutputFormat] Parquet block size to 134217728
>
> [INFO] [ParquetOutputFormat] Parquet block size to 134217728
>
> [INFO] [ParquetOutputFormat] Parquet page size to 1048576
>
> [INFO] [ParquetOutputFormat] Parquet block size to 134217728
>
> [INFO] [ParquetOutputFormat] Parquet block size to 134217728
>
> [INFO] [ParquetOutputFormat] Parquet page size to 1048576
>
> [INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576
>
> [INFO] [ParquetOutputFormat] Parquet page size to 1048576
>
> [INFO] [ParquetOutputFormat] Dictionary is on
>
> [INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576
>
> [INFO] [ParquetOutputFormat] Parquet page size to 1048576
>
> [INFO] [ParquetOutputFormat] Dictionary is on
>
> [INFO] [ParquetOutputFormat] Validation is off
>
> [INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576
>
> [INFO] [ParquetOutputFormat] Validation is off
>
> [INFO] [ParquetOutputFormat] Parquet dictionary page size to 1048576
>
> [INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0
>
> [INFO] [ParquetOutputFormat] Dictionary is on
>
> [INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes
>
> [INFO] [ParquetOutputFormat] Dictionary is on
>
> [INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0
>
> [INFO] [ParquetOutputFormat] Validation is off
>
> [INFO] [ParquetOutputFormat] Validation is off
>
> [INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0
>
> [INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes
>
> [INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes
>
> [INFO] [ParquetOutputFormat] Writer version is: PARQUET_1_0
>
> [INFO] [ParquetOutputFormat] Maximum row group padding size is 0 bytes
>
> [ERROR] [Executor] Exception in task 1.0 in stage 1.0 (TID 5)
>
> [ERROR] [Executor] Exception in task 2.0 in stage 1.0 (TID 6)
>
> [ERROR] [Executor] Exception in task 3.0 in stage 1.0 (TID 7)
>
> [ERROR] [Executor] Exception in task 0.0 in stage 1.0 (TID 4)
>
> [WARN] [TaskSetManager] Lost task 2.0 in stage 1.0 (TID 6, localhost,
> executor driver): java.io.IOException: Mkdirs failed to create
> file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000002_0
> (exists=false, cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:455)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:440)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>
> at org.apache.parquet.hadoop.ParquetFileWriter.<init>(
> ParquetFileWriter.java:223)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:312)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:283)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$
> SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)
>
> at org.apache.spark.sql.execution.datasources.
> FileFormatWriter$.org$apache$spark$sql$execution$
> datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
> [ERROR] [TaskSetManager] Task 2 in stage 1.0 failed 1 times; aborting job
>
> [WARN] [TaskSetManager] Lost task 1.0 in stage 1.0 (TID 5, localhost,
> executor driver): java.io.IOException: Mkdirs failed to create
> file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000001_0
> (exists=false, cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:455)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:440)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>
> at org.apache.parquet.hadoop.ParquetFileWriter.<init>(
> ParquetFileWriter.java:223)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:312)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:283)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$
> SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)
>
> at org.apache.spark.sql.execution.datasources.
> FileFormatWriter$.org$apache$spark$sql$execution$
> datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
> [WARN] [TaskSetManager] Lost task 0.0 in stage 1.0 (TID 4, localhost,
> executor driver): java.io.IOException: Mkdirs failed to create
> file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000000_0
> (exists=false, cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:455)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:440)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>
> at org.apache.parquet.hadoop.ParquetFileWriter.<init>(
> ParquetFileWriter.java:223)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:312)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:283)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$
> SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)
>
> at org.apache.spark.sql.execution.datasources.
> FileFormatWriter$.org$apache$spark$sql$execution$
> datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
> [WARN] [TaskSetManager] Lost task 3.0 in stage 1.0 (TID 7, localhost,
> executor driver): java.io.IOException: Mkdirs failed to create
> file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000003_0
> (exists=false, cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:455)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:440)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>
> at org.apache.parquet.hadoop.ParquetFileWriter.<init>(
> ParquetFileWriter.java:223)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:312)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:283)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$
> SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)
>
> at org.apache.spark.sql.execution.datasources.
> FileFormatWriter$.org$apache$spark$sql$execution$
> datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
> [ERROR] [FileFormatWriter] Aborting job null.
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted.
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1.apply$mcV$sp(FileFormatWriter.scala:147)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1.apply(FileFormatWriter.scala:121)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1.apply(FileFormatWriter.scala:121)
>
> at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(
> SQLExecution.scala:57)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(
> FileFormatWriter.scala:121)
>
> at org.apache.spark.sql.execution.datasources.
> InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationComm
> and.scala:101)
>
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.
> sideEffectResult$lzycompute(commands.scala:58)
>
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.
> sideEffectResult(commands.scala:56)
>
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(
> commands.scala:74)
>
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$
> execute$1.apply(SparkPlan.scala:114)
>
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$
> execute$1.apply(SparkPlan.scala:114)
>
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(
> SparkPlan.scala:135)
>
> at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:151)
>
> at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:
> 132)
>
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
>
> at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(
> QueryExecution.scala:87)
>
> at org.apache.spark.sql.execution.QueryExecution.
> toRdd(QueryExecution.scala:87)
>
> at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.
> scala:492)
>
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)
>
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:198)
>
> at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:494)
>
> at org.apache.predictionio.tools.export.EventsToFile$$anonfun$
> main$1.apply(EventsToFile.scala:102)
>
> at org.apache.predictionio.tools.export.EventsToFile$$anonfun$
> main$1.apply(EventsToFile.scala:71)
>
> at scala.Option.map(Option.scala:146)
>
> at org.apache.predictionio.tools.export.EventsToFile$.main(
> EventsToFile.scala:71)
>
> at org.apache.predictionio.tools.export.EventsToFile.main(
> EventsToFile.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
>
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
>
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task
> 2.0 in stage 1.0 (TID 6, localhost, executor driver): java.io.IOException:
> Mkdirs failed to create file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000002_0
> (exists=false, cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:455)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:440)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>
> at org.apache.parquet.hadoop.ParquetFileWriter.<init>(
> ParquetFileWriter.java:223)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:312)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:283)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$
> SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)
>
> at org.apache.spark.sql.execution.datasources.
> FileFormatWriter$.org$apache$spark$sql$execution$
> datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
> Driver stacktrace:
>
> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$
> scheduler$DAGScheduler$$failJobAndIndependentStages(
> DAGScheduler.scala:1435)
>
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(
> DAGScheduler.scala:1423)
>
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(
> DAGScheduler.scala:1422)
>
> at scala.collection.mutable.ResizableArray$class.foreach(
> ResizableArray.scala:59)
>
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>
> at org.apache.spark.scheduler.DAGScheduler.abortStage(
> DAGScheduler.scala:1422)
>
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
>
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
>
> at scala.Option.foreach(Option.scala:257)
>
> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(
> DAGScheduler.scala:802)
>
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> doOnReceive(DAGScheduler.scala:1650)
>
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> onReceive(DAGScheduler.scala:1605)
>
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> onReceive(DAGScheduler.scala:1594)
>
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
>
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
>
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)
>
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1.apply$mcV$sp(FileFormatWriter.scala:127)
>
> ... 34 more
>
> Caused by: java.io.IOException: Mkdirs failed to create
> file:/data/_temporary/0/_temporary/attempt_20170605073251_0001_m_000002_0
> (exists=false, cwd=file:/Users/shane/Desktop/Apps/liftiq_prototype/pio-
> engine-classi)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:455)
>
> at org.apache.hadoop.fs.ChecksumFileSystem.create(
> ChecksumFileSystem.java:440)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>
> at org.apache.parquet.hadoop.ParquetFileWriter.<init>(
> ParquetFileWriter.java:223)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:312)
>
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(
> ParquetOutputFormat.java:283)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
>
> at org.apache.spark.sql.execution.datasources.parquet.
> ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:145)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$
> SingleDirectoryWriteTask.<init>(FileFormatWriter.scala:234)
>
> at org.apache.spark.sql.execution.datasources.
> FileFormatWriter$.org$apache$spark$sql$execution$
> datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
>
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$
> write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
> [INFO] [ServerConnector] Stopped ServerConnector@6edf29c1{HTTP/1.1}{
> 127.0.0.1:4040}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 6e4f263e{/stages/stage/kill,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> f1d0004{/jobs/job/kill,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 6f911326{/api,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 74ea46e2{/,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 6ad16c5d{/static,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 2b7962a2{/executors/threadDump/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 423b2b62{/executors/threadDump,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> ff23ae7{/executors/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 127a7272{/executors,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 2938127d{/environment/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 56637cff{/environment,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 76ad6715{/storage/rdd/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 5a1c3cb4{/storage/rdd,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 5ea4300e{/storage/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 2630dbc4{/storage,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 934b52f{/stages/pool/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 142213d5{/stages/pool,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 7b306b9f{/stages/stage/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 53cf9c99{/stages/stage,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 10b1a751{/stages/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> eac3a26{/stages,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 4c58255{/jobs/job/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 1d9bd4da{/jobs/job,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 4dba773d{/jobs/json,null,UNAVAILABLE}
>
> [INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@
> 58ba5b30{/jobs,null,UNAVAILABLE}
>
>
> *Shane Johnson | 801.360.3350 <(801)%20360-3350>*
> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
> <https://www.facebook.com/shane.johnson.71653>
>

Mime
View raw message