ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antonio Si <antonio...@gmail.com>
Subject using hdfs as secondary file system
Date Fri, 09 Jun 2017 20:57:41 GMT
Hi,

I am new to ignite. I have followed the doc in configuring ignite to use
hdfs as secondary
file system.

I go to spark-shell and when I try to do

df.write.save("test")

where df is a dataframe, I am getting the following Exception:

Driver stacktrace:

at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)

at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)

at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)

at scala.Option.foreach(Option.scala:236)

at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)

at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)

at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)

at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)

at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)

at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:148)

... 62 more

Caused by: java.io.IOException: Generic IGFS error occurred.

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsUtils.cast(HadoopIgfsUtils.java:139)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:341)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:256)

at
org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:455)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:176)

at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:160)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:289)

at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetRelation.scala:94)

at
org.apache.spark.sql.execution.datasources.parquet.ParquetRelation$$anon$3.newInstance(ParquetRelation.scala:286)

at
org.apache.spark.sql.execution.datasources.BaseWriterContainer.newOutputWriter(WriterContainer.scala:129)

at
org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:255)

at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148)

at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)

at org.apache.spark.scheduler.Task.run(Task.scala:89)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: class org.apache.ignite.IgniteCheckedException: Generic IGFS
error occurred.

at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7242)

at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:258)

at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:170)

at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsOutProc.create(HadoopIgfsOutProc.java:373)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:259)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:256)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:324)

... 21 more

Caused by: class org.apache.ignite.igfs.IgfsException: Generic IGFS error
occurred.

at
org.apache.ignite.internal.igfs.common.IgfsControlResponse.throwError(IgfsControlResponse.java:308)

at
org.apache.ignite.internal.igfs.common.IgfsControlResponse.throwError(IgfsControlResponse.java:317)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsOutProc$1.apply(HadoopIgfsOutProc.java:513)

at
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsOutProc$1.apply(HadoopIgfsOutProc.java:507)

at
org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)

at
org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)

at
org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)

at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)

at
org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:352)

at
org.apache.ignite.internal.util.future.GridFutureAdapter$ChainFuture.<init>(GridFutureAdapter.java:556)

at
org.apache.ignite.internal.util.future.GridFutureAdapter.chain(GridFutureAdapter.java:357)

... 25 more


I try to step into the code and debug a little. It can create the folder
test, but it fails when it tries do a create with a path:

igfs://dfmlIgfs@localhost
:10500/user/antoniosi/test/_temporary/0/_temporary/attempt_201706091355_0008_m_000000_0/part-r-00000-bcd32521-bd71-4cc9-9046-f096a7e906d7.gz.parquet

Any suggestions would be helpful.

Thanks.

Antonio.

Mime
View raw message