ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joewang ...@joewang.net>
Subject Re: igfs-meta behavior when node restarts
Date Thu, 01 Jun 2017 17:39:31 GMT
The reads are from a non-IGFS source, but the writes are through IGFS. Spark
uses Hadoop's FileOutputCommitter to write the output to IGFS. I think what
happens is essentially:

- During processing, temporary files are written by each of n executors
running on different nodes to some /data/path/output/_temporary/part-n...
- When the job completes, each of the executor performs the final "commit"
by renaming the files under /data/path/output/_temporary/part-n... to
/data/path/output/part-n... and deletes the _temporary directory.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/igfs-meta-behavior-when-node-restarts-tp13155p13322.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message