ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: igfs-meta behavior when node restarts
Date Tue, 06 Jun 2017 09:14:22 GMT
Hi,

Ignite performs delete in a "soft" fashion:
1) When "remove" command is executed, we propagate it to the secondary file
system;
2) For IGFS meta cache, we do not remove all records immediately, but
rather execute a single "move" operation and move removed tree to a hidden
"TRASH" folder.
3) "TRASH" folder is cleared periodically - this is what you see in logs.

Removal of trash content should not interfere with normal operations
anyhow, nor should it cause any performance issues. Do you observe some
real slowdown, or you are only concerned with metrics summary?

Vladimir.


On Thu, Jun 1, 2017 at 8:39 PM, joewang <me@joewang.net> wrote:

> The reads are from a non-IGFS source, but the writes are through IGFS.
> Spark
> uses Hadoop's FileOutputCommitter to write the output to IGFS. I think what
> happens is essentially:
>
> - During processing, temporary files are written by each of n executors
> running on different nodes to some /data/path/output/_temporary/part-n...
> - When the job completes, each of the executor performs the final "commit"
> by renaming the files under /data/path/output/_temporary/part-n... to
> /data/path/output/part-n... and deletes the _temporary directory.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/igfs-meta-behavior-when-node-restarts-
> tp13155p13322.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Mime
View raw message