ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan V." <iveselovs...@gridgain.com>
Subject Re: Igfs PURGE events: do we need them?
Date Wed, 25 Nov 2015 13:54:02 GMT
Hi, Konstantin,
"TRASH" (the name comes from
org.apache.ignite.internal.processors.igfs.IgfsFileInfo#TRASH_ID Java
constant) notion is only applicable to primary (IGFS) file system. This is
a special "folder" that does not have file system path. When IGFS is
running over a secondary Fs, TRASH also exists in the primary IGFS, but
does not exist in the secondary Fs.
In secondary Fs deletion is performed just through the ordinary Fs API. So,
we *do not* employ any assumption regarding the TRASH existence and
behavior in the secondary Fs.

As Dmitriy mentioned above, TRASH in primary Fs is needed for performance
reasons: with it we delete file with only 1 transaction in Meta cache: we
do not do any transactions in Data cache.
(Similar technique is frequently applied frequently in real Fs deletion,
like mv foo /tmp/ && rm -r /tmp/foo/ .)

Currently we have fix of https://issues.apache.org/jira/browse/IGNITE-1679
that makes PURGE events enabled for all files.
I still not quite realize how this functionality will be used by customers,
but now it is repaired: once merged, you will be able to use it.

On Tue, Nov 24, 2015 at 2:52 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> Cos,
>
> The main reason soft delete was added is performance. Without soft-delete,
> the delete operation would have to wait until a file is fully deleted from
> a folder, which may take time.
>
> As far as secondary FS handling it, IGFS does not require a secondary FS,
> so we should account for cases when IGFS is running stand-alone.
>
> Thoughts?
>
> D.
>
> On Mon, Nov 23, 2015 at 11:00 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
>
> > Let me ask a different question: what's the point of having the concept
> of
> > TRASH?
> >
> > Here's an example why I think the 'soft' delete would only complicate
> > thing.
> > Suppose IGFS is sitting on top of HDFS and both have 'Trash' enabled.
> Now,
> > the file is getting soft-deleted from IGFS and is moved to TRASH folder.
> > But
> > in HDFS it is also a move to a place which doesn't have any special
> meaning
> > for HDFS.
> >
> > Even worst, if IFGS TRASH is linked to HDFS .Trash. HDFS has it's own
> > policy
> > on how to clean that up, which is likely to be different from that on
> IGFS.
> > Often enough, HDFS .Trash is simply disabled. This discrepancy is going
> to
> > create a situation when a file should still be in TRASH, but the
> secondary
> > FS
> > has already purged it.
> >
> > And what if yet another secondary file system like S3 has yet another
> > policy
> > around their own trash, which they don't even have, I believe?
> >
> > Where I am going with this is pretty straight forward: let's drop the
> > soft-delete support and let the secondary FS to deal with it. If there's
> no
> > secondary FS configured - the content of deleted file will have to
> > retrieved
> > by other means.
> >
> > Thoughts?
> >   Cos
> >
> > On Fri, Nov 20, 2015 at 07:15PM, Ivan V. wrote:
> > > Hi, dev,
> > > need opinions on the question discussed in
> > > https://issues.apache.org/jira/browse/IGNITE-1679  (IGFS: Purge event
> is
> > > inconsistent).
> > > In short: in Igfs we have "soft" delete that moves the deleted file or
> > > folder to special "TRASH" folder.
> > > Special async worker walks inside TRASH and removes the items
> > permanently.
> > > When an item is completely removed, an event of type
> > > org.apache.ignite.events.EventType#EVT_IGFS_FILE_PURGED  is fired.
> > > But such events are now fired only for files, and only in case if such
> > file
> > > was deleted itself, but not a part of a folder sub-tree. It's quite
> > obvious
> > > that such behavior is not quite consistent, so we should either get rid
> > of
> > > PURGE events at all, or make them consistent.
> > > In the latter case it would be good to have answer to the question:
> what
> > > are real  use cases when we may need the purge events ? (Now they seem
> to
> > > be used in tests only).
> > > If we don't have such real use cases, are there any objections to get
> rid
> > > of the purge events at all?
> > > Thanks in advance.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message