Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: rahul gidwani <rahul.gidwani@gmail.com>
Date: Fri, 1 Jul 2016 10:28:38 -0700
Message-ID: <CAEpFk2ew8r1j=tXCPx1CX7BTssWgQNMtNu0mm-11-eZENcX0UA@mail.gmail.com>
Subject: inotify
To: hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a1135626e367ddf0536965121
archived-at: Fri, 01 Jul 2016 17:28:45 -0000

--001a1135626e367ddf0536965121
Content-Type: text/plain; charset=UTF-8

Hello kind folks,

I was wondering if anyone would be interested in adding a AddBlock Event to
the inotify pipeline?  We were thinking about using Inotify for hdfs
replication to another datacenter.

Right now the problem is with the appends.  We get a notification when an
append starts with an AppendEvent and then we know the file is complete
when we get a CloseEvent.  This would increase our latency considerably.

So if we could add an AddBlockEvent whenever we get an OP_ADD_BLOCK that
would give us the ability to ship blocks.

This would basically give us an ExtendedBlock and then we could ask the
namenode to give us the corresponding LocatedBlock which we could just get
a stream of bytes ready to ship to our destination cluster.

Is this something the community would be interested in?

Thank you
rahul

--001a1135626e367ddf0536965121--