hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rahul gidwani <rahul.gidw...@gmail.com>
Subject inotify
Date Fri, 01 Jul 2016 17:28:38 GMT
Hello kind folks,

I was wondering if anyone would be interested in adding a AddBlock Event to
the inotify pipeline?  We were thinking about using Inotify for hdfs
replication to another datacenter.

Right now the problem is with the appends.  We get a notification when an
append starts with an AppendEvent and then we know the file is complete
when we get a CloseEvent.  This would increase our latency considerably.

So if we could add an AddBlockEvent whenever we get an OP_ADD_BLOCK that
would give us the ability to ship blocks.

This would basically give us an ExtendedBlock and then we could ask the
namenode to give us the corresponding LocatedBlock which we could just get
a stream of bytes ready to ship to our destination cluster.

Is this something the community would be interested in?

Thank you
rahul

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message