hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@apache.org>
Subject Re: inotify
Date Tue, 05 Jul 2016 18:36:32 GMT
I think it makes sense to have an AddBlockEvent.  It seems like we could
provide something like the block ID, block pool ID, and genstamp, as
well as the inode ID and path of the file which the block was added to. 
Clearly, we cannot provide the length, since we don't know how many
bytes the client will write.  Would you mind filing a JIRA for this?

We should also set up inotify events for the snapshot operations at some
point as well.

regards,
Colin

On Fri, Jul 1, 2016, at 10:28, rahul gidwani wrote:
> Hello kind folks,
> 
> I was wondering if anyone would be interested in adding a AddBlock Event
> to
> the inotify pipeline?  We were thinking about using Inotify for hdfs
> replication to another datacenter.
> 
> Right now the problem is with the appends.  We get a notification when an
> append starts with an AppendEvent and then we know the file is complete
> when we get a CloseEvent.  This would increase our latency considerably.
> 
> So if we could add an AddBlockEvent whenever we get an OP_ADD_BLOCK that
> would give us the ability to ship blocks.
> 
> This would basically give us an ExtendedBlock and then we could ask the
> namenode to give us the corresponding LocatedBlock which we could just
> get
> a stream of bytes ready to ship to our destination cluster.
> 
> Is this something the community would be interested in?
> 
> Thank you
> rahul

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org


Mime
View raw message