hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin P. McCabe" <cmcc...@apache.org>
Subject Re: INotify stability
Date Tue, 22 Sep 2015 16:28:59 GMT
Hi Mohammad,

Like ATM said, HDFS-8965 is an important fix in this area.  We have
found that it prevents cases where INotify tries to read invalid
sequences of bytes (sometimes because the edit log was truncated or
corrupted; other times because it is in the middle of a write).
HDFS-8964 fixes the "attempt to read edit log entries that are still
being written."  Both fixes have value when using INotify.

The other question is how many INotify clients you have
simultaneously?  Right now, INotify works best with a small number of
clients, since it involves doing RPCs to the NameNode.  There is some
work going on to scale out INotify to support many clients in
HDFS-8940.

best,
Colin

On Tue, Sep 15, 2015 at 11:52 AM, Mohammad Islam
<mislam77@yahoo.com.invalid> wrote:
> Hi,We were using INotify feature in one of our internal service. Looks like it creates
a lot of memory pressure on NN. Memory usage goes very high and remains the same causing expensive
GC.
> Did anyone use this feature in any service? Is there any con to setup? We are using latest
CDH.
> Regards,Mohammad
>

Mime
View raw message