hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei-Chiu Chuang <weic...@apache.org>
Subject Unnecessary FsDatasetImpl lock in FsVolumeImpl#getDfsUsed?
Date Mon, 19 Dec 2016 19:30:52 GMT
I am trying to understand the lock in FsVolumeImpl#getDfsUsed().
As many of you know, a busy datanode can occasionally block on the call to
FsVolumeImpl#getDfsUsed(). This method is used in a number of places,
especially used in storage reports in heartbeat messages.

There have been some attempt to make this situation better, for example
HDFS-7999 stops a writer if it blocks FsDatasetImpl for too long,
preventing heartbeat messages.

However, when I looked at the code closer, I started to wonder if
FsVolumeImpl#getDfsUsed() needs FsDatasetImpl lock at all.
Because the BlockPoolSlice hash table bpSlices is a ConcurrentHashMap so
iteration over this object doesn’t need a lock. It also seems to be an
overkill because the operation does not modify BlockPoolSlice object at
all. I do not see extra benefits locking on FsDatasetImpl object.


Wei-Chiu Chuang
A very happy Hadoop contributor

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message