hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuanbo Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
Date Mon, 18 Jul 2016 06:54:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381802#comment-15381802
] 

Yuanbo Liu edited comment on HDFS-10645 at 7/18/16 6:54 AM:
------------------------------------------------------------

If the cluster grows big enough, it will hit this error:
{code}
org.apache.hadoop.ipc.RemoteException: java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to
increase the size limit.
{code}
Apparently the block report size exceed the limit of PB, and the blocks in the data directory
will be marked as unavailable in namespace. This is a bad sign for the cluster despite of
3 replications. It's will be better if the administrators get the "Max block report size"
in time. So I propose to add this metric to datanode web ui.


was (Author: yuanbo):
If the cluster grows big enough, it will hit this error:
{noformat}
org.apache.hadoop.ipc.RemoteException: java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to
increase the size limit.
{noformat}
Apparently the block report size exceed the limit of PB, and the blocks in the data directory
will be marked as unavailable in namespace. This is a bad sign for the cluster despite of
3 replications. It's will be better if the administrators get the "Max block report size"
in time. So I propose to add this metric to datanode web ui.

> Make block report size as a metric and add this metric to datanode web ui
> -------------------------------------------------------------------------
>
>                 Key: HDFS-10645
>                 URL: https://issues.apache.org/jira/browse/HDFS-10645
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, ui
>            Reporter: Yuanbo Liu
>            Assignee: Yuanbo Liu
>
> Add a new metric called "Max block report size". It's important for administrators to
know the bottleneck of  block report, and the metric is also a good tuning metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message