hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11740) Ozone: Differentiate time interval for different DatanodeStateMachine state tasks
Date Wed, 03 May 2017 17:54:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995325#comment-15995325
] 

Weiwei Yang commented on HDFS-11740:
------------------------------------

Hi [~anu]

Thanks for your quick response, appreciate. I read your comments, I have some thoughts.

bq. The original 90 seconds is the time that datanode would use to read the various containers
and make sure that it is ready to communicate with both SCM and the world.

What I saw the time was spent on state transitions, given the heartbeat time is 30s (default),
it moves "slowly" from one state to another which is not supposed to behave like this. 

bq. Right now, we have some missing pieces, we need to launch a background thread – that
does a directory scan for containers and volumes – so that when SCM asks for container reports
we are ready.

Yes this is probably still a missing piece, but it doesn't seem to relate to this work. If
dn is not ready for container reports, it would just response an error when SCM pulls, so
SCM can try again later. 

bq. So I am doubtful if we will gain anything by accelerating the initial boot time.

Say dn needs time to scan containers/volumes and it takes a while, I am guessing that will
be another (new) state. When we add that, it will be much easier to utilize the work in this
jira so that we can control the time necessary for this task, other than a hard coded heartbeat
interval. This time could be very different on different data size, e.g few seconds on brand
new cluster, minutes on large cluster (I am just saying).

I uploaded a patch, it doesn't have UT and I will add later if the patch is on the right direction.
Some note about the fix, datanode state machine currently uses {{ScmConfigKeys#OZONE_SCM_HEARTBEAT_INTERVAL_SECONDS}}
as the fixed state interval, it now looks like [^statemachine_1.png]; I am proposing to change
to [^statemachine_2.png]. I added a {{EndpointTask}} interface, all end point tasks need to
implement {{long getTaskDuration()}} to decide how long to wait before scheduling the task.
For {{VersionEndpointTask}} and {{RegisterEndpointTask}}, they simply return 0 which means
no lag and directly schedule; for {{HeartbeatEndpointTask}}, it returns the heartbeat interval
so it sends heartbeat with respect to that interval.

Hope it helps. Thanks

> Ozone: Differentiate time interval for different DatanodeStateMachine state tasks
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-11740
>                 URL: https://issues.apache.org/jira/browse/HDFS-11740
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: HDFS-11725-HDFS-7240.001.patch, statemachine_1.png, statemachine_2.png
>
>
> Currently datanode state machine transitioned between tasks in a fixed time interval,
defined by {{ScmConfigKeys#OZONE_SCM_HEARTBEAT_INTERVAL_SECONDS}}, the default value is 30s.
Once datanode is started, it will need 90s before transited to {{Heartbeat}} state, such a
long lag is not necessary. Propose to improve the logic of time interval handling, it seems
only the heartbeat task needs to be scheduled in {{OZONE_SCM_HEARTBEAT_INTERVAL_SECONDS}}
interval, rest should be done without any lagging.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message