hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-7343) HDFS smart storage management
Date Fri, 06 Jan 2017 20:40:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15805521#comment-15805521
] 

Anu Engineer edited comment on HDFS-7343 at 1/6/17 8:40 PM:
------------------------------------------------------------

[~zhouwei] Thank you for addressing all the issues I had. The updated design doc looks excellent.
if there are any JIRAs that you need help with, please let me know and I will be happy to
chip in. I am almost at a +1. However, I had some questions and comments. Please treat the
following sections as Questions (Things that I don’t understand and would like to know),
comments (some subjective comments, please feel free to ignore them), nitpick (completely
ignore them, written down to avoid someone else asking the same question later).

h4. Questions

{noformat}
NameNode then stores the data into database. In this way, SSM has no need to maintain state
checkpoints. 
{noformat}
1. I would like to understand the technical trade-offs that was considered in making this
choice. Other applications that do this, like Amabri chooses to store this data in a database
maintained within the application. However, for SSM you are choosing to store them in Namenode.
In fact when I look at the architecture diagram (it is very well drawn, thank you) It looks
to me that it is trivial to have the LevelDB on the SSM side instead of Namenode. So I am
wondering what advantage we have by maintaining more metadata on namenode side.

2. 1.	In Rule Engine section, can we throttle the number of times a particular rule is executed
in a time window? What I am trying to prevent is some kind of flapping where two opposite
rules are triggered continuously. 

3. Also there is a reference {noformat} SSM also stores some data (for example, rules) into
database through NameNode. {noformat} Do we need to store the rules inside Namenode ? Would
it make more sense to store it in SSM itself. The reason why I am asking is that in future
I see that this platform could be leveraged by Hive or HBase. If that is that case, having
an independent rule store might be more interesting than a pure HDFS one.
4. {noformat} HA supporting will be considered later {noformat} I am all for postponing HA
support, but I am not able to understand how we are going to store rules in namenode and ignore
HA. Are we going to say SSM will not work if HA is enabled? Most clusters we see are HA enabled.
However, if we avoid dependencies on Namenode, SSM might work with HA enabled cluster. I cannot
see anything in SSM that inherently prevents it from working with an HA enabled cluster.
5. {noformat} SSM also exports interface (for example, through RESTful API) for administrator
and client to manage SSM or query information.{noformat} May be something for later, but how
do you intend to protect this end point?  Is it going to be Kerberos? Usually I would never
ask a service to consider security till you have the core modules, but in this case the potential
for abuse is very high. I understand that we might not have it in the initial releases of
SSM, but it might be good to think about it.

6. How do we prevent a run-away rule? Let me give you an example; recently I was cleaning
a cluster and decided to run “rm –rf “ on the data directories. Each datanode had more
than 12 million files and just a normal file system operation was taking forever. So a rule
like *file.path matches "/fooA/*.dat"*, might run forever (I am looking at you , Hive). Are
you planning to provide a timeout on the execution of a rule or is it that rule will run until
we reach the end of processing?  if we don’t have timeouts, it might be hard to honor other
rules which want to run at a specific time. Even with multiple threads, you might not be able
to make too much progress and since most of these rules are going to be run against Namenode
and you would have a limited bandwidth to work with.

7.  On the HDFS client querying SSM before writing, what happens if the SSM is down? Will
client wait and retry , potentially making I/O slower and eventually bypassing SSM ? Have
you considered using  Storage Policy Satisfier, HDFS-10285. Even if SSM is down or a client
does not talk to SSM, we could rely on SPS to move the data to the right location. Some of
your storage manager functionality can leverage what is being done in Storage Policy Satisfier.
So can you please clarify how you will handle clients that does not talk to SSM and what happens
to I/O when SSM is down.

h4. Comments 

1. I still share some of the concerns voiced by  [~andrew.wang]. It is going to be challenging
to create a set of static rules for changing conditions of the cluster, especially when works
loads are different. But sometimes we learn surprising things by doing rather than talking.
I would love to learn how this is working out in real world clusters.  If you have any data
to share I would l appreciate it.
h4. Nitpic
{noformat}
 HSM, Cache, SPS, DataNode Disk Balancer(HDFS-1312) and EC to do the actual data manipulation
work.
{noformat}
I think we have accidentally omitted reference to our classic balancer here.







was (Author: anu):
[~zhouwei] Thank you for addressing all the issues I had. The updated design doc looks excellent.
if there are any JIRAs that you need help with, please let me know and I will be happy to
chip in. I am almost at a +1. However, I had some questions and comments. Please treat the
following sections as Questions (Things that I don’t understand and would like to know),
comments (some subjective comments, please feel free to ignore them), nitpick (completely
ignore them, written down to avoid someone else asking the same question later).

h4. Questions

{noformat}
NameNode then stores the data into database. In this way, SSM has no need to maintain state
checkpoints. 
{noformat}
1. I would like to understand the technical trade-offs that was considered in making this
choice. Other applications that do this, like Amabri chooses to store this data in a database
maintained within the application. However, for SSM you are choosing to store them in Namenode.
In fact when I look at the architecture diagram (it is very well drawn, thank you) It looks
to me that it is trivial to have the LevelDB on the SSM side instead of Namenode. So I am
wondering what advantage we have by maintaining more metadata on namenode side.

2. 1.	In Rule Engine section, can we throttle the number of times a particular rule is executed
in a time window? What I am trying to prevent is some kind of flapping where two opposite
rules are triggered continuously. 

3. Also there is a reference {noformat} SSM also stores some data (for example, rules) into
database through NameNode. {noformat} Do we need to store the rules inside Namenode ? Would
it make more sense to store it in SSM itself. The reason why I am asking is that in future
I see that this platform could be leveraged by Hive or HBase. If that is that case, having
an independent rule store might be more interesting than a pure HDFS one.
4. {noformat} HA supporting will be considered later {noformat} I am all for postponing HA
support, but I am not able to understand how we are going to store rules in namenode and ignore
HA. Are we going to say SSM will not work if HA is enabled? Most clusters we see are HA enabled.
However, if we avoid dependencies on Namenode, SSM might work with HA enabled cluster. I cannot
see anything in SSM that inherently prevents it from working with an HA enabled cluster.
5. {noformat} SSM also exports interface (for example, through RESTful API) for administrator
and client to manage SSM or query information.{noformat} May be something for later, but how
do you intend to protect this end point?  Is it going to be Kerberos? Usually I would never
ask a service to consider security till you have to core modules, but in this case the potential
for abuse is very high. I understand that we might not have it in the initial releases of
SSM, but it might be good to think about it.

6. How do we prevent a run-away rule? Let me give you an example, recently I was cleaning
a cluster and decided to run “rm –rf “ on the data directories. Each datanode had more
than 12 million files and just a normal file system operation was taking forever. So a rule
like *file.path matches "/fooA/*.dat"*, might run forever (I am looking at you , Hive). Are
you planning to provide a timeout on the execution of a rule or is it that rule will run until
we reach the end of processing?  if we don’t have timeouts, it might be hard to honor other
rules which want to run at a specific time. Even with multiple threads, you might not be able
to make too much progress and since most of these rules are going to be run against Namenode
and you would have a limited bandwidth to work with.

7.  On the HDFS client querying SSM before writing, what happens if the SSM is down? Will
client wait and retry , potentially making I/O slower and eventually bypassing SSM ? Have
you considered using  Storage Policy Satisfier, HDFS-10285. Even if SSM is down or a client
does not talk to SSM, we could rely on SPS to move the data to the right location. Some of
your storage manager functionality can leverage what is being done in Storage Policy Satisfier.
So can you please clarify how you will handle clients that does not talk to SSM and what happens
to I/O when SSM is down.

h4. Comments 

1. I still share some of the concerns voiced by  [~andrew.wang]. It is going to be challenging
to create a set of static rules for changing conditions of the cluster, especially when works
loads are different. But sometimes we learn surprising things by doing rather than talking.
I would love to learn how this is working out in real world clusters.  If you have any data
to share I would l appreciate it.
h4. Nitpic
{noformat}
 HSM, Cache, SPS, DataNode Disk Balancer(HDFS-1312) and EC to do the actual data manipulation
work.
{noformat}
I think we have accidentally omitted reference to our classic balancer here.






> HDFS smart storage management
> -----------------------------
>
>                 Key: HDFS-7343
>                 URL: https://issues.apache.org/jira/browse/HDFS-7343
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kai Zheng
>            Assignee: Wei Zhou
>         Attachments: HDFS-Smart-Storage-Management-update.pdf, HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and flexible storage
policy engine considering file attributes, metadata, data temperature, storage type, EC codec,
available hardware capabilities, user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution to provide
smart storage management service in order for convenient, intelligent and effective utilizing
of erasure coding or replicas, HDFS cache facility, HSM offering, and all kinds of tools (balancer,
mover, disk balancer and so on) in a large cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message