Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 19 Oct 2016 05:10:59 +0000 (UTC)
From: "Wei Zhou (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12752632.1415080928000.25244.1476853859849@Atlassian.JIRA>
In-Reply-To: <JIRA.12752632.1415080928000@Atlassian.JIRA>
References: <JIRA.12752632.1415080928000@Atlassian.JIRA> <JIRA.12752632.1415080928921@arcas>
Subject: [jira] [Commented] (HDFS-7343) HDFS smart storage management
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 19 Oct 2016 05:11:07 -0000


    [ https://issues.apache.org/jira/browse/HDFS-7343?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15587=
704#comment-15587704 ]=20

Wei Zhou commented on HDFS-7343:
--------------------------------

Thanks [~anu] for reviewing the design document and great comments!
For your comments:
{quote}
1. Is this service attempting to become the answer for all administrative p=
roblems of HDFS? In other words, Is this service is trying to be a catch al=
l service?
I am not able to see what is common between caching and file placement and =
between running distcp for remote replication and running balancer and disk=
 balancer.
{quote}
For long run, SSM is going to provide user an end-to-end storage management=
 automation solution. Any facility can be used in this project towards the =
solution. The use cases and examples listed in the document just give examp=
les of the possible scenarios where SSM can be used and what can SSM do. SS=
M can help from different angles by using these facilities.
{quote}
But then in the latter parts of the document we drag in distcp, disk balanc=
er and balancer issues. SSM might be the right place for these in the long =
run, but my suggestion is to focus on the core parts of the service and the=
n extend it to other things once the core is stabilized.
{quote}
You are absolutely right, we have to be focused on implementing the core pa=
rt/module now instead of involving too much beyond at the same time, it's t=
he basis of other functions.

{quote}
2. Do we need a new rules language =E2=80=93 Would you please consider usin=
g a language which admins will already know, for example, if we can write t=
hese rules in python or even JavaScript, you don=E2=80=99t need to invent a=
 whole new language. Every time I have to configure Kerberos rules, I have =
to lookup the mini-regex meanings. I am worried that this little rule langu=
age will blow up and once it is in, this is something that we will need to =
support for the long term.
{quote}
Yes, it's a very good question and we do have thought about it before. We a=
imed at providing administrator/user a simple and specific rule language wi=
thout touching too much besides the rule logic itself. In fact, a rule is v=
ery simple that only have to declare when and which action to be implied on=
 some objects (can be a file, node, etc.). A general and systematic languag=
e like python or java script maybe too heavy for defining a rule.=20

{quote}
3. In this model we are proposing a push model where the datanode and Namen=
ode pushes data to some kafka endpoint. I would prefer if namenode and data=
node was not aware of this service at all. This service can easily connect =
to namenode and read almost all the data which is needed. If you need extra=
 RPC to be added in datanode and namenode that would an option too. But cre=
ating a dependency from namenode and all datanodes in a cluster seems to be=
 something that you want to do after very careful consideration. If we move=
 to a pull model, you might not even need kafka service to be running in th=
e initial versions.
{quote}
Good point! This is also a very good way to implement SSM.
If using pull model, the advantage are:
   (1) No dependency on Kafka service, and it=E2=80=99s indeed much easier =
for development, testing and deployment.
   (2) Closer relationship with HDFS which may be able to support features =
that cannot be done in the model described in the design document.=20
The disadvantage are:=20
(1) It may have potential performance issue. SSM have to know the messages =
timely in order to work effectively. In order to decrease the overhead of g=
etting messages, SSM have to query NameNodes for the messages on a very hig=
h frequency all the time. It=E2=80=99s also very hard for SSM to query Data=
Nodes one by one to get messages in a large scale cluster.=20
(2) It simplifies the process of message collecting and management. If SSM =
stopped by user or crashed while the HDFS cluster is still working, then me=
ssages from nodes shall be lost without Kafka, and it=E2=80=99s not friendl=
y for SSM to collect historical data.=20
Above, both of the models are workable and we may need more discussion on i=
t. What=E2=80=99s your opinion?


> HDFS smart storage management
> -----------------------------
>
>                 Key: HDFS-7343
>                 URL: https://issues.apache.org/jira/browse/HDFS-7343
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kai Zheng
>            Assignee: Wei Zhou
>         Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and=
 flexible storage policy engine considering file attributes, metadata, data=
 temperature, storage type, EC codec, available hardware capabilities, user=
/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solut=
ion to provide smart storage management service in order for convenient, in=
telligent and effective utilizing of erasure coding or replicas, HDFS cache=
 facility, HSM offering, and all kinds of tools (balancer, mover, disk bala=
ncer and so on) in a large cluster.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org