hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Giovanni Matteo Fumarola (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-3663) Federation State and Policy Store (DBMS implementation)
Date Wed, 19 Apr 2017 21:27:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975555#comment-15975555
] 

Giovanni Matteo Fumarola edited comment on YARN-3663 at 4/19/17 9:27 PM:
-------------------------------------------------------------------------

Thanks [~curino] for the accurate review. Let me go in order.

Conf
1. Every process should have at most 1 connection open to the StateStore except for Router.
It has 10. This is for performance and scalability reason of the Database. FederationInterceptor
open one connection for each application running in the node. More than that can cause a possible
DoS to StateStore.
2. Done.

SQLFederationStateStore
1. Done.
2. Done.
3. Done.
4. Done.
5. When I do auto-format from eclipse by using hadoop-format.xml it gives me it in 2 lines.
  
6. Done.
7.  We can improve/reduce the amount of code later under a different Jira. I am quite confident
in this part and I would avoid to change it.
8. Done.
9. We can improve/reduce the amount of code later under a different Jira. I am quite confident
in this part and I would avoid to change it.
10. That is a safe check, since admin's cluster can setup their own database.
11. Done.
12. Done.
13. Done.
14. Done.
15. Done.

FederationStateStoreUtils
1. Done

HSQLDBFederationStateStore
1. Done 
2. I tried to move in a file but I am adding a lot of complexity and I did not find a way
to let it work. I tried to format in SQL standard inside the file itself but the autoformat
messes it up.
3. Done.
4. Fixed it - good catch.
5. In that way I only insert a new application. That code is like if (! app exists) {add app}.
6. Done.

Stored procedure
1. Done.
2. Done.
3. Increased to 6000.
4. My bad, it was a for a performance test - introduced in the patch by mistake.
5. We can DEREGISTER it or put it in a LOST state.
6. We did not run in any issues with those settings. I would keep like this.
7. Let's open a JIRA about it.
8. Done.
9. Increase to 6000.
10. As I said before, let's keep this one since we have one stored procedure for each function.
We can reduce/improve later.

Top level concerns:
1. The current design was not edited over the last year. However in case we need to evolve
the schema we will need to add them.
2. This cannot happen because we have input validators.
3. For that scenario, we have to add only the stored procedure for the specific database.
4. I edited some tests to cover some scenario we did not check. However, the current state
covers all the possible scenarios.


was (Author: giovanni.fumarola):
Thanks [~curino] for the accurate review. Let me go in order.

Conf
1. Every process should have at most 1 connection open to the StateStore except for Router.
It has 10. This is for performance and scalability reason of the Database. FederationInterceptor
open one connection for each application running in the node. More than that can cause a possible
DoS to StateStore.
2. Done.

SQLFederationStateStore
1. Done.
2. Done.
3. Done.
4. Done.
5. When I do auto-format from eclipse by using hadoop-format.xml it gives me it in 2 lines.
  
6. Done.
7.  We can improve/reduce the amount of code later under a different Jira. I am quite confident
in this part and I would avoid to change it.
8. Done.
9. We can improve/reduce the amount of code later under a different Jira. I am quite confident
in this part and I would avoid to change it.
10. That is a safe check, since admin's cluster can setup their own database.
11. Done.
12. Done.
13. Done.
14. Done.
15. Done.

FederationStateStoreUtils
1. Done

HSQLDBFederationStateStore
1. Done 
2. I tried to move in a file but I am adding a lot of complexity and I did not find a way
to let it work. I tried to format in SQL standard inside the file itself but the autoformat
messes it up.
3. Done.
4. Fixed it - good catch.
5. In that way I only insert a new application. That code is like if (! app exists){add app}.
6. Done.

Stored procedure
1. Done.
2. Done.
3. Increased to 6000.
4. My bad, it was a for a performance test - introduced in the patch by mistake.
5. We can DEREGISTER it or put it in a LOST state.
6. We did not run in any issues with those settings. I would keep like this.
7. Let's open a JIRA about it.
8. Done.
9. Increase to 6000.
10. As I said before, let's keep this one since we have one stored procedure for each function.
We can reduce/improve later.

Top level concerns:
1. The current design was not edited over the last year. However in case we need to evolve
the schema we will need to add them.
2. This cannot happen because we have input validators.
3. For that scenario, we have to add only the stored procedure for the specific database.
4. I edited some tests to cover some scenario we did not check. However, the current state
covers all the possible scenario.

> Federation State and Policy Store (DBMS implementation)
> -------------------------------------------------------
>
>                 Key: YARN-3663
>                 URL: https://issues.apache.org/jira/browse/YARN-3663
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>    Affects Versions: YARN-2915
>            Reporter: Giovanni Matteo Fumarola
>            Assignee: Giovanni Matteo Fumarola
>         Attachments: YARN-3663-YARN-2915.v1.patch, YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch
>
>
> This JIRA tracks a SQL-based implementation of the Federation State and Policy Store,
which implements YARN-3662 APIs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message