hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
Date Fri, 16 Aug 2019 16:41:00 GMT

     [ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=296408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296408
]

ASF GitHub Bot logged work on HDDS-1972:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 16/Aug/19 16:40
            Start Date: 16/Aug/19 16:40
    Worklog Time Spent: 10m 
      Work Description: bharatviswa504 commented on issue #1304: HDDS-1972. Provide example
ha proxy with multiple s3 servers back end.
URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522073531
 
 
   On my desktop I see tests are passing.
   Below is the result.
   
   ```
   HW13865:ozones3-haproxy bviswanadham$ ./test.sh 
   Removing network ozones3-haproxy_default
   WARNING: Network ozones3-haproxy_default not found.
   Creating network "ozones3-haproxy_default" with the default driver
   Creating ozones3-haproxy_datanode_1 ... done
   Creating ozones3-haproxy_datanode_2 ... done
   Creating ozones3-haproxy_datanode_3 ... done
   Creating ozones3-haproxy_scm_1      ... done
   Creating ozones3-haproxy_om_1       ... done
   Creating ozones3-haproxy_s3g_1      ... done
   Creating ozones3-haproxy_s3g1_1     ... done
   Creating ozones3-haproxy_s3g2_1     ... done
   Creating ozones3-haproxy_s3g3_1     ... done
   0 datanode is up and healthy (until now)
   0 datanode is up and healthy (until now)
   3 datanodes are up and registered to the scm
   ==============================================================================
   ozones3-haproxy-basic :: Smoketest ozone cluster startup                      
   ==============================================================================
   Check webui static resources                                          | PASS |
   ------------------------------------------------------------------------------
   Start freon testing                                                   | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-basic :: Smoketest ozone cluster startup              | PASS |
   2 critical tests, 2 passed, 0 failed
   2 tests total, 2 passed, 0 failed
   ==============================================================================
   Output:  /tmp/smoketest/ozones3-haproxy/result/robot-ozones3-haproxy-ozones3-haproxy-basic-scm.xml
   ==============================================================================
   ozones3-haproxy-s3                                                            
   ==============================================================================
   ozones3-haproxy-s3.Awss3 :: S3 gateway test with aws cli                      
   ==============================================================================
   File upload and directory list                                        | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Awss3 :: S3 gateway test with aws cli              | PASS |
   1 critical test, 1 passed, 0 failed
   1 test total, 1 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Bucketcreate :: S3 gateway test with aws cli               
   ==============================================================================
   Create bucket which already exists                                    | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Bucketcreate :: S3 gateway test with aws cli       | PASS |
   1 critical test, 1 passed, 0 failed
   1 test total, 1 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Buckethead :: S3 gateway test with aws cli                 
   ==============================================================================
   Head Bucket not existent                                              | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Buckethead :: S3 gateway test with aws cli         | PASS |
   1 critical test, 1 passed, 0 failed
   1 test total, 1 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Bucketlist :: S3 gateway test with aws cli                 
   ==============================================================================
   List buckets                                                          | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Bucketlist :: S3 gateway test with aws cli         | PASS |
   1 critical test, 1 passed, 0 failed
   1 test total, 1 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.MultipartUpload :: S3 gateway test with aws cli            
   ==============================================================================
   Test Multipart Upload                                                 | PASS |
   ------------------------------------------------------------------------------
   Test Multipart Upload Complete                                        | PASS |
   ------------------------------------------------------------------------------
   Test Multipart Upload Complete Entity too small                       | PASS |
   ------------------------------------------------------------------------------
   Test Multipart Upload Complete Invalid part                           | PASS |
   ------------------------------------------------------------------------------
   Test abort Multipart upload                                           | PASS |
   ------------------------------------------------------------------------------
   Test abort Multipart upload with invalid uploadId                     | PASS |
   ------------------------------------------------------------------------------
   Upload part with Incorrect uploadID                                   | PASS |
   ------------------------------------------------------------------------------
   Test list parts                                                       | PASS |
   ------------------------------------------------------------------------------
   Test Multipart Upload with the simplified aws s3 cp API               | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.MultipartUpload :: S3 gateway test with aws cli    | PASS |
   9 critical tests, 9 passed, 0 failed
   9 tests total, 9 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Objectcopy :: S3 gateway test with aws cli                 
   ==============================================================================
   Copy Object Happy Scenario                                            | PASS |
   ------------------------------------------------------------------------------
   Copy Object Where Bucket is not available                             | PASS |
   ------------------------------------------------------------------------------
   Copy Object Where both source and dest are same with change to sto... | PASS |
   ------------------------------------------------------------------------------
   Copy Object Where Key not available                                   | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Objectcopy :: S3 gateway test with aws cli         | PASS |
   4 critical tests, 4 passed, 0 failed
   4 tests total, 4 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Objectdelete :: S3 gateway test with aws cli               
   ==============================================================================
   Delete file with s3api                                                | PASS |
   ------------------------------------------------------------------------------
   Delete file with s3api, file doesn't exist                            | PASS |
   ------------------------------------------------------------------------------
   Delete dir with s3api                                                 | PASS |
   ------------------------------------------------------------------------------
   Delete file with s3api, file doesn't exist, prefix of a real file     | PASS |
   ------------------------------------------------------------------------------
   Delete file with s3api, bucket doesn't exist                          | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Objectdelete :: S3 gateway test with aws cli       | PASS |
   5 critical tests, 5 passed, 0 failed
   5 tests total, 5 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Objectmultidelete :: S3 gateway test with aws cli          
   ==============================================================================
   Delete file with multi delete                                         | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Objectmultidelete :: S3 gateway test with aws cli  | PASS |
   1 critical test, 1 passed, 0 failed
   1 test total, 1 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Objectputget :: S3 gateway test with aws cli               
   ==============================================================================
   Put object to s3                                                      | PASS |
   ------------------------------------------------------------------------------
   Get object from s3                                                    | PASS |
   ------------------------------------------------------------------------------
   Get Partial object from s3 with both start and endoffset              | PASS |
   ------------------------------------------------------------------------------
   Get Partial object from s3 with both start and endoffset(start off... | PASS |
   ------------------------------------------------------------------------------
   Get Partial object from s3 with both start and endoffset(end offse... | PASS |
   ------------------------------------------------------------------------------
   Get Partial object from s3 with only start offset                     | PASS |
   ------------------------------------------------------------------------------
   Get Partial object from s3 with both start and endoffset which are... | PASS |
   ------------------------------------------------------------------------------
   Get Partial object from s3 to get last n bytes                        | PASS |
   ------------------------------------------------------------------------------
   Incorrect values for end and start offset                             | PASS |
   ------------------------------------------------------------------------------
   Zero byte file                                                        | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Objectputget :: S3 gateway test with aws cli       | PASS |
   10 critical tests, 10 passed, 0 failed
   10 tests total, 10 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3.Webui :: S3 gateway web ui test                            
   ==============================================================================
   S3 Gateway Web UI                                                     | PASS |
   ------------------------------------------------------------------------------
   ozones3-haproxy-s3.Webui :: S3 gateway web ui test                    | PASS |
   1 critical test, 1 passed, 0 failed
   1 test total, 1 passed, 0 failed
   ==============================================================================
   ozones3-haproxy-s3                                                    | PASS |
   34 critical tests, 34 passed, 0 failed
   34 tests total, 34 passed, 0 failed
   ==============================================================================
   Output:  /tmp/smoketest/ozones3-haproxy/result/robot-ozones3-haproxy-ozones3-haproxy-s3-scm.xml
   Stopping ozones3-haproxy_datanode_2 ... done
   Stopping ozones3-haproxy_s3g3_1     ... done
   Stopping ozones3-haproxy_om_1       ... done
   Stopping ozones3-haproxy_s3g1_1     ... done
   Stopping ozones3-haproxy_s3g2_1     ... done
   Stopping ozones3-haproxy_s3g_1      ... done
   Stopping ozones3-haproxy_datanode_1 ... done
   Stopping ozones3-haproxy_datanode_3 ... done
   Stopping ozones3-haproxy_scm_1      ... done
   Removing ozones3-haproxy_datanode_2 ... done
   Removing ozones3-haproxy_s3g3_1     ... done
   Removing ozones3-haproxy_om_1       ... done
   Removing ozones3-haproxy_s3g1_1     ... done
   Removing ozones3-haproxy_s3g2_1     ... done
   Removing ozones3-haproxy_s3g_1      ... done
   Removing ozones3-haproxy_datanode_1 ... done
   Removing ozones3-haproxy_datanode_3 ... done
   Removing ozones3-haproxy_scm_1      ... done
   Removing network ozones3-haproxy_default
   Log:     /Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3-haproxy/result/log.html
   Report:  /Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3-haproxy/result/report.html
   ```
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 296408)
    Time Spent: 1.5h  (was: 1h 20m)

> Provide example ha proxy with multiple s3 servers back end.
> -----------------------------------------------------------
>
>                 Key: HDDS-1972
>                 URL: https://issues.apache.org/jira/browse/HDDS-1972
>             Project: Hadoop Distributed Data Store
>          Issue Type: New Feature
>            Reporter: Bharat Viswanadham
>            Assignee: Bharat Viswanadham
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In this Jira, we shall provide docker-compose files where we start 3 s3 gateway servers,
and ha-proxy is used to load balance these S3 Gateway Servers.
>  
> In this Jira, all are proxy configurations are hardcoded, we can make improvements to
scale and automatically configure with environment variables as a future improvement. This
is just a starter example.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message