libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua Hawkinson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LIBCLOUD-986) List / Iterate container with prefix for Azure Blob service (This support exists in the other drivers already)
Date Wed, 21 Mar 2018 17:43:00 GMT

    [ https://issues.apache.org/jira/browse/LIBCLOUD-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408315#comment-16408315
] 

Joshua Hawkinson commented on LIBCLOUD-986:
-------------------------------------------

Okay, done... Here are the diffs for your review:
{code:java}
diff --git a/libcloud/storage/drivers/azure_blobs.py b/libcloud/storage/drivers/azure_blobs.py
index 82ee03a..2475793 100644
--- a/libcloud/storage/drivers/azure_blobs.py
+++ b/libcloud/storage/drivers/azure_blobs.py
@@ -381,11 +381,12 @@ class AzureBlobsStorageDriver(StorageDriver):
if not params['marker']:
break

-     def iterate_container_objects(self, container):
+     def iterate_container_objects(self, container, ex_prefix=None):
        """
        @inherits: :class:`StorageDriver.iterate_container_objects`
        """
-       params = {'restype': 'container',
+       params = {'prefix': ex_prefix,
+                 'restype': 'container',
                  'comp': 'list',
                  'maxresults': RESPONSES_PER_REQUEST,
                  'include': 'metadata'}
@@ -416,6 +417,22 @@ class AzureBlobsStorageDriver(StorageDriver):
           if not params['marker']:
               break

+    def list_container_objects(self, container, ex_prefix=None):
+      """
+      Return a list of objects for the given container.
+
+      :param container: Container instance.
+      :type container: :class:`Container`
+
+      :param ex_prefix: Only return objects starting with ex_prefix
+      :type ex_prefix: ``str``
+
+      :return: A list of Object instances.
+      :rtype: ``list`` of :class:`Object`
+      """
+      return list(self.iterate_container_objects(container,
+                                                 ex_prefix=ex_prefix))
+
def get_container(self, container_name):
"""
@inherits: :class:`StorageDriver.get_container`
diff --git a/libcloud/test/storage/test_azure_blobs.py b/libcloud/test/storage/test_azure_blobs.py
index 95ab3f1..69bdebb 100644
--- a/libcloud/test/storage/test_azure_blobs.py
+++ b/libcloud/test/storage/test_azure_blobs.py
@@ -447,6 +447,28 @@ class AzureBlobsTests(unittest.TestCase):
         self.assertTrue('content_encoding' in obj.extra)
         self.assertTrue('content_language' in obj.extra)

+    def test_list_container_objects_with_prefix(self):
+        self.mock_response_klass.type = None
+        AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
+
+        container = Container(name='test_container', extra={},
+                              driver=self.driver)
+        objects = self.driver.list_container_objects(container=container,
+        ex_prefix='test_prefix')
+        self.assertEqual(len(objects), 4)
+
+        obj = objects[1]
+        self.assertEqual(obj.name, 'object2.txt')
+        self.assertEqual(obj.hash, '0x8CFB90F1BA8CD8F')
+        self.assertEqual(obj.size, 1048576)
+        self.assertEqual(obj.container.name, 'test_container')
+        self.assertTrue('meta1' in obj.meta_data)
+        self.assertTrue('meta2' in obj.meta_data)
+        self.assertTrue('last_modified' in obj.extra)
+        self.assertTrue('content_type' in obj.extra)
+        self.assertTrue('content_encoding' in obj.extra)
+        self.assertTrue('content_language' in obj.extra)
+
{code}
 

And the before and after test results:
{code:java}
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test session starts ====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items 

libcloud/test/storage/test_azure_blobs.py ....................F................ [100%]

========================================================================= FAILURES =========================================================================
_________________________________________________ AzureBlobsTests.test_list_container_objects_with_prefix
__________________________________________________

self = <libcloud.test.storage.test_azure_blobs.AzureBlobsTests testMethod=test_list_container_objects_with_prefix>

def test_list_container_objects_with_prefix(self):
self.mock_response_klass.type = None
AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2

container = Container(name='test_container', extra={},
driver=self.driver)
objects = self.driver.list_container_objects(container=container,
> ex_prefix='test_prefix')
E TypeError: list_container_objects() got an unexpected keyword argument 'ex_prefix'

libcloud/test/storage/test_azure_blobs.py:457: TypeError
================================================================== 9079 tests deselected ===================================================================
=================================================== 1 failed, 36 passed, 9079 deselected in
3.39 seconds ===================================================
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test session starts ====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items 

libcloud/test/storage/test_azure_blobs.py ..................................... [100%]

================================================================== 9079 tests deselected ===================================================================
======================================================== 37 passed, 9079 deselected in 3.08
seconds ========================================================{code}
Some notes about the unit test for this feature... Ideally you'd have one container that the
Mock HTTP server would be able to return different results for based on the prefix,  I didn't
see a way to simply accomplish this without modifying some of the base mock server code. 
This test method was largely pulled from the other driver's tests, but it seems to meet your
base requirements.  Another note,  I'm not sure why this Jira instance always strips spaces,
even in noformat, or code blocks (Sorry about the formatting).

 

> List / Iterate container with prefix for Azure Blob service (This support exists in the
other drivers already)
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: LIBCLOUD-986
>                 URL: https://issues.apache.org/jira/browse/LIBCLOUD-986
>             Project: Libcloud
>          Issue Type: Improvement
>          Components: Storage
>            Reporter: Joshua Hawkinson
>            Priority: Major
>
> While testing out the libcloud storage abstraction I've come across a curious omission
in driver support. It seems that for many drivers there is additional support for the {{list_container_objects()}} and
{{iterate_container_objects()}} to support a prefix (or {{ex_prefix)}}. 
> This functionality is not present in the Azure driver for some reason.   I've gone
and checked the list_blobs API and it also supports this functionality as seen here:
> [https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs]
>  
> I've attempted a quick and dirty patch that seems to work as well.
> {code}
> azure_blobs.py Line 384
> -    def iterate_container_objects(self, container):
> +    def iterate_container_objects(self, container, ex_prefix=None):
>         """
>         @inherits: :class:`StorageDriver.iterate_container_objects`
>         """
>         params = {'restype': 'container',
>                   'comp': 'list',
> +                  'prefix': ex_prefix,
>                   'maxresults': RESPONSES_PER_REQUEST,
>                   'include': 'metadata'}
> Line 419 (Add new function)
>     def list_container_objects(self, container, ex_prefix=None):
>         """
>         Return a list of objects for the given container.
>     
>         :param container: Container instance.
>         :type container: :class:`Container`
>     
>         :param ex_prefix: Only return objects starting with ex_prefix
>         :type ex_prefix: ``str``
>     
>         :return: A list of Object instances.
>         :rtype: ``list`` of :class:`Object`
>         """
>         return list(self.iterate_container_objects(container,
>                                                    ex_prefix=ex_prefix))
>     
>  {code}
>  
> I've just worked around this at a higher level for now. There isn't any urgency on my
end,  but it would be nice if the abstraction was more consistent so I could remove the workaround
code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message