Mailing-List: contact user-help@jclouds.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@jclouds.apache.org
To: "user@jclouds.apache.org" <user@jclouds.apache.org>
From: Ryan Shoemaker <ryan.shoemaker@enterprisedb.com>
Subject: OpenStack identity endpoint and serviceCatalog usage
Message-ID: <d88d61c5-22c1-a797-7e40-0aade7ab2d4d@enterprisedb.com>
Date: Thu, 5 Jan 2017 14:33:32 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
 Gecko/20100101 Thunderbird/45.5.1
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------68D910B48DB20E1723BC0AC8"
archived-at: Thu, 05 Jan 2017 19:30:26 -0000

This is a multi-part message in MIME format.
--------------68D910B48DB20E1723BC0AC8
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I'm hoping to get a better understanding of how jclouds discovers and 
decides which identity endpoint to use.  Feel free to redirect me to 
docs if this is already written up somewhere.

In my case, we have a community Mitaka installation running and an app 
that uses jclouds v2.0.0.  Initially, nothing identity related was 
working because Mitaka wasn't configured to respond to V2.0 Identity API 
requests - only V3, which isn't supported by jclouds yet.  I didn't 
setup/configure this installation of OpenStack, so I'm not sure if 
that's a common issue or not - I don't know if the V2.0 Identity API is 
normally turned off in a fresh install or what.  It's entirely possible 
that there's something misconfigured in there causing the behavior I 
describe below.  Anyway, once the V2.0 Identity endpoint was enabled, I 
turned on tracing and can see that even though I create my context 
builder explicitly using the V2.0 endpoint URL, jclouds switches over 
and uses whatever comes back in the 'serviceCatalog' param in the token 
response.  In my case, that happens to point to the V3 Identity 
endpoint.  At that point, jclouds fails to work because it is making 
V2.0 requests on the V3 endpoints.  Here's some tracing showing that in 
more detail:

FINE: >> "{"auth":{"passwordCredentials":{"username":"someuser","password":"somepass"},"tenantName":"sometenant"}}"
FINE: >> POST http://mitaka-host.com:5000/v2.0/tokens HTTP/1.1
FINE: >> Accept: application/json
FINE: >> Content-Type: application/json
FINE: >> Content-Length: 108

FINE: << HTTP/1.1 200 OK
FINE: << Vary: X-Auth-Token
FINE: << Date: Wed, 04 Jan 2017 20:45:54 GMT
FINE: << Keep-Alive: timeout=5, max=97
FINE: << Connection: Keep-Alive
FINE: << x-openstack-request-id: req-390cebba-7cec-472c-88f4-8be6f6a0c236
FINE: << Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
FINE: << Content-Type: application/json
FINE: << Content-Length: 3620
FINE: <<
"{"access":{
    "token":{
       "issued_at":"2017-01-04T20:45:54.000000Z",
       "expires":"2017-01-04T21:45:54Z",
       "id":"gAAAAABYbV8C5pTf30IRs7ybxmbflLk539RDldDU6Q7DBHTTXTY79pabgotPSfy41Nt2nZ82isP0RxsfRuxWRlb1fnlWy1E8zlQdM3xZbrkjP27gSlmLcsV298v-CH8R4GY7wnIvfJjO0MO4DMAgFvvMIA12ByG4kqIJHZ92VV_F6Vby33gR574",
       "tenant":{
          "description":"somedescription",
          "enabled":true,
          "id":"6408ffc2d9034498bc764c230a90e591",
          "name":"sometenant"
       },
       "audit_ids":[
          "sEaG03ZdRXSeRMOr9GVCDA"
       ]
    },
    "serviceCatalog":[
       {
          "endpoints":[
             {
                "adminURL":"http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591",
                "region":"uk",
                "internalURL":"http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591",
                "id":"9d531ab7259e44738c953ec8766b0bc4",
                "publicURL":"http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591"
             }
          ],
          "endpoints_links":[

          ],
          "type":"compute",
          "name":"nova"
       },
       {
          "endpoints":[
             {
                "adminURL":"http://mitaka-host.com:9696",
                "region":"uk",
                "internalURL":"http://mitaka-host.com:9696",
                "id":"1312e551abfe4c6d886e1a471f92bddd",
                "publicURL":"http://mitaka-host.com:9696"
             }
          ],
          "endpoints_links":[

          ],
          "type":"network",
          "name":"neutron"
       },
       {
          "endpoints":[
             {
                "adminURL":"http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591",
                "region":"uk",
                "internalURL":"http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591",
                "id":"02fb1ff3b41a40ceb4cb28abfa5bd547",
                "publicURL":"http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591"
             }
          ],
          "endpoints_links":[

          ],
          "type":"volumev2",
          "name":"cinderv2"
       },
       {
          "endpoints":[
             {
                "adminURL":"http://mitaka-host.com:9292",
                "region":"uk",
                "internalURL":"http://mitaka-host.com:9292",
                "id":"3681888e8b62421f81e93d4270201520",
                "publicURL":"http://mitaka-host.com:9292"
             }
          ],
          "endpoints_links":[

          ],
          "type":"image",
          "name":"glance"
       },
       {
          "endpoints":[
             {
                "adminURL":"http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591",
                "region":"uk",
                "internalURL":"http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591",
                "id":"252797f5a4bb44c0844bff5586a23e16",
                "publicURL":"http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591"
             }
          ],
          "endpoints_links":[

          ],
          "type":"volume",
          "name":"cinder"
       },
       {
          "endpoints":[
             {
                "adminURL":"http://mitaka-host.com:8080/v1",
                "region":"uk",
                "internalURL":"http://mitaka-host.com:8080/v1/AUTH_6408ffc2d9034498bc764c230a90e591",
                "id":"32f4fa49d90e4ad58b2b694bc1aab1ed",
                "publicURL":"http://mitaka-host.com:8080/v1/AUTH_6408ffc2d9034498bc764c230a90e591"
             }
          ],
          "endpoints_links":[

          ],
          "type":"object-store",
          "name":"swift"
       },
{ "endpoints":[ { "adminURL":"http://mitaka-host.com:35357/v3", 
"region":"uk", "internalURL":"http://mitaka-host.com:5000/v3", 
"id":"23ace8e63eab431d81ccf5bfabc1ad53", 
"publicURL":"http://mitaka-host.com:5000/v3" } ], "endpoints_links":[ ], 
"type":"identity", "name":"keystone" }     ],
    "user":{
       "username":"someuser",
       "roles_links":[

       ],
       "id":"a0da46b26af74a44a7c89cab6d9a6439",
       "roles":[
          {
             "name":"admin"
          }
       ],
       "name":"someuser"
    },
    "metadata":{
       "is_admin":0,
       "roles":[
          "dca7c9a77e294f97a619e2b7da246550"
       ]
    }
}
}"


FINE: >> GET http://mitaka-host.com:35357/v3/tenants?name=sometenant 
HTTP/1.1
FINE: >> Accept: application/json
FINE: >> X-Auth-Token: gAAAAABYbV8C5pTf30IRs7ybxmbflLk539RDldDU6Q7DBHTTXTY79pabgotPSfy41Nt2nZ82isP0RxsfRuxWRlb1fnlWy1E8zlQdM3xZbrkjP27gSlmLcsV298v-CH8R4GY7wnIvfJjO0MO4DMAgFvvMIA12ByG4kqIJHZ92VV_F6Vby33gR574

FINE: << HTTP/1.1 404 Not Found
FINE: << Vary: X-Auth-Token
FINE: << Date: Wed, 04 Jan 2017 20:45:55 GMT
FINE: << Keep-Alive: timeout=5, max=100
FINE: << Connection: Keep-Alive
FINE: << x-openstack-request-id: req-94bea180-8447-485a-8485-bf3f57e8d225
FINE: << Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
FINE: << Content-Type: application/json
FINE: << Content-Length: 93
FINE: << "{"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}"


Again, even though I created the jclouds ContextBuilder with the V2.0 
endpoint, it looks like jclouds read the identity admin endpoint from 
the 'serviceCatalog' in the response and then switched to using it, even 
though it is a V3 endpoint.  Is this normal expected behavior for 
jclouds?  I'm guessing it has more to do with how Mitaka is configured, 
but I don't see how jclouds would be able to talk to newer versions of 
OpenStack unless they disable the V3 Identity endpoint or configure it 
in such a way that the V2.0 endpoint is exposed for jclouds to see it.  
Are there any READMEs on how to use jclouds keystone APIs with newer 
versions of OpenStack?

In general, it seems like OpenStack is moving more towards a model where 
API versioning is supposed to be discovered by clients.  For example: 
https://review.openstack.org/#/c/130669/, specifically the patch comment 
that says "if some one is using third party client, it will break. Only 
keystone client handles version discovery. Keystone doesn't even 
document how to handle version discovery, so any third party clients 
such as jcloud will likely fail"

Is there any update on jclouds adding support for the V3 Identity API?

Thanks,

--Ryan

--------------68D910B48DB20E1723BC0AC8
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Hi,<br>
    <br>
    I'm hoping to get a better understanding of how jclouds discovers
    and decides which identity endpoint to use.  Feel free to redirect
    me to docs if this is already written up somewhere.<br>
    <br>
    In my case, we have a community Mitaka installation running and an
    app that uses jclouds v2.0.0.  Initially, nothing identity related
    was working because Mitaka wasn't configured to respond to V2.0
    Identity API requests - only V3, which isn't supported by jclouds
    yet.  I didn't setup/configure this installation of OpenStack, so
    I'm not sure if that's a common issue or not - I don't know if the
    V2.0 Identity API is normally turned off in a fresh install or
    what.  It's entirely possible that there's something misconfigured
    in there causing the behavior I describe below.  Anyway, once the
    V2.0 Identity endpoint was enabled, I turned on tracing and can see
    that even though I create my context builder explicitly using the
    V2.0 endpoint URL, jclouds switches over and uses whatever comes
    back in the 'serviceCatalog' param in the token response.  In my
    case, that happens to point to the V3 Identity endpoint.  At that
    point, jclouds fails to work because it is making V2.0 requests on
    the V3 endpoints.  Here's some tracing showing that in more detail:<br>
     <br>
    <pre style="background-color:#ffffff;color:#000000;font-family:'Menlo';font-size:9.0pt;">FINE: &gt;&gt; "{"auth":{"passwordCredentials":{"username":"someuser","password":"somepass"},"tenantName":"sometenant"}}"
<font color="#ff0000">FINE: &gt;&gt; POST <a class="moz-txt-link-freetext" href="http://os-mitaka-controller.ox.uk.enterprisedb.com:5000/v2.0/tokens">http://mitaka-host.com:5000/v2.0/tokens</a> HTTP/1.1</font>
FINE: &gt;&gt; Accept: application/json
FINE: &gt;&gt; Content-Type: application/json
FINE: &gt;&gt; Content-Length: 108

FINE: &lt;&lt; HTTP/1.1 200 OK
FINE: &lt;&lt; Vary: X-Auth-Token
FINE: &lt;&lt; Date: Wed, 04 Jan 2017 20:45:54 GMT
FINE: &lt;&lt; Keep-Alive: timeout=5, max=97
FINE: &lt;&lt; Connection: Keep-Alive
FINE: &lt;&lt; x-openstack-request-id: req-390cebba-7cec-472c-88f4-8be6f6a0c236
FINE: &lt;&lt; Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
FINE: &lt;&lt; Content-Type: application/json
FINE: &lt;&lt; Content-Length: 3620
FINE: &lt;&lt; 
"{"access":{
   "token":{
      "issued_at":"2017-01-04T20:45:54.000000Z",
      "expires":"2017-01-04T21:45:54Z",
      "id":"gAAAAABYbV8C5pTf30IRs7ybxmbflLk539RDldDU6Q7DBHTTXTY79pabgotPSfy41Nt2nZ82isP0RxsfRuxWRlb1fnlWy1E8zlQdM3xZbrkjP27gSlmLcsV298v-CH8R4GY7wnIvfJjO0MO4DMAgFvvMIA12ByG4kqIJHZ92VV_F6Vby33gR574",
      "tenant":{
         "description":"somedescription",
         "enabled":true,
         "id":"6408ffc2d9034498bc764c230a90e591",
         "name":"sometenant"
      },
      "audit_ids":[
         "sEaG03ZdRXSeRMOr9GVCDA"
      ]
   },
   "serviceCatalog":[
      {
         "endpoints":[
            {
               "adminURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591"</a>,
               "region":"uk",
               "internalURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591"</a>,
               "id":"9d531ab7259e44738c953ec8766b0bc4",
               "publicURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8774/v2.1/6408ffc2d9034498bc764c230a90e591"</a>
            }
         ],
         "endpoints_links":[

         ],
         "type":"compute",
         "name":"nova"
      },
      {
         "endpoints":[
            {
               "adminURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:9696">"http://mitaka-host.com:9696"</a>,
               "region":"uk",
               "internalURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:9696">"http://mitaka-host.com:9696"</a>,
               "id":"1312e551abfe4c6d886e1a471f92bddd",
               "publicURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:9696">"http://mitaka-host.com:9696"</a>
            }
         ],
         "endpoints_links":[

         ],
         "type":"network",
         "name":"neutron"
      },
      {
         "endpoints":[
            {
               "adminURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591"</a>,
               "region":"uk",
               "internalURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591"</a>,
               "id":"02fb1ff3b41a40ceb4cb28abfa5bd547",
               "publicURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8776/v2/6408ffc2d9034498bc764c230a90e591"</a>
            }
         ],
         "endpoints_links":[

         ],
         "type":"volumev2",
         "name":"cinderv2"
      },
      {
         "endpoints":[
            {
               "adminURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:9292">"http://mitaka-host.com:9292"</a>,
               "region":"uk",
               "internalURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:9292">"http://mitaka-host.com:9292"</a>,
               "id":"3681888e8b62421f81e93d4270201520",
               "publicURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:9292">"http://mitaka-host.com:9292"</a>
            }
         ],
         "endpoints_links":[

         ],
         "type":"image",
         "name":"glance"
      },
      {
         "endpoints":[
            {
               "adminURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591"</a>,
               "region":"uk",
               "internalURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591"</a>,
               "id":"252797f5a4bb44c0844bff5586a23e16",
               "publicURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8776/v1/6408ffc2d9034498bc764c230a90e591"</a>
            }
         ],
         "endpoints_links":[

         ],
         "type":"volume",
         "name":"cinder"
      },
      {
         "endpoints":[
            {
               "adminURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8080/v1">"http://mitaka-host.com:8080/v1"</a>,
               "region":"uk",
               "internalURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8080/v1/AUTH_6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8080/v1/AUTH_6408ffc2d9034498bc764c230a90e591"</a>,
               "id":"32f4fa49d90e4ad58b2b694bc1aab1ed",
               "publicURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:8080/v1/AUTH_6408ffc2d9034498bc764c230a90e591">"http://mitaka-host.com:8080/v1/AUTH_6408ffc2d9034498bc764c230a90e591"</a>
            }
         ],
         "endpoints_links":[

         ],
         "type":"object-store",
         "name":"swift"
      },
<font color="#ff0000">      {
         "endpoints":[
            {
               "adminURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:35357/v3">"http://mitaka-host.com:35357/v3"</a>,
               "region":"uk",
               "internalURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:5000/v3">"http://mitaka-host.com:5000/v3"</a>,
               "id":"23ace8e63eab431d81ccf5bfabc1ad53",
               "publicURL":<a class="moz-txt-link-rfc2396E" href="http://mitaka-host.com:5000/v3">"http://mitaka-host.com:5000/v3"</a>
            }
         ],
         "endpoints_links":[

         ],
         "type":"identity",
         "name":"keystone"
      }
</font>   ],
   "user":{
      "username":"someuser",
      "roles_links":[

      ],
      "id":"a0da46b26af74a44a7c89cab6d9a6439",
      "roles":[
         {
            "name":"admin"
         }
      ],
      "name":"someuser"
   },
   "metadata":{
      "is_admin":0,
      "roles":[
         "dca7c9a77e294f97a619e2b7da246550"
      ]
   }
}
}"


<font color="#ff0000">FINE: &gt;&gt; GET <a class="moz-txt-link-freetext" href="http://os-mitaka-controller.ox.uk.enterprisedb.com:35357/v3/tenants?name=ark-dev">http://mitaka-host.com:35357/v3/tenants?name=sometenant</a> HTTP/1.1</font>
FINE: &gt;&gt; Accept: application/json
FINE: &gt;&gt; X-Auth-Token: gAAAAABYbV8C5pTf30IRs7ybxmbflLk539RDldDU6Q7DBHTTXTY79pabgotPSfy41Nt2nZ82isP0RxsfRuxWRlb1fnlWy1E8zlQdM3xZbrkjP27gSlmLcsV298v-CH8R4GY7wnIvfJjO0MO4DMAgFvvMIA12ByG4kqIJHZ92VV_F6Vby33gR574

FINE: &lt;&lt; HTTP/1.1 404 Not Found
FINE: &lt;&lt; Vary: X-Auth-Token
FINE: &lt;&lt; Date: Wed, 04 Jan 2017 20:45:55 GMT
FINE: &lt;&lt; Keep-Alive: timeout=5, max=100
FINE: &lt;&lt; Connection: Keep-Alive
FINE: &lt;&lt; x-openstack-request-id: req-94bea180-8447-485a-8485-bf3f57e8d225
FINE: &lt;&lt; Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
FINE: &lt;&lt; Content-Type: application/json
FINE: &lt;&lt; Content-Length: 93
FINE: &lt;&lt; "{"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}"
</pre>
    <br>
    Again, even though I created the jclouds ContextBuilder with the
    V2.0 endpoint, it looks like jclouds read the identity admin
    endpoint from the 'serviceCatalog' in the response and then switched
    to using it, even though it is a V3 endpoint.  Is this normal
    expected behavior for jclouds?  I'm guessing it has more to do with
    how Mitaka is configured, but I don't see how jclouds would be able
    to talk to newer versions of OpenStack unless they disable the V3
    Identity endpoint or configure it in such a way that the V2.0
    endpoint is exposed for jclouds to see it.  Are there any READMEs on
    how to use jclouds keystone APIs with newer versions of OpenStack?<br>
    <br>
    In general, it seems like OpenStack is moving more towards a model
    where API versioning is supposed to be discovered by clients.  For
    example: <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/130669/">https://review.openstack.org/#/c/130669/</a>, specifically the
    patch comment that says "if some one is using third party client, it
    will break. Only keystone client handles version discovery.   
    Keystone doesn't even document how to handle version discovery, so
    any third party clients such as jcloud will likely fail"<br>
    <br>
    Is there any update on jclouds adding support for the V3 Identity
    API?<br>
    <br>
    <div class="moz-signature">Thanks,<br>
      <br>
      --Ryan</div>
  </body>
</html>

--------------68D910B48DB20E1723BC0AC8--