From issues-return-52532-archive-asf-public=cust-asf.ponee.io@mesos.apache.org Thu Feb 6 20:25:03 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0356718067C for ; Thu, 6 Feb 2020 21:25:02 +0100 (CET) Received: (qmail 45641 invoked by uid 500); 6 Feb 2020 20:25:01 -0000 Mailing-List: contact issues-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mesos.apache.org Delivered-To: mailing list issues@mesos.apache.org Received: (qmail 45575 invoked by uid 99); 6 Feb 2020 20:25:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2020 20:25:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id CBB12E2CA2 for ; Thu, 6 Feb 2020 20:25:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 4673E78007F for ; Thu, 6 Feb 2020 20:25:00 +0000 (UTC) Date: Thu, 6 Feb 2020 20:25:00 +0000 (UTC) From: "Andrei Sekretenko (Jira)" To: issues@mesos.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (MESOS-10092) Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MESOS-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031894#comment-17031894 ] Andrei Sekretenko commented on MESOS-10092: ------------------------------------------- {noformat} {noformat} commit 4ed912c6b6eee95ad5a1267fad20b7c279ccc153 Author: Andrei Sekretenko Date: Thu Jan 30 12:39:31 2020 +0100 Factored out common code for building URIs on a registry host. This is a prerequisite for adding fallback authorization server URI generation (see MESOS-10092) in the next patches, which will need one more URI extractor similar to `getManifestUri()`/`getBlobUri()`. Review: https://reviews.apache.org/r/72077 {noformat} {noformat} commit d97a01b5f7e5351564b530c08ee6fd937ef62732 Author: Andrei Sekretenko Date: Thu Jan 30 13:02:45 2020 +0100 Factored out getting authorization service URI. Review: https://reviews.apache.org/r/72078 {noformat} {noformat} commit c37e6df142322917288aa01e7226f9a5fb59125d Author: Andrei Sekretenko Date: Thu Jan 30 14:26:38 2020 +0100 Added workaround for Docker repositories not providing scope/service. This patch adds a fallback Docker authorization server URI generation mechanism (see MESOS-10092) for repository servers that provide no "scope"/"service" params in the "WWW-Authenticate" header of the initial "401 Unathorized" response. Review: https://reviews.apache.org/r/72079 {noformat} {noformat} commit b7076c9fe5ece573361dcfb899962f4f30df2fd2 Author: Andrei Sekretenko Date: Thu Jan 23 12:42:33 2020 +0100 Added test for fetch from repositories not providing scope/service. Review: https://reviews.apache.org/r/72080 {noformat} > Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header > ------------------------------------------------------------------------------------------------------------- > > Key: MESOS-10092 > URL: https://issues.apache.org/jira/browse/MESOS-10092 > Project: Mesos > Issue Type: Bug > Reporter: Andrei Sekretenko > Assignee: Andrei Sekretenko > Priority: Critical > > This problem was encountered when trying to specify container image nvcr.io/nvidia/tensorflow:19.12-tf1-py3 > When initiating Docker Registry authentication (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI fetcher receives 'WWW-Authenticate' header without 'service' and 'scope' params, and fails here: > https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083 > This is an example of an unsuccessful request made by Mesos: > {code} > curl -s -S -L -i --raw --http1.1 -H "Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws" -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3 > HTTP/1.1 401 Unauthorized > Content-Type: text/html > Date: Wed, 22 Jan 2020 19:01:57 GMT > Server: nginx/1.14.2 > Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push" > Content-Length: 195 > Connection: keep-alive > > 401 Authorization Required > >

401 Authorization Required

>
nginx/1.14.2
> > > {code} > At the same time, docker is perfectly capable of pulling this image. > Note that the document "Token Authentication Specification" (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos implementation is based, is vague on the issue of registries that do not provide 'scope'/'service' in WWW-Authenticate header. > What Docker does differently (at the very least, in the case of nvcr.io): > It sends the initial request not to the maniferst/blob URI, but to the repository root URI (http:://nvcr.io/v2 in this case): > {code} > GET /v2/ HTTP/1.1 > Host: nvcr.io > User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd kernel/4.15.0-60-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.7 \(linux\)) > {code} > To this, it receives response with a "realm" that contains no query arguments: > {code} > HTTP/1.1 401 Unauthorized > Connection: close > Content-Length: 195 > Content-Type: text/html > Date: Wed, 29 Jan 2020 12:22:43 GMT > Server: nginx/1.14.2 > Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth > {code} > Then, it composes the scope using the image ref and a hardcoded "pull" action: > https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174 > (in a full accordance with this spec: https://docs.docker.com/registry/spec/auth/scope/) > and sends the following request to https://nvcr.io/proxy_auth : > {code} > GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1 > Host: nvcr.io > User-Agent: Go-http-client/1.1 > {code} > (Note that 'push' is absent from the scope) -- This message was sent by Atlassian Jira (v8.3.4#803005)