Mailing-List: contact user-help@kudu.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@kudu.apache.org
MIME-Version: 1.0
In-Reply-To: <CADY20s5zQvdZb8WOctkaQfn1wn7f3_VP8q6VZrHYV+O+rd-Nfg@mail.gmail.com>
References: <CALQ8dW52CfwRM+nouuJSZOkaRBEgtRov_vw26Q-+QyRwoM6Wcg@mail.gmail.com>
 <CADY20s5zQvdZb8WOctkaQfn1wn7f3_VP8q6VZrHYV+O+rd-Nfg@mail.gmail.com>
From: Matteo Durighetto <m.durighetto@miriade.it>
Date: Mon, 16 Oct 2017 23:29:54 +0200
Message-ID: <CALQ8dW6XtPNs0Ms0N+TCJWCLp+Xi2_q2KhpsfRL=1dFE3Sm9Nw@mail.gmail.com>
Subject: Re: kudu 1.4 kerberos
To: user@kudu.apache.org
Content-Type: multipart/alternative; boundary="089e08264d2c8246e0055bb0b72f"
archived-at: Mon, 16 Oct 2017 21:30:48 -0000

--089e08264d2c8246e0055bb0b72f
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello Todd,
                   thank you very much for the answer. I think I have found
something interesting.

Kudu is doing the ACL list with the sAMAccountName or CN  as it writes in
the logs:

 "Logged in from keytab as kudu/<host>@REALM (short username
<sAMAccountName> )"

I begin to think that the problem is between sssd with plugin
sssd_krb5_localauth_plugin.so
so for every principal kudu/<host>@REALM kudu maps to  <sAMAccountName>,
so seems impossible to have the all  kudu/<host>@REALM mapped to the same
"kudu"
as suggested "So, basically, it's critical that the username that the
master determines for itself (from this function)
matches the username that it has determined for the tablet servers when
they authenticate
 (what you pasted as 'abcdefgh1234' above)."

The strange thing is that with hadoop I have the correct mapping ( probably
because I have no rule, so
it switch to default rule )

hadoop org.apache.hadoop.security.HadoopKerberosName  kudu/<host>@REALM
 =3D=3D> kudu


Matteo Durighetto


2017-10-13 1:32 GMT+02:00 Todd Lipcon <todd@cloudera.com>:

> Hey Matteo,
>
> Looks like you did quite a bit of digging in the code! Responses inline
> below.
>
> On Wed, Oct 11, 2017 at 1:24 PM, Matteo Durighetto <
> m.durighetto@miriade.it> wrote:
>
>> Hello,
>>            I have a strange behaviour with Kudu 1.4 and kerberos.
>> I enabled kerberos on kudu, I have the principal correctly in the OU of
>> an AD, but
>> at startup I got a lot of errors on method TSHeartbeat between tablet
>> server and
>> master server as unauthorized. There's no firewall between nodes.
>>
>
> right, "unauthorized" indicates that the connection was made fine, but th=
e
> individual RPC call was determined to not be allowed for the identity
> presented on the other side of the connection.
>
>
>>
>> W1011 <time>   server_base.cc:316] Unauthorized access attempt
>> to method kudu.master.MasterService.TSHeartbeat
>> from {username=3D'abcdefgh1234', principal=3D'kudu/HOSTNAME@DOMAIN.XYZ'}
>> at <IP>:37360
>>
>> the "abcdefgh1234" it's an example of the the string created by the
>> cloudera manager during the enable kerberos.
>>
>
> This output indicates that it successfully authenticated via Kerberos as
> the principal listed above. That's good news and means you don't need to
> worry about rdns, etc (if you had issues with that it would have had
> trouble finding a service ticket or authenticating the connection). This
> means you got past the "authentication" step and having problems at the
> "authorization" step.
>
>
>>
>> The other services (hdfs and so on ) are under kerberos without problem
>> and there is the rdns at true in the /etc/krb5.conf (  KUDU-2032 ).
>> As I understand the problem is something about the 3=C2=B0 level of
>> authorization between master servers and tablet servers.
>>
>
> Right.
>
>
>> ... <snipped>
>> So I think the problem, as I say before, could be in  ContainsKey(users_=
,
>> username);  :
>>
>> bool SimpleAcl::UserAllowed(const string& username) {
>>   return ContainsKey(users_, "*") || ContainsKey(users_, username);
>> }
>>
>>
>> At this point It's not clear for me how Kudu build the array/key list
>> users for daemon service ( it's not as super users or user ACL an extern=
al
>> parameter).
>>
>
> Exactly. The users here for the 'service' ACL are set in
> ServerBase::InitAcls():
>
>   boost::optional<string> keytab_user =3D security::
> GetLoggedInUsernameFromKeytab();
>   if (keytab_user) {
>     // If we're logged in from a keytab, then everyone should be, and we
> expect them
>     // to use the same mapped username.
>     service_user =3D *keytab_user;
>   } else {
>     // If we aren't logged in from a keytab, then just assume that the
> services
>     // will be running as the same Unix user as we are.
>     RETURN_NOT_OK_PREPEND(GetLoggedInUser(&service_user),
>                           "could not deterine local username");
>   }
>
> Since you're using Kerberos, the top branch here would apply -- it's
> calling GetLoggedInUsernameFromKeytab() from init.cc.
>
> You can see what username the server is getting by looking for a log
> message at startup like "Logged in from keytab as kudu/<host>@REALM (shor=
t
> username <XYZ>)". Here 'XYZ' is the username that ends up in the service
> ACL.
>
> So, basically, it's critical that the username that the master determines
> for itself (from this function) matches the username that it has determin=
ed
> for the tablet servers when they authenticate (what you pasted as
> 'abcdefgh1234' above).
>
> That brings us to the next question: how do we convert from a principal
> like kudu/<HOST>@REALM to a short "username"? The answer there is the
> function 'MapPrincipalToLocalName' again from security/init.cc. This
> function delegates the mapping to the krb5 library itself using the
> krb5_aname_to_localname() API. The results of this API can vary depending
> on the kerberos configuration, but in typical configurations it's
> determined by the 'auth_to_local' configuration in your krb5.conf. See th=
e
> corresponding section in the docs here:
>
> https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.htm=
l
>
> My guess is that your host has been configured such that when the master
> maps its own principal, it's getting a different result than when it maps
> the principal being used by the tservers.
>
> Hope that gets you on the right track.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--089e08264d2c8246e0055bb0b72f
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello Todd,<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0thank you very much for the answer. I think I ha=
ve found something interesting.</div><div><br></div><div>Kudu is doing the =
ACL list with the sAMAccountName or CN=C2=A0 as it writes in the logs:</div=
><div><br></div><div>=C2=A0&quot;Logged in from keytab as kudu/&lt;host&gt;=
@REALM (short username &lt;sAMAccountName&gt; )&quot;</div><div><br></div><=
div>I begin to think that the problem is between sssd with plugin sssd_krb5=
_localauth_plugin.so</div><div>so for every principal kudu/&lt;host&gt;@REA=
LM kudu maps to=C2=A0 &lt;sAMAccountName&gt;,</div><div>so seems impossible=
 to have the all=C2=A0 kudu/&lt;host&gt;@REALM mapped to the same &quot;kud=
u&quot;=C2=A0</div><div>as suggested &quot;So, basically, it&#39;s critical=
 that the username that the master determines for itself (from this functio=
n)=C2=A0</div><div>matches the username that it has determined for the tabl=
et servers when they authenticate</div><div>=C2=A0(what you pasted as &#39;=
abcdefgh1234&#39; above).&quot;</div><div><br></div><div><div>The strange t=
hing is that with hadoop I have the correct mapping ( probably because I ha=
ve no rule, so</div><div>it switch to default rule )</div><div><br></div><d=
iv>hadoop org.apache.hadoop.security.HadoopKerberosName=C2=A0 kudu/&lt;host=
&gt;@REALM=C2=A0 =C2=A0=3D=3D&gt; kudu<br></div><div><br></div><div><br></d=
iv><div class=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"gmail_si=
gnature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div>=
<div dir=3D"ltr">Matteo Durighetto<br><div style=3D"white-space:pre-wrap;co=
lor:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255=
,255)"><div style=3D"white-space:normal;font-family:arial"><div><div style=
=3D"color:rgb(68,68,68);font-family:arial,sans-serif;line-height:18px"><div=
><div style=3D"font-size:13px"><font size=3D"1"><br></font></div></div></di=
v></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class=3D"gmail_quote">2017-10-13 1:32 GMT+02:00 Todd Lipcon <span =
dir=3D"ltr">&lt;<a href=3D"mailto:todd@cloudera.com" target=3D"_blank">todd=
@cloudera.com</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D=
"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le=
ft:1ex"><div dir=3D"ltr">Hey Matteo,<div><br></div><div>Looks like you did =
quite a bit of digging in the code! Responses inline below.</div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote"><span class=3D"gmail-">On W=
ed, Oct 11, 2017 at 1:24 PM, Matteo Durighetto <span dir=3D"ltr">&lt;<a hre=
f=3D"mailto:m.durighetto@miriade.it" target=3D"_blank">m.durighetto@miriade=
.it</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1=
ex"><div dir=3D"ltr"><div><div>Hello,</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0I have a strange behaviour with Kudu 1.4 and kerberos.</div><=
div>I enabled kerberos on kudu, I have the principal correctly in the OU of=
 an AD, but</div><div>at startup I got a lot of errors on method TSHeartbea=
t between tablet server and=C2=A0</div><div>master server as unauthorized. =
There&#39;s no firewall between nodes.</div></div></div></blockquote><div><=
br></div></span><div>right, &quot;unauthorized&quot; indicates that the con=
nection was made fine, but the individual RPC call was determined to not be=
 allowed for the identity presented on the other side of the connection.</d=
iv><span class=3D"gmail-"><div>=C2=A0</div><blockquote class=3D"gmail_quote=
" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);=
padding-left:1ex"><div dir=3D"ltr"><div><div><br></div><div>W1011 &lt;time&=
gt;=C2=A0 =C2=A0server_base.cc:316] Unauthorized access attempt=C2=A0</div>=
<div>to method kudu.master.MasterService.TSHe<wbr>artbeat=C2=A0</div><div>f=
rom {username=3D&#39;abcdefgh1234&#39;, principal=3D&#39;kudu/<a href=3D"ma=
ilto:HOSTNAME@DOMAIN.XYZ" target=3D"_blank">HOSTNAME@DOMAI<wbr>N.XYZ</a>=
9;}=C2=A0</div><div>at &lt;IP&gt;:37360</div><div><br></div><div>the &quot;=
abcdefgh1234&quot; it&#39;s an example of the the string created by the clo=
udera manager during the enable kerberos.</div></div></div></blockquote><di=
v><br></div></span><div>This output indicates that it successfully authenti=
cated via Kerberos as the principal listed above. That&#39;s good news and =
means you don&#39;t need to worry about rdns, etc (if you had issues with t=
hat it would have had trouble finding a service ticket or authenticating th=
e connection). This means you got past the &quot;authentication&quot; step =
and having problems at the &quot;authorization&quot; step.</div><span class=
=3D"gmail-"><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1=
ex"><div dir=3D"ltr"><div><div><br></div><div>The other services (hdfs and =
so on ) are under kerberos without problem and there is the rdns at true in=
 the /etc/krb5.conf (=C2=A0=C2=A0<span style=3D"background-color:transparen=
t;color:rgb(0,0,0);font-family:Arial;font-size:11pt;white-space:pre-wrap">K=
UDU-2032 </span>).</div><div>As I understand the problem is something about=
 the 3=C2=B0 level of authorization between master servers and tablet serve=
rs.</div></div></div></blockquote><div><br></div></span><div>Right.</div><d=
iv>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p=
x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=
=3D"ltr"><div><div>... &lt;snipped&gt;</div><span class=3D"gmail-"><div>So =
I think the problem, as I say before, could be in=C2=A0=C2=A0ContainsKey(us=
ers_, username);=C2=A0 :</div><div><br></div><div><div>bool SimpleAcl::User=
Allowed(const string&amp; username) {</div><div>=C2=A0 return ContainsKey(u=
sers_, &quot;*&quot;) || ContainsKey(users_, username);</div><div>}</div></=
div><div><br></div><div>=C2=A0=C2=A0</div><div>At this point It&#39;s not c=
lear for me how Kudu build the array/key list users for daemon service ( it=
&#39;s not as super users or user ACL an external parameter).</div></span><=
/div></div></blockquote><div><br></div><div>Exactly. The users here for the=
 &#39;service&#39; ACL are set in ServerBase::InitAcls():</div><div><br></d=
iv><div><div>=C2=A0 boost::optional&lt;string&gt; keytab_user =3D security:=
:<wbr>GetLoggedInUsernameFromKeytab(<wbr>);</div><div>=C2=A0 if (keytab_use=
r) {</div><div>=C2=A0 =C2=A0 // If we&#39;re logged in from a keytab, then =
everyone should be, and we expect them</div><div>=C2=A0 =C2=A0 // to use th=
e same mapped username.</div><div>=C2=A0 =C2=A0 service_user =3D *keytab_us=
er;</div><div>=C2=A0 } else {</div><div>=C2=A0 =C2=A0 // If we aren&#39;t l=
ogged in from a keytab, then just assume that the services</div><div>=C2=A0=
 =C2=A0 // will be running as the same Unix user as we are.</div><div>=C2=
=A0 =C2=A0 RETURN_NOT_OK_PREPEND(<wbr>GetLoggedInUser(&amp;service_user)<wb=
r>,</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;could not deterine local username&quo=
t;);</div><div>=C2=A0 }</div></div><div><br></div><div>Since you&#39;re usi=
ng Kerberos, the top branch here would apply -- it&#39;s calling GetLoggedI=
nUsernameFromKeytab(<wbr>) from init.cc.</div><div><br></div><div>You can s=
ee what username the server is getting by looking for a log message at star=
tup like &quot;Logged in from keytab as kudu/&lt;host&gt;@REALM (short user=
name &lt;XYZ&gt;)&quot;. Here &#39;XYZ&#39; is the username that ends up in=
 the service ACL.</div><div><br></div><div>So, basically, it&#39;s critical=
 that the username that the master determines for itself (from this functio=
n) matches the username that it has determined for the tablet servers when =
they authenticate (what you pasted as &#39;abcdefgh1234&#39; above).</div><=
div><br></div><div>That brings us to the next question: how do we convert f=
rom a principal like kudu/&lt;HOST&gt;@REALM to a short &quot;username&quot=
;? The answer there is the function &#39;MapPrincipalToLocalName&#39; again=
 from security/init.cc. This function delegates the mapping to the krb5 lib=
rary itself using the krb5_aname_to_localname() API. The results of this AP=
I can vary depending on the kerberos configuration, but in typical configur=
ations it&#39;s determined by the &#39;auth_to_local&#39; configuration in =
your krb5.conf. See the corresponding section in the docs here:</div><div><=
br></div><div><a href=3D"https://web.mit.edu/kerberos/krb5-1.12/doc/admin/c=
onf_files/krb5_conf.html" target=3D"_blank">https://web.mit.edu/kerberos/<w=
br>krb5-1.12/doc/admin/conf_<wbr>files/krb5_conf.html</a></div></div><div c=
lass=3D"gmail_extra"><br></div><div class=3D"gmail_extra">My guess is that =
your host has been configured such that when the master maps its own princi=
pal, it&#39;s getting a different result than when it maps the principal be=
ing used by the tservers.</div><div class=3D"gmail_extra"><br></div><div cl=
ass=3D"gmail_extra">Hope that gets you on the right track.</div><div class=
=3D"gmail_extra"><br></div><div class=3D"gmail_extra">Thanks</div><span cla=
ss=3D"gmail-HOEnZb"><font color=3D"#888888"><div class=3D"gmail_extra">-Tod=
d</div>-- <br><div class=3D"gmail-m_-7429482503714641908gmail_signature">To=
dd Lipcon<br>Software Engineer, Cloudera</div>
</font></span></div></div>
</blockquote></div><br></div></div></div>

--089e08264d2c8246e0055bb0b72f--