ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chanel Loïc <loic.cha...@worldline.com>
Subject RE: Restarting nodes to avoid HTTP 403
Date Wed, 13 May 2015 12:31:30 GMT
Hi Jonathan,

Additional information : the restart does not have to be necessarily AFTER the cluster have
been deployed using a Blueprint. Restarting the ambari-agent before using a blueprint to deploy
the cluster make a perfectly clear cluster, without the 11 Warnings/Errors I mentioned in
my previous emails.

Hope this will help understand where the problem comes from.
Have a nice day,


Loïc

De : Jonathan Hurley [mailto:jhurley@hortonworks.com]
Envoyé : jeudi 7 mai 2015 18:53
À : user@ambari.apache.org
Objet : Re: Restarting nodes to avoid HTTP 403

All of this really seems to point to some kind of firewall/proxy issue on the agent hosts.
From your agent named "vm-03cfbc97-f027-46fe-8e65-cb8c54edf377.frida.priv.atos.fr<http://vm-03cfbc97-f027-46fe-8e65-cb8c54edf377.frida.priv.atos.fr>",
could you try the following python code:

>>> import urllib2
>>> response = urllib2.urlopen("http://vm-03cfbc97-f027-46fe-8e65-cb8c54edf377.frida.priv.atos.fr:8042/ws/v1/node/info",
timeout=10.0)
>>> print(response.code)

I'm really curious if executing that from your agent host results in a 200 or a 403.

On May 7, 2015, at 11:45 AM, Chanel Loïc <loic.chanel@worldline.com<mailto:loic.chanel@worldline.com>>
wrote:

Hi Jonathan,

Here are the answers to your questions :

1) Which operating system are you running the agents on?
--> CentOS6

2) Is there a linux proxy setup, such as "export http_proxy=foo" - curl doesn't respect this
proxy setting but python's urllib2 does, which has caused some issues before
--> There is a proxy, but there is no such setup (I mean "echo $http_proxy" returns nothing,
and I do not remember setting Ambari properly in order to access to the Internet via the proxy)

3) Which stack are you deploying?
--> I am deploying HDP-2.2.4.2-2

Have a nice weekend,


Loïc


De : Jonathan Hurley [mailto:jhurley@hortonworks.com]
Envoyé : jeudi 7 mai 2015 17:34
À : user@ambari.apache.org<mailto:user@ambari.apache.org>
Objet : Re: Restarting nodes to avoid HTTP 403

The logs indicate that the alerts are running correctly and are simply hitting a 403. Normally,
you might encounter this kind of problem from Python when making web connections without specifying
a known user agent header. Python's default header sometimes causes issues since it's not
standard. However, that problem would continue to happen after you restarted the agents. The
fact that a simple agent restart completely fixes the issue is baffling.

I've certainly never seen this type of behavior before. I'd like to know a few more details
on your environment:
1) Which operating system are you running the agents on?
2) Is there a linux proxy setup, such as "export http_proxy=foo" - curl doesn't respect this
proxy setting but python's urllib2 does, which has caused some issues before
3) Which stack are you deploying?

On May 7, 2015, at 4:05 AM, Chanel Loïc <loic.chanel@worldline.com<mailto:loic.chanel@worldline.com>>
wrote:

Hi Jonathan,

You will find in attachment what I get from the API you gave me the URL.
As far as the log file is concerned, here is an extract from the one on the NameNode :

WARNING 2015-05-07 10:01:47,221 base_alert.py:365 - [Alert][namenode_directory_status] HA
nameservice value is present but there are no aliases for {{hdfs-site/dfs.ha.namenodes.{{ha-nameservice}}}}
WARNING 2015-05-07 10:01:47,222 base_alert.py:365 - [Alert][datanode_health_summary] HA nameservice
value is present but there are no aliases for {{hdfs-site/dfs.ha.namenodes.{{ha-nameservice}}}}
INFO 2015-05-07 10:01:47,228 scheduler.py:509 - Running job "e89e2c29-1f2c-4bf5-a37c-7a6c5b43433a
(trigger: interval[0:01:00], next run at: 2015-05-07 10:02:47.211937)" (scheduled at 2015-05-07
10:01:47.211937)
INFO 2015-05-07 10:01:47,229 scheduler.py:509 - Running job "ad0e0fd3-d5f3-471f-955e-ec10f53cdd5b
(trigger: interval[0:01:00], next run at: 2015-05-07 10:01:47.212773)" (scheduled at 2015-05-07
10:01:47.212773)
WARNING 2015-05-07 10:01:47,230 base_alert.py:365 - [Alert][namenode_webui] HA nameservice
value is present but there are no aliases for {{hdfs-site/dfs.ha.namenodes.{{ha-nameservice}}}}
INFO 2015-05-07 10:01:47,230 scheduler.py:527 - Job "71988556-19e3-4871-92e0-e3c0a838df13
(trigger: interval[0:01:00], next run at: 2015-05-07 10:02:47.206667)" executed successfully
INFO 2015-05-07 10:01:47,238 scheduler.py:509 - Running job "2de4dc85-8993-4c68-9915-db25c4313d6e
(trigger: interval[0:01:00], next run at: 2015-05-07 10:02:47.213529)" (scheduled at 2015-05-07
10:01:47.213529)
WARNING 2015-05-07 10:01:47,240 base_alert.py:140 - [Alert][datanode_health_summary] Unable
to execute alert. HTTP Error 403: Forbidden
INFO 2015-05-07 10:01:47,242 scheduler.py:527 - Job "5bcf7b5d-73e7-4d59-bc0a-f4772d4e3166
(trigger: interval[0:01:00], next run at: 2015-05-07 10:02:47.208176)" executed successfully
INFO 2015-05-07 10:01:47,241 scheduler.py:509 - Running job "ff1fc102-bdec-4858-baf5-8b60de4488e4
(trigger: interval[0:01:00], next run at: 2015-05-07 10:01:47.214826)" (scheduled at 2015-05-07
10:01:47.214826)
WARNING 2015-05-07 10:01:47,244 base_alert.py:365 - [Alert][yarn_resourcemanager_webui] HA
nameservice value is present but there are no aliases for {{yarn-site/yarn.resourcemanager.ha.rm-ids}}
INFO 2015-05-07 10:01:47,240 scheduler.py:527 - Job "72e5031e-4c2a-4236-b09e-6a749100bc9a
(trigger: interval[0:01:00], next run at: 2015-05-07 10:02:47.203349)" executed successfully
INFO 2015-05-07 10:01:47,250 scheduler.py:509 - Running job "a083e3c2-d4b2-430b-9b64-b60c783a06a5
(trigger: interval[0:01:00], next run at: 2015-05-07 10:01:47.218706)" (scheduled at 2015-05-07
10:01:47.218706)
INFO 2015-05-07 10:01:47,250 scheduler.py:509 - Running job "7e9470cb-043b-48a8-990a-0050f2c63311
(trigger: interval[0:01:00], next run at: 2015-05-07 10:02:47.217519)" (scheduled at 2015-05-07
10:01:47.217519)
WARNING 2015-05-07 10:01:47,253 base_alert.py:140 - [Alert][namenode_directory_status] Unable
to execute alert. HTTP Error 403: Forbidden

So yes, I can see some output in the logs corresponding to the alerts I get on the Ambari
web app.
Please tell me if you need any complementary information about my problem,

Thanks,


Loïc


De : Jonathan Hurley [mailto:jhurley@hortonworks.com]
Envoyé : mercredi 6 mai 2015 16:24
À : user@ambari.apache.org<mailto:user@ambari.apache.org>
Objet : Re: Restarting nodes to avoid HTTP 403

OK, so I think I have a clear picture of how you get to this situation. I'd still like to
know a few things:

1) When you have the warnings present in the web client, can you try the alerts URL I posted
below to see the actual alerts coming back from the API. I'm mostly interesting in whether
any come back in the first place, and what the most recent timestamp was (indicating they
are actually running and still reporting a warning status)

2) When the alerts are present in the web client, do you see any output in the agent log file
that I mentioned for alerts that start with [Alert].

On May 6, 2015, at 5:14 AM, Chanel Loïc <loic.chanel@worldline.com<mailto:loic.chanel@worldline.com>>
wrote:

I encounter the warnings and critical alerts when deploying the cluster. I install Ambari
server 2.0 on a VM and Ambari agent 2.0 on 4 others VMs. Then, I give the Ambari server a
blueprint coming from a functioning cluster to instantiate my new cluster and have a quickstart
configuration.

Then, when logging into the Ambari web application to ensure everything is running properly,
I have these alerts concerning the HTTP 403 errors returned by all host VMs but the one which
only handles Ambari metrics.

As I am not sure my explanations are quite understandable, do not hesitate to tell me if something
remains unclear.
Thanks,


De : Jonathan Hurley [mailto:jhurley@hortonworks.com]
Envoyé : mardi 5 mai 2015 20:38
À : user@ambari.apache.org<mailto:user@ambari.apache.org>
Objet : Re: Restarting nodes to avoid HTTP 403

If restarting the agents fixes everything, can you explain when you first encounter the warnings?
Is this only after a cluster deployment? You can also check to see if this is some sort of
web client issue by issuing the following GET:

http://server/api/v1/clusters/<cluster>/alerts?fields=*&Alert/state.in(CRITICAL,WARNING)<http://server/api/v1/clusters/%3Ccluster%3E/alerts?fields=*&Alert/state.in(CRITICAL,WARNING)>

This will show you alerts which are actually being returned from the agents in a warning or
critical state.

For reference, you can also look in /var/log/ambari-agent/ambari-agent.log to see if you see
any alert issues. Most important messages are prefixed with "[Alert]".

On May 5, 2015, at 11:43 AM, Chanel Loïc <loic.chanel@worldline.com<mailto:loic.chanel@worldline.com>>
wrote:

Indeed, I did not gave so much information about my problem, sorry about that. Here is your
answers :

1) Version of Ambari
--> I'm using Ambari 2.0

2) Whether the environment is kerberized
--> Not it's not. Kerberos security is not enabled on this cluster.

3) Are you running the Ambari agent as root, or another user
--> I am running it as root

4) Any information from the ambari-agent.log file that might seem to indicate a problem
--> Here is another weird thing I did not mentioned : I could not find logs referring to
the problem.

5) You said that restarting the agents resolves the issue. Does it continue to happen after
restarting? If so, how long before new warnings start to show up
--> Restarting the agent totally resolves the problem. It does not happen anymore, and
everything run quite normally.


De : Jonathan Hurley [mailto:jhurley@hortonworks.com]
Envoyé : mardi 5 mai 2015 17:36
À : user@ambari.apache.org<mailto:user@ambari.apache.org>
Objet : Re: Restarting nodes to avoid HTTP 403

Can you provide some more information on your environment, such as:

1) Version of Ambari
2) Whether the environment is kerberized
3) Are you running the Ambari agent as root, or another user.
4) Any information from the ambari-agent.log file that might seen to indicate a problem
5) You said that restarting the agents resolves the issue. Does it continue to happen after
restarting? If so, how long before new warnings start to show up.

I'm guessing you have a kerberized environment running Ambari 2.0. Ambari will use curl in
this case to attempt to make a connection to the web endpoints. It uses the keytabs and principals
defined on the alert definition. For NameNode, as an example, it would use:

hdfs-site/dfs.web.authentication.kerberos.keytab
hdfs-site/dfs.web.authentication.kerberos.principal

You'll want to verify that these properties are correctly set and that the keytab file is
accessible to the agent user.

It could also be a cache problem as the agents cache the kerberos credentials in the agent's
temp directory. How long it takes after alerts start producing warnings would in determining
if it's a caching issue.


On May 5, 2015, at 10:49 AM, Chanel Loïc <loic.chanel@worldline.com<mailto:loic.chanel@worldline.com>>
wrote:

Hi,

I have currently some issues with cluster nodes. According to the Ambari web User Interface,
I have 11 alerts linked to the fact that all nodes (except the monitoring one) return HTTP
403 response for services such as NameNode web UI, DataNode web UI, or NodeManager Health.

The first weird thing is the fact that I can easily see that the so-called Forbidden ports
are actually quite available (via cURL for example) and indicate that the cluster is totally
ok.
The second is the fact that the 11 alerts magically disappear when rebooting the agent on
the hosts related to the errors.

Does anyone know where these errors might come from ?

Thanks,


Loïc

________________________________

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de
ses destinataires. Il peut également être protégé par le secret professionnel. Si vous
recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire.
L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline
ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne
aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout
dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee;
it may also be privileged. If you receive this e-mail in error, please notify the sender immediately
and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability
cannot be triggered for the message content. Although the sender endeavours to maintain a
computer virus-free network, the sender does not warrant that this transmission is virus-free
and will not be liable for any damages resulting from any virus transmitted.


________________________________

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de
ses destinataires. Il peut également être protégé par le secret professionnel. Si vous
recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire.
L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline
ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne
aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout
dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee;
it may also be privileged. If you receive this e-mail in error, please notify the sender immediately
and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability
cannot be triggered for the message content. Although the sender endeavours to maintain a
computer virus-free network, the sender does not warrant that this transmission is virus-free
and will not be liable for any damages resulting from any virus transmitted.


________________________________

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de
ses destinataires. Il peut également être protégé par le secret professionnel. Si vous
recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire.
L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline
ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne
aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout
dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee;
it may also be privileged. If you receive this e-mail in error, please notify the sender immediately
and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability
cannot be triggered for the message content. Although the sender endeavours to maintain a
computer virus-free network, the sender does not warrant that this transmission is virus-free
and will not be liable for any damages resulting from any virus transmitted.


________________________________

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de
ses destinataires. Il peut également être protégé par le secret professionnel. Si vous
recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire.
L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline
ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne
aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout
dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee;
it may also be privileged. If you receive this e-mail in error, please notify the sender immediately
and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability
cannot be triggered for the message content. Although the sender endeavours to maintain a
computer virus-free network, the sender does not warrant that this transmission is virus-free
and will not be liable for any damages resulting from any virus transmitted.
<API alerts.txt>


________________________________

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de
ses destinataires. Il peut également être protégé par le secret professionnel. Si vous
recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire.
L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline
ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne
aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout
dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee;
it may also be privileged. If you receive this e-mail in error, please notify the sender immediately
and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability
cannot be triggered for the message content. Although the sender endeavours to maintain a
computer virus-free network, the sender does not warrant that this transmission is virus-free
and will not be liable for any damages resulting from any virus transmitted.


________________________________

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de
ses destinataires. Il peut également être protégé par le secret professionnel. Si vous
recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire.
L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline
ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne
aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout
dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee;
it may also be privileged. If you receive this e-mail in error, please notify the sender immediately
and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability
cannot be triggered for the message content. Although the sender endeavours to maintain a
computer virus-free network, the sender does not warrant that this transmission is virus-free
and will not be liable for any damages resulting from any virus transmitted.

Mime
View raw message