ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meghavi Sugandhi <meghavi.sugan...@tcs.com>
Subject Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI
Date Mon, 16 Jun 2014 05:33:12 GMT
 

 Hi,

rrd.py script have right permission and rr is running fine.

[pts/2][17:43:05:root@INBBRDSSVM50 ] /var/nagios/spool/checkresults>ls -l /var/www/cgi-bin/rrd.py
-rwxr-xr-x. 1 root root 5097 Jun  2 11:13 /var/www/cgi-bin/rrd.py

[pts/2][10:55:13:root@INBBRDSSVM50 ] /var/nagios/spool/checkresults>ps aux | grep rrd

nobody   21957  1.0  0.0 511564  4640 ?        Ssl  Jun13  43:27 /usr/bin/rrdcached -p /var/run/ganglia/hdp/rrdcached.pid
-m 664 -l unix:/var/run/ganglia/hdp/rrdcached.sock -m 777 -P FLUSH,STATS,HELP -l unix:/var/run/ganglia/hdp/rrdcached.limited.sock
-b /var/lib/ganglia/rrds -B
root     28571  0.0  0.0 103248   856 pts/2    S+   10:55   0:00 grep rrd

[pts/2][10:55:43:root@INBBRDSSVM50 ] /var/nagios/spool/checkresults>



I am running pseudo mode cluster.

Am I missing anything?


Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
Website: http://www.tcs.com

-----Siddharth Wagle <swagle@hortonworks.com> wrote: -----
To: "user@ambari.apache.org" <user@ambari.apache.org>
From: Siddharth Wagle <swagle@hortonworks.com>
Date: 06/13/2014 10:09PM
Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI

Hi Meghavi,

Few more things to check for,

 ~]# ls -l /var/www/cgi-bin/rrd.py
-rwxr-xr-x. 1 root root 5706 Jun  9 18:42 /var/www/cgi-bin/rrd.py

Make sure this file exists with right permissions.
 
Also, make sure rrdcahced process is running, 
~]# ps aux | grep rrd

Lastly, make sure you have increased ulimits, default is 1024 which could be very small, based
on you cluster size.
 
-Sid



On Thu, Jun 12, 2014 at 10:03 PM, Meghavi Sugandhi <meghavi.sugandhi@tcs.com> wrote:
  

Hi Sid,

I changes debug level of gmetad.conf to 1 and restarted the process
 
[pts/2][10:19:52:root@INBBRDSSVM50 ] /var/lib/ganglia/rrds/HDPHBaseMaster/INBBRDSSVM50>service
hdp-gmetad restart
==================================
Shutting down hdp-gmetad...
==================================
 Stopped /usr/sbin/gmetad (with PID 11110)
Stopped /usr/bin/rrdcached (with PID 11080)

=============================
Starting hdp-gmetad...
=============================
Started /usr/bin/rrdcached with PID 10344
 Sources are ...
Source: [HDPHBaseMaster, step 15] has 1 sources
    10.18.15.75
Source: [HDPResourceManager, step 15] has 1 sources
    10.18.15.75
Source: [HDPSlaves, step 15] has 1 sources
    10.18.15.75
 Source: [HDPHistoryServer, step 15] has 1 sources
    10.18.15.75
Source: [HDPNameNode, step 15] has 1 sources
    10.18.15.75
Data thread 139698190071552 is monitoring [HDPHBaseMaster] data source
    10.18.15.75
 Data thread 139698179581696 is monitoring [HDPResourceManager] data source
    10.18.15.75
Data thread 139698169091840 is monitoring [HDPSlaves] data source
    10.18.15.75
Data thread 139698024384256 is monitoring [HDPHistoryServer] data source
     10.18.15.75
Data thread 139698158601984 is monitoring [HDPNameNode] data source
    10.18.15.75


There is enough space available, Even Namenode rrd files and other related files are there
under /var/lib/ganglia/rrds.
 
[pts/0][10:26:37:root@INBBRDSSVM50 ] /var/lib/ganglia/rrds>ll
total 72
drwxr-xr-x. 4 nobody nobody  4096 Jun  9 16:29 HDPHBaseMaster
drwxr-xr-x. 4 nobody nobody  4096 Jun  9 16:29 HDPNameNode
drwxr-xr-x. 4 nobody nobody  4096 Jun  9 16:29 HDPResourceManager
 drwxr-xr-x. 4 nobody nobody  4096 Jun  9 16:29 HDPSlaves
drwxr-xr-x. 2 nobody nobody 57344 Jun 12 17:25 __SummaryInfo__

In /var/log/messeges there is no Error

Jun 13 10:01:23 INBBRDSSVM50 ntpd[1935]: time reset +0.155663 s
 Jun 13 10:03:52 INBBRDSSVM50 ntpd[1935]: synchronized to 172.18.225.143, stratum 5
Jun 13 10:20:06 INBBRDSSVM50 rrdcached[11080]: caught SIGTERM
Jun 13 10:20:06 INBBRDSSVM50 rrdcached[11080]: starting shutdown
Jun 13 10:20:07 INBBRDSSVM50 rrdcached[11080]: clean shutdown; all RRDs flushed
 Jun 13 10:20:07 INBBRDSSVM50 rrdcached[11080]: goodbye
Jun 13 10:20:12 INBBRDSSVM50 rrdcached[10344]: starting up
Jun 13 10:20:12 INBBRDSSVM50 rrdcached[10344]: listening for connections
Jun 13 10:24:27 INBBRDSSVM50 nagios: Auto-save of retention data completed successfully.
 

Graphs are displayed on Ganglia WEB UI but the same are not Displayed on Ambari Web-UI



Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
 Website: http://www.tcs.com


-----Siddharth Wagle <swagle@hortonworks.com> wrote: ----- 
 To: "user@ambari.apache.org" <user@ambari.apache.org>
From: Siddharth Wagle <swagle@hortonworks.com>
 Date: 06/12/2014 08:59PM

Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI
 
Hi Meghavi,

You should be able to debug this by editing, /etc/ganglia/hdp/gmetad.conf

Search for debug_level and set to above 1. 

Setting debug level above 1 should start process in foreground and you can also look at the
log file (/var/log/messages) for errors.
  Also, make there are rrd files present for HDFS by looking a say Namenode rrd files under
/var/lib/ganglia/rrds.

Some other this to look for, make sure there is enough space on the mount point where ganglia
is writing files.
  
-Sid


On Thu, Jun 12, 2014 at 6:41 AM, Meghavi Sugandhi <meghavi.sugandhi@tcs.com> wrote:
   
 Hi,

I have disabled the SELinux.
 
[pts/2][19:06:59:root@INBBRDSSVM50 ] /u01/HomeDir/NGA>getenforce
Disabled

But stil I am not able to view all graphs on Ambari Web_UI.
Only Hbase Graphs are visible but HDFS and YARN realated graphs are Not visible.
  Same case with Heatmaps.

/var/log/messeges

Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for HistoryServer;OK;service_sys_logger;TCP OK - 0.000 second response time
on port 8666
  Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30409]: OK: Hadoop: ganglia_monitor_process_ok# Event
Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for HistoryServer(OK),
TCP OK - 0.000 second response time on port 8666
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for HistoryServer;OK;notify-service-by-email;TCP OK - 0.000 second response
time on port 8666
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE ALERT: INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for HBase Master;OK;HARD;4;TCP OK - 0.001 second response time on port 8663
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for HBase Master;OK;service_sys_logger;TCP OK - 0.001 second response time
on port 8663
  Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30417]: OK: Hadoop: ganglia_monitor_process_ok# Event
Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for HBase
Master(OK), TCP OK - 0.001 second response time on port 8663
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for HBase Master;OK;notify-service-by-email;TCP OK - 0.001 second response
time on port 8663
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE ALERT: INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for NameNode;OK;HARD;4;TCP OK - 0.000 second response time on port 8661
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for NameNode;OK;service_sys_logger;TCP OK - 0.000 second response time on
port 8661
  Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30425]: OK: Hadoop: ganglia_monitor_process_ok# Event
Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for NameNode(OK),
TCP OK - 0.000 second response time on port 8661
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for NameNode;OK;notify-service-by-email;TCP OK - 0.000 second response time
on port 8661
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE ALERT: INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for ResourceManager;OK;HARD;4;TCP OK - 0.000 second response time on port
8664
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for ResourceManager;OK;service_sys_logger;TCP OK - 0.000 second response time
on port 8664
  Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30433]: OK: Hadoop: ganglia_monitor_process_ok# Event
Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for ResourceManager(OK),
TCP OK - 0.000 second response time on port 8664
  Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia
Monitor process for ResourceManager;OK;notify-service-by-email;TCP OK - 0.000 second response
time on port 8664
  Jun 12 18:24:27 INBBRDSSVM50 nagios: Auto-save of retention data completed successfully.


I tried by restting whole setup of ambari.
Even tried ambari-server and aganet, hdp-gmond and hdp-gmetad restart, but Graphs are not
displayed.
  
I am able to view all graphs on ganglia web UI but those are not visible on Ambari Web UI

:(



Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
  Website: http://www.tcs.com

-----Meghavi Sugandhi <meghavi.sugandhi@tcs.com> wrote: -----
  
To: user@ambari.apache.org
  From: Meghavi Sugandhi <meghavi.sugandhi@tcs.com>
Date: 06/10/2014 06:19PM
  
Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI

 
  Hi,

Is SELinux disabling is last option to resolve the problem.


Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
  Website: http://www.tcs.com

-----Olivier Renault <orenault@hortonworks.com> wrote: -----
  
To: user@ambari.apache.org
  From: Olivier Renault <orenault@hortonworks.com>
Date: 06/10/2014 04:30PM
Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI

  
If it's permissive, it's not in your way. It will show what it would have stop but it should
not stop anything.ÿ

Olivier

  
 On 10 June 2014 09:36, Meghavi Sugandhi <meghavi.sugandhi@tcs.com> wrote:
    
Hi

Getenforce status is Permissive only.

[pts/0][14:05:43:root@INBBRDSSVM50 ] /var/nagios>getenforce
Permissive

but its not working. Do I have to disable it?
   Is there any other option?


Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
Website: http://www.tcs.com   

-----Olivier Renault <orenault@hortonworks.com> wrote: -----
   To: user@ambari.apache.org
From: Olivier Renault <orenault@hortonworks.com>
Date: 06/10/2014 12:58PM
   

Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI

You can run 
 getenforce, it should give you the status (enforcing, disabled or permissive) 
 If you're in enforcing, you can switch it to permissive by running 
 setenforce 0
 Fyi, I don't believe that Ambari can runÿ with selinux enabled. It might be better to switch
it off once for all by editing /etc/selinux/config
 Olivier 
 
On 10 Jun 2014 08:21, "Meghavi Sugandhi" <meghavi.sugandhi@tcs.com> wrote:
    
Hi 

After running the below command,
echo 0 >/selinux/enforce
I am still getting same error in /var/log/messages.

Junÿ  9 15:40:59 INBBRDSSVM50 setroubleshoot: SELinux is preventing  /usr/bin/python from
getattr access on the file  /var/lib/ganglia/rrds/HDPNameNode/INBBRDSSVM50/rpcdetailed.rpcdetailed.RegisterDatanodeNumOps.rrd.
 For complete SELinux messages. run sealert -l  62ff979d-6c24-4a58-8a1b-c463f6b34533
  Junÿ 9 15:41:00 INBBRDSSVM50 setroubleshoot: SELinux is preventing  /usr/bin/python from
read access on the file  /var/lib/ganglia/rrds/HDPSlaves/__SummaryInfo__/dfs.datanode.BytesWritten.rrd.
 For complete SELinux messages. run sealert -l  3dbb1d53-e603-490b-9395-6166581428e3
  Junÿ 9 15:41:01 INBBRDSSVM50 setroubleshoot: SELinux is preventing  /usr/sbin/httpd from
getattr access on the file /var/nagios/status.dat.  For complete SELinux messages. run sealert
-l  f693d59d-37b4-489e-837b-eb6b79375298
  Junÿ 9 15:41:02 INBBRDSSVM50 setroubleshoot: SELinux is preventing  /usr/sbin/httpd from
read access on the file /var/nagios/status.dat. For  complete SELinux messages. run sealert
-l  a7ea6343-d141-4d9f-bf35-8930130ac881
  Junÿ 9 15:41:48 INBBRDSSVM50 setroubleshoot: SELinux is preventing  /usr/bin/python from
getattr access on the file  /var/lib/ganglia/rrds/HDPResourceManager/INBBRDSSVM50/load_one.rrd.
For  complete SELinux messages. run sealert -l  62ff979d-6c24-4a58-8a1b-c463f6b34533
  



Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
Website: http://www.tcs.com
    
-----Yusaku Sako <yusaku@hortonworks.com> wrote: -----
    To: user@ambari.apache.org
From: Yusaku Sako <yusaku@hortonworks.com>
Date: 06/10/2014 11:21AM
    Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI

You can run the following to temporarily disable SELinux until the next server reboot.
echo 0 >/selinux/enforce

    
Yusaku


 On Mon, Jun 9, 2014 at 10:46 PM, Meghavi Sugandhi <meghavi.sugandhi@tcs.com> wrote:
     

Hi Sid,

Thanks for reply.

All rrd files related to Namenode, HBase, Mapreduceare are being written to /var/lib/ganglia/rrds/

     I cant disable SELinux. Because disabling it requires rebooting of system.

Is there any other way which can handle this issue.

Few graphs that of hbase are visible but that of HDFS and YARN Mapreduce are Not Displyed
     


Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
Website: http://www.tcs.com
     
-----Siddharth Wagle <swagle@hortonworks.com> wrote: -----
     To: "user@ambari.apache.org" <user@ambari.apache.org>
From: Siddharth Wagle <swagle@hortonworks.com>
     Date: 06/09/2014 11:45PM
Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI


Hi Meghavi,

Are you able to obtain metrics data by turning off SELinux temporarily on Ganglia host ?
     
Also, are rrd files being written to /var/lib/ganglia/rrds/ ?
 
-Sid



On Mon, Jun 9, 2014 at 4:19 AM, Meghavi Sugandhi <meghavi.sugandhi@tcs.com> wrote:
       
  Hello,

I am using Ambari 1.4.4.
 I have installed it successfully and all services are up and running
But I am not able to view few graphs and heatmaps
I am getting following error in /var/log/messeges-->

Junÿ 9 15:40:59 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python from getattr
access on the file /var/lib/ganglia/rrds/HDPNameNode/INBBRDSSVM50/rpcdetailed.rpcdetailed.RegisterDatanodeNumOps.rrd.
For complete SELinux messages. run sealert -l 62ff979d-6c24-4a58-8a1b-c463f6b34533
      Junÿ 9 15:41:00 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python
from read access on the file /var/lib/ganglia/rrds/HDPSlaves/__SummaryInfo__/dfs.datanode.BytesWritten.rrd.
For complete SELinux messages. run sealert -l 3dbb1d53-e603-490b-9395-6166581428e3
      Junÿ 9 15:41:01 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/sbin/httpd
from getattr access on the file /var/nagios/status.dat. For complete SELinux messages. run
sealert -l f693d59d-37b4-489e-837b-eb6b79375298
      Junÿ 9 15:41:02 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/sbin/httpd
from read access on the file /var/nagios/status.dat. For complete SELinux messages. run sealert
-l a7ea6343-d141-4d9f-bf35-8930130ac881
      Junÿ 9 15:41:48 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python
from getattr access on the file /var/lib/ganglia/rrds/HDPResourceManager/INBBRDSSVM50/load_one.rrd.
For complete SELinux messages. run sealert -l 62ff979d-6c24-4a58-8a1b-c463f6b34533
      
SElinux is in Permissive mode

and owner of /var/nagios
drwxr-xr-x.ÿ 4 nagios nagios 4096 Junÿ 9 15:52 nagios

and owner of /var/lib/ganglia/rrds is nobody:nobody
drwxr-xr-x. 7 nobody nobody 4096 Junÿ 6 11:50 rrds
      
Thanks & Regards,
Meghavi Sugandhi
Tata Consultancy Services
Mailto: meghavi.sugandhi@tcs.com
Website: http://www.tcs.com
      =====-----=====-----=====
 Notice: The information contained in this e-mail
 message and/or attachments to it may contain 
 confidential or privileged information. If you are 
 not the intended recipient, any dissemination, use, 
 review, distribution, printing or copying of the 
 information contained in this e-mail message 
 and/or attachments to it are strictly prohibited. If 
 you have received this communication in error, 
 please notify us by reply e-mail or telephone and 
 immediately and permanently delete the message 
 and any attachments. Thank you
  
  
 CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.     
  
 CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.      
 CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.   


-- 
  
   
    
     Olivier Renault        
           Solution Engineer          ,           Hortonworks                            
  
               Mobile:              +44 7500 933 036            
               Email:              orenault@hortonworks.com            
               Website:              http://www.hortonworks.com/            
                                                                                         
               
  
       Latest From Our Blog:               How to Analyze Server Logs with Cascading and Apache
Hadoop on Hortonworks Data Platform             
 CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.  
  
 CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You. 
  
 CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.
Mime
View raw message