accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terry P." <texpi...@gmail.com>
Subject Re: No info for Accumulo Master on Monitor Overview page
Date Fri, 02 Aug 2013 21:41:28 GMT
Thanks Charles, I finally got around to looking at this on a new
installation.  I didn't have to set the monitor to DEBUG mode though to
find the faulty path, as the default monitor_logger.xml config file creates
a DEBUG log (though you may have disabled it).

For those that are still on 1.4.x, take a look at the monitor DEBUG log
file.  Look for an entry with AccessControlException: Permission denied,
and the path it points to.  My mapred system directory is /mapred-system,
so for me the fix was:

sudo -u hdfs hadoop fs -chmod 775 /mapred-system

With that, the Accumulo Master box on the Monitor's Overview page populates
as it should.  Not certain if the hadoop permission fix will persist across
restarts though, as doesn't the daemon set the perms each time?  (I'm
running CDH3u5).

-Terry


On Fri, Jun 7, 2013 at 8:56 AM, Ott, Charles H. <CHARLES.H.OTT@saic.com>wrote:

> I had the exact same issue a couple weeks back.****
>
> ** **
>
> I was able to find out what specific HDFS path had permission issues by
> switching the Monitor logger to DEBUG mode.  Then a chmod 775 fixed the
> issue by giving the group write permissions.****
>
> ** **
>
> *From:* user-return-2668-CHARLES.H.OTT=saic.com@accumulo.apache.org[mailto:
> user-return-2668-CHARLES.H.OTT=saic.com@accumulo.apache.org] *On Behalf
> Of *Terry P.
> *Sent:* Thursday, June 06, 2013 6:13 PM
> *To:* user@accumulo.apache.org
> *Subject:* Re: No info for Accumulo Master on Monitor Overview page****
>
> ** **
>
> In our installation the hdfs account has hdfs as his primary group and
> hadoop as a secondary group.  Our accumulo account has accumulo as his
> primary group.****
>
> I added both hdfs and hadoop as secondary groups for the accumulo account,
> but no joy -- still a blank Accumulo Master summary box on the overview
> page.****
>
> So do both the accumulo and hdfs users have to have the same *primary *group
> then?
>
> I put accumulo into a separate group in order to protect the shared secret
> in accumulo-site.xml, but I guess the better approach is to have both
> accumulo and hdfs in the same Linux group and just protect this file by
> making it readable by owner only?  If both accumulo and hdfs users must
> have the same primary group, looks like that's about the only option.****
>
> ** **
>
> On Thu, Jun 6, 2013 at 4:57 PM, Eric Newton <eric.newton@gmail.com> wrote:
> ****
>
> Yea, that's a 1.4.2 issue.  You can work around it by putting the accumulo
> and hdfs accounts into the same group, or just ignore the display issue.**
> **
>
> ** **
>
> ACCUMULO-1282****
>
> ** **
>
> -Eric****
>
> ** **
>
> ** **
>
> On Thu, Jun 6, 2013 at 5:48 PM, Terry P. <texpilot@gmail.com> wrote:****
>
> Hi Eric,
> Sorry I should have thought of including a screenshot.  I've attached one
> to this email -- first one I've done, I hope it works just like a regular
> attachment?****
>
> To ensure I had a clean monitor log, I shutdown Accumulo and started it up
> again.  First there are all the [server.Accumulo] entries, here's the head
> of that which looks good:
>
> 2013-06-06 21:34:37,823 [server.Accumulo] INFO :  monitor starting
> 2013-06-06 21:34:37,824 [server.Accumulo] INFO :  Instance
> 22e13d96-d1ac-45c7-9d90-ae2e813e983c
> 2013-06-06 21:34:37,841 [server.Accumulo] INFO :  Data Version 4
> 2013-06-06 21:34:37,841 [server.Accumulo] INFO :  Attempting to talk to
> zookeeper
> 2013-06-06 21:34:38,026 [server.Accumulo] INFO :  Zookeeper connected and
> initialized, attemping to talk to HDFS
> 2013-06-06 21:34:38,030 [server.Accumulo] INFO :  Connected to HDFS
> 2013-06-06 21:34:38,034 [server.Accumulo] INFO :  gc.cycle.delay = 5m
> 2013-06-06 21:34:38,034 [server.Accumulo] INFO :  gc.cycle.start = 30s
> 2013-06-06 21:34:38,034 [server.Accumulo] INFO :  gc.port.client = 50091
> 2013-06-06 21:34:38,034 [server.Accumulo] INFO :  gc.threads.delete = 16
> 2013-06-06 21:34:38,034 [server.Accumulo] INFO :  general.classpaths =
>     $ACCUMULO_HOME/src/server/target/classes/,
>     $ACCUMULO_HOME/src/core/target/classes/,
>     $ACCUMULO_HOME/src/start/target/classes/,
>     $ACCUMULO_HOME/src/examples/target/classes/,
>         $ACCUMULO_HOME/lib/[^.].$ACCUMULO_VERSION.jar,
>         $ACCUMULO_HOME/lib/[^.].*.jar,
>         $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
>         $HADOOP_HOME/conf,
>         $HADOOP_HOME/[^.].*.jar,
>         $HADOOP_HOME/lib/[^.].*.jar,****
>
> Then at the very end I found (with only INFO messages in between):
>
> 2013-06-06 21:34:39,132 [monitor.Monitor] WARN :
> java.lang.NullPointerException
>         at
> org.apache.accumulo.server.monitor.Monitor.fetchData(Monitor.java:308)
>         at
> org.apache.accumulo.server.monitor.Monitor$2.run(Monitor.java:476)
>         at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>         at java.lang.Thread.run(Unknown Source)****
>
> That's the last entry in the monitor log.
>
> The second image I attached shows the Master Status page, showing there is
> no problem with that display.****
>
> This is Accumulo 1.4.2 running as linux account 'accumulo' -- HDFS runs as
> the hdfs account, which is why the NameNode section shows "permission
> denied."  NameNode and Accumulo Master reside on the same server.  OS is
> Red Hat Enterprise Linux 6.3 64-bit.****
>
>
> Thanks in advance,
> Terry****
>
> ** **
>
> On Thu, Jun 6, 2013 at 2:58 PM, Eric Newton <eric.newton@gmail.com> wrote:
> ****
>
> Would you have a screenshot?  Any messages in the monitor log?****
>
> ** **
>
> -Eric****
>
> ** **
>
> ** **
>
> On Thu, Jun 6, 2013 at 2:44 PM, Terry P. <texpilot@gmail.com> wrote:****
>
> Greetings all,
> We have two Accumulo clusters built from essentially the same build
> procedure.  Everything is running well with the exception that on the
> Accumulo Monitor Overview web page, the Accumulo Master box is blank -- the
> "Accumulo Master" title/hyperlink works (takes you to the "Master Server"
> page), but nothing is in the box.  The NameNode, JobTracker, and Zookeeper
> boxes are all populated.****
>
> On the Master Server page, I see the Master Status that shows 3 Online
> Tablet Servers, Loggers, Last GC, etc.  It also shows the Table List.  The
> Tablet Servers page looks good.****
>
> In these small clusters we have one Accumulo Master that runs on the
> NameNode, 3 DataNodes, 3 Zookeepers.  I'm in the process of firing up a
> Secondary NameNode which will also run an Alternate Accumulo Master in case
> the Master on the NameNode were to fail for some reason, but I have not
> fired up the Accumulo processes on the Secondary NameNode server yet at all.
> ****
>
> ** **
>
> Any idea where I can look to try and determine why the Accumulo Master box
> is empty?  It's not really a big deal to me, but I can see an Operator
> asking questions about it someday ;-)****
>
> Thanks and best regards,
> Terry****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>

Mime
View raw message