accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Accumulo Standby Master question
Date Thu, 14 Nov 2013 00:18:39 GMT
Sounds like that would be a good ticket to create that affects 1.4.4 and 
1.5.0 something we should try to fix for 1.4.5, 1.5.1 and 1.6.0.

Would be great to get a patch too!

On 11/13/13, 3:43 PM, Terry P. wrote:
> Correction: just did an Accumulo cluster shutdown and the gc on the
> secondary namenode did not shutdown along with the rest of the cluster.
> No errors on the Master when stop-all.sh was called.
>
> No time to research right now though, but wanted to let everyone know.
>
>
> On Wed, Nov 13, 2013 at 4:29 PM, Terry P. <texpilot@gmail.com
> <mailto:texpilot@gmail.com>> wrote:
>
>     I was able to get to the 1.5.0 source, and start-here.sh has the
>     same problem as described above for 1.4.2.
>
>
>     On Wed, Nov 13, 2013 at 4:12 PM, Terry P. <texpilot@gmail.com
>     <mailto:texpilot@gmail.com>> wrote:
>
>         Revisiting this, looks like I found even when I have my
>         secondary namenode hostname in the gc file on the master and
>         secondary namenode (my standby master), the gc process would not
>         start on any node but the master.
>
>         Looking at the config.sh script on lines 102-104:
>
>         if [ -f "$ACCUMULO_HOME/conf/gc" ]; then
>              GC=`grep -v '^#' "$ACCUMULO_HOME/conf/gc" | head -1`
>         fi
>
>         That only puts the first node found in the GC variable, similar
>         to what is done with MASTER.
>
>         Then in start-here.sh, whereas the masters file is checked to
>         see if the current host is in the masters file (starting at line
>         38):
>
>         for host in $HOSTS
>         do
>         *if grep -q "^${host}\$" $ACCUMULO_HOME/conf/masters*
>              then
>                 ${bin}/accumulo
>         org.apache.accumulo.server.master.state.SetGoalState NORMAL
>                 ${bin}/start-server.sh $host master
>                 break
>              fi
>         done
>
>         *For GC, only the ${GC} variable is checked (starting on line
>         48), and start of the gc is only attempted on the one host in
>         the GC variable*:
>
>         for host in $HOSTS
>         do
>         *if [ ${host} = ${GC} ]*
>              then
>                  ${bin}/start-server.sh *$GC* gc "garbage collector"
>                  break
>              fi
>         done
>
>         *I've modified my start-here.sh script's GC section to*:
>
>         for host in $HOSTS
>         do
>         *if grep -q "^${host}\$" $ACCUMULO_HOME/conf/gc*
>              then
>                  ${bin}/start-server.sh *${host}* gc "garbage collector"
>                  break
>              fi
>         done
>
>         And that did the trick.  I didn't see any changes needed in the
>         stop-here.sh script, as it already iterates over all possible
>         processes to stop them if they are found.
>
>         Can anyone think of anything I might have missed?
>
>         I searched for any JIRAs on this but did not find any.
>
>         I tried to download the 1.5.0 binaries, but download link
>         doesn't work for me (has happened before with my work's overly
>         restrictive internet gateway) -- but FYI I asked my brother to
>         download it, and the "mirrors" link under GENERIC BINARIES
>         didn't work for his as well.
>
>         If I can get my hands on the 1.5.0 binaries, I'll check to see
>         if this is already OBE in 1.5.0 today else it will have to wait
>         until tonight.
>
>
>
>         On Wed, Nov 6, 2013 at 12:12 PM, Terry P. <texpilot@gmail.com
>         <mailto:texpilot@gmail.com>> wrote:
>
>             Ahh thanks Billie, I'll stick with just one monitor then.
>             Thanks!
>
>
>             On Wed, Nov 6, 2013 at 10:32 AM, Billie Rinaldi
>             <billie.rinaldi@gmail.com <mailto:billie.rinaldi@gmail.com>>
>             wrote:
>
>                 On Tue, Nov 5, 2013 at 12:51 PM, Josh Elser
>                 <josh.elser@gmail.com <mailto:josh.elser@gmail.com>> wrote:
>
>                     It looks like you can configure multiple hosts for
>                     GC and they'll use ZooKeeper to perform failover
>                     (like the master).
>
>                     Tracers -- You can run multiple tracer processes.
>                     You likely don't need 1:1 as you run tservers, but
>                     you can run a few if you're concerned about it.
>                     They're not required for Accumulo operation.
>
>                     Same for the monitor. If you need to have multiple
>                     running for failover purposes, it looks like you can
>                     specify multiple and it will just launch a monitor
>                     on each host you specified. There's no centralize
>                     URL you can always hit here. You would have to check
>                     each one to find one that was, unless you want to
>                     run some sort of reverse-proxy in front of them all.
>
>
>                 I think additional monitors won't work entirely as
>                 expected.  The log forwarding from the other processes
>                 is set up when the processes are started, and the logs
>                 are only sent to the first host in the monitor file.
>
>
>                     _obligatory I just looked to source code to come to
>                     this conclusion_
>
>
>                     On 11/5/13, 3:13 PM, Terry P. wrote:
>
>                         I put my secondary namenode in the masters file
>                         for the first time with
>                         this latest Accumulo 1.4.2 cluster deployment so
>                         it would run as a
>                         Standby Accumulo Master which I read about in
>                         the User Guide.  Recently
>                         had the Master lose its Zookeeper lock (network
>                         glitch, being
>                         researched), and was glad to see the secondary
>                         namenode Master process
>                         took over as it should.
>
>                         But if my Namenode / Accumulo Master server goes
>                         down, I also lose the
>                         gc, monitor, and tracer processes as well. *Can
>                         I configure the
>
>                         secondary namenode in gc, tracer, and monitor
>                         files as well, or should
>                         they run on only one host at a time*?  In this
>                         case, the GC also lost
>
>                         its Zookeeper lock, which resulted in a cluster
>                         with no GC running at
>                         all until I caught it.
>
>                         Thanks in advance.
>
>
>
>
>
>

Mime
View raw message