accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terry P." <texpi...@gmail.com>
Subject Re: Accumulo Standby Master question
Date Wed, 13 Nov 2013 23:43:07 GMT
Correction: just did an Accumulo cluster shutdown and the gc on the
secondary namenode did not shutdown along with the rest of the cluster. No
errors on the Master when stop-all.sh was called.

No time to research right now though, but wanted to let everyone know.


On Wed, Nov 13, 2013 at 4:29 PM, Terry P. <texpilot@gmail.com> wrote:

> I was able to get to the 1.5.0 source, and start-here.sh has the same
> problem as described above for 1.4.2.
>
>
> On Wed, Nov 13, 2013 at 4:12 PM, Terry P. <texpilot@gmail.com> wrote:
>
>> Revisiting this, looks like I found even when I have my secondary
>> namenode hostname in the gc file on the master and secondary namenode (my
>> standby master), the gc process would not start on any node but the master.
>>
>> Looking at the config.sh script on lines 102-104:
>>
>> if [ -f "$ACCUMULO_HOME/conf/gc" ]; then
>>     GC=`grep -v '^#' "$ACCUMULO_HOME/conf/gc" | head -1`
>> fi
>>
>> That only puts the first node found in the GC variable, similar to what
>> is done with MASTER.
>>
>> Then in start-here.sh, whereas the masters file is checked to see if the
>> current host is in the masters file (starting at line 38):
>>
>> for host in $HOSTS
>> do
>>     *if grep -q "^${host}\$" $ACCUMULO_HOME/conf/masters*
>>     then
>>        ${bin}/accumulo
>> org.apache.accumulo.server.master.state.SetGoalState NORMAL
>>        ${bin}/start-server.sh $host master
>>        break
>>     fi
>> done
>>
>> *For GC, only the ${GC} variable is checked (starting on line 48), and
>> start of the gc is only attempted on the one host in the GC variable*:
>>
>> for host in $HOSTS
>> do
>>     *if [ ${host} = ${GC} ]*
>>     then
>>         ${bin}/start-server.sh *$GC* gc "garbage collector"
>>         break
>>     fi
>> done
>>
>> *I've modified my start-here.sh script's GC section to*:
>>
>> for host in $HOSTS
>> do
>>     *if grep -q "^${host}\$" $ACCUMULO_HOME/conf/gc*
>>     then
>>         ${bin}/start-server.sh *${host}* gc "garbage collector"
>>         break
>>     fi
>> done
>>
>> And that did the trick.  I didn't see any changes needed in the
>> stop-here.sh script, as it already iterates over all possible processes to
>> stop them if they are found.
>>
>> Can anyone think of anything I might have missed?
>>
>> I searched for any JIRAs on this but did not find any.
>>
>> I tried to download the 1.5.0 binaries, but download link doesn't work
>> for me (has happened before with my work's overly restrictive internet
>> gateway) -- but FYI I asked my brother to download it, and the "mirrors"
>> link under GENERIC BINARIES didn't work for his as well.
>>
>> If I can get my hands on the 1.5.0 binaries, I'll check to see if this is
>> already OBE in 1.5.0 today else it will have to wait until tonight.
>>
>>
>>
>> On Wed, Nov 6, 2013 at 12:12 PM, Terry P. <texpilot@gmail.com> wrote:
>>
>>> Ahh thanks Billie, I'll stick with just one monitor then.  Thanks!
>>>
>>>
>>> On Wed, Nov 6, 2013 at 10:32 AM, Billie Rinaldi <
>>> billie.rinaldi@gmail.com> wrote:
>>>
>>>> On Tue, Nov 5, 2013 at 12:51 PM, Josh Elser <josh.elser@gmail.com>wrote:
>>>>
>>>>> It looks like you can configure multiple hosts for GC and they'll use
>>>>> ZooKeeper to perform failover (like the master).
>>>>>
>>>>> Tracers -- You can run multiple tracer processes. You likely don't
>>>>> need 1:1 as you run tservers, but you can run a few if you're concerned
>>>>> about it. They're not required for Accumulo operation.
>>>>>
>>>>> Same for the monitor. If you need to have multiple running for
>>>>> failover purposes, it looks like you can specify multiple and it will
just
>>>>> launch a monitor on each host you specified. There's no centralize URL
you
>>>>> can always hit here. You would have to check each one to find one that
was,
>>>>> unless you want to run some sort of reverse-proxy in front of them all.
>>>>>
>>>>
>>>> I think additional monitors won't work entirely as expected.  The log
>>>> forwarding from the other processes is set up when the processes are
>>>> started, and the logs are only sent to the first host in the monitor file.
>>>>
>>>>
>>>>> _obligatory I just looked to source code to come to this conclusion_
>>>>>
>>>>>
>>>>> On 11/5/13, 3:13 PM, Terry P. wrote:
>>>>>
>>>>>> I put my secondary namenode in the masters file for the first time
>>>>>> with
>>>>>> this latest Accumulo 1.4.2 cluster deployment so it would run as
a
>>>>>> Standby Accumulo Master which I read about in the User Guide.
>>>>>>  Recently
>>>>>> had the Master lose its Zookeeper lock (network glitch, being
>>>>>> researched), and was glad to see the secondary namenode Master process
>>>>>> took over as it should.
>>>>>>
>>>>>> But if my Namenode / Accumulo Master server goes down, I also lose
the
>>>>>> gc, monitor, and tracer processes as well. *Can I configure the
>>>>>>
>>>>>> secondary namenode in gc, tracer, and monitor files as well, or should
>>>>>> they run on only one host at a time*?  In this case, the GC also
lost
>>>>>>
>>>>>> its Zookeeper lock, which resulted in a cluster with no GC running
at
>>>>>> all until I caught it.
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message