accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terry P." <texpi...@gmail.com>
Subject Re: Accumulo init over existing instance
Date Wed, 09 Oct 2013 00:48:59 GMT
Thanks Keith, great information.  We're just entering formal test though so
1.5 isn't an option with this project.  But great to know that the move to
HDFS with the walogs at least helped this issue significantly it looks like.

Thanks again.


On Tue, Oct 8, 2013 at 7:37 PM, Keith Turner <keith@deenlo.com> wrote:

>
>
>
> On Tue, Oct 8, 2013 at 7:50 PM, Terry P. <texpilot@gmail.com> wrote:
>
>> Thanks Jared.
>>
>> John, thanks for the warning!  I lost a dev cluster once when we had to
>> re-IP the Accumulo servers, but reverse DNS wasn't configured and I assumed
>> that was why.  Guess that wasn't.
>>
>> Keith, I read through ACCUMULO-1585 but it wasn't completely clear if the
>> change proposed would also allow a server or servers in a cluster to have
>> its IP address changed.  I hope it will, as while having to re-IP a server
>> or cluster is fairly rare, it certainly happens (as it did in our case).
>>
>
> I think moving from 1.4 to 1.5 will help.  In 1.4 Accumulo has logger
> servers that store write ahead logs/edit logs.  Data stored on these
> loggers are needed when a tablet server crashes.  Accumulo stores pointers
> to loggers using IP addresses.  So if the IP address of the machine running
> a logger changes, then Accumulo can no longer find the data need to recover
> from a fault.
>
> Starting w/ 1.5 Accumulo started storing write ahead logs in HDFS and the
> pointers to these WAlogs are now hdfs paths.   The IP addrs that are still
> stored in 1.5 in zookeeper and the metadata table are more transient.  For
> example locations of tablets are stored in the metadata table using IP
> addrs.  If a tablet server dies and restarts w/ a different IP addr its
> probably ok, because the tablet will just be reassigned to a different
> tablet server.  You may lose some locality because Accumulo prefers to
> assign a tablet to the last place it compacted data, but things should
> still work.
>
> I have not tried changing IP addrs w/ a 1.5 instance, so I do not know if
> there are other problems.  But I do know that the walogs were a problem in
> 1.4 and that should no longer be a problem in 1.5.
>
>
>>
>> Thanks all,
>> Terry
>>
>>
>>
>> On Tue, Oct 8, 2013 at 5:14 PM, Keith Turner <keith@deenlo.com> wrote:
>>
>>>
>>>
>>>
>>> On Tue, Oct 8, 2013 at 6:07 PM, John Vines <vines@apache.org> wrote:
>>>
>>>> Like Jared said, wiping /accumulo out of hdfs is all you need to do.
>>>>
>>>> But Accumulo still uses IP addresses internally, so I'm not quite
>>>> certain you're going to achieve what you set out for.
>>>>
>>>
>>> Until 1.6.0 w/ ACCUMULO-1585
>>>
>>>
>>>>
>>>>
>>>> On Tue, Oct 8, 2013 at 5:32 PM, Terry P. <texpilot@gmail.com> wrote:
>>>>
>>>>> So reverse DNS wasn't working when I deployed my new cluster, thus all
>>>>> my Tablet Servers were showing up in the Monitor as IP addresses (even
>>>>> though all configuration files had hostnames only).  Lesson learned:
trust,
>>>>> but verify (and ensure your hardened base servers still have nslookup
>>>>> and/or dig on them).
>>>>>
>>>>> Now that DNS is fixed, I want to wipe everything clean and re-init
>>>>> Accumulo to ensure everything is legit using hostnames to ensure the
>>>>> cluster is not tied to IP addresses.
>>>>>
>>>>> I know I need to do a new 'accumulo init' -- I'll pass in the same
>>>>> instance name, and my understanding is that will overwrite everything
>>>>> currently in Zookeeper.
>>>>>
>>>>> My question is: is there anything else I could/should do first to
>>>>> "clean up" from this botched instance?  E.g. should I delete all files
in
>>>>> HDFS, the write-ahead logs on the Tablet Servers, etc.?  I'm running
>>>>> Accumulo 1.4.2.
>>>>>
>>>>> Thanks,
>>>>> Terry
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message