On Thu, Aug 15, 2013 at 12:05 PM, Terry = P. <texpilot@gmail.com> wrote:

Hi Keith,
Many thanks for your det= ailed reply. I forgot to mention that yes indeed this is on Accumulo 1.4.2,= and it was the write-ahead logs that were the issue -- partly because two = of the tabletservers were not properly shutdown before the re-IP operation,= so recovery may have been needed on them.

My naivety on Zookeeper certainly hampered the research as well.= =A0 How does one "look in zookeeper to see what is going on?"=A0 = Any pointers would be really helpful.

I wish we could go to 1.= 5 and take advantage of the walogs in HDFS, but no can do at this point unf= ortunately.

On Thu, Aug 15, 2013 at 10:24 AM, Keith Tu= rner <keith@deenlo.com> wrote:

On Thu, Aug 15, 2013 at 11:01 A= M, Terry P. <texpilot@gmail.com> wrote:

Greetin= gs everyone,
We had to re-IP our entire cluster recently to change= subnetworks, and we essentially lost everything (it was development, so no= big deal).=A0 However, doing a re-IP operation may be required in actual o= perational cases, and I'd like to know if it can be done or not so we c= an note it for the future (as in document "what not to do" to avo= id data loss).

The issue we had was that after shutting down the cluster, re-IPi= ng all servers, and starting everything back up, the tablets were still ass= igned to Tabletservers with the old IP addresses, even though all the hostn= ames were the same.=A0 So the system showed 3 Tabletservers, but no tablets= , and no entries in the tables where previously there were 400 million.

So:

A) Does Zookeeper track Tabletservers by IP addr= ess only, and not hostname?

It does track by IP address, but not only IP address. =A0Each tablet serve= r has an ephemeral node in zookeeper under the IP address. =A0This ehpemera= l node should go away when the tserver process dies, and then the master wi= ll assume that tserver is dead. =A0The location of a tablet in the metadata= table is conceptually <ephemeral node id>+<IP address>, so onc= e that ephemeral node goes away the location in metadata table is assumed i= nvalid and the tablet is reassigned. =A0 If another tserver starts at the s= ame IP, then the master can differentiate because the ephemeral node is dif= ferent.

You can look at the children nodes under a tserver ip i= n zookeeper. =A0Look at the data for the lowest numbered ephemeral node to = to get infor about who holds the lock for that IP.

B) If A is true, is there a mechanism to change those entr= ies in Zookeeper so that a re-IP operation could be performed?

A first step would be to look = in zookeeper and see what going on with the ephemeral nodes. =A0=A0

In Accumulo 1.3 and 1.4 one thing that normally causes pr= oblems when changing lots of IP addrs is write ahead logs. =A0 Tablets poin= t to their write ahead logs using the IP address of the logger. This can ca= use walog recovery to fail. =A0In 1.5 walog are stored in HDFS so this not = an issue.