accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: Accumulo 1.7.1 on Docker
Date Tue, 22 Mar 2016 18:47:40 GMT
Sven, did you ever put up a Dockerfile?  I've done something like this
waiting for HDFS to come out of safe mode.

echo "Starting Hadoop"
${HADOOP_HOME}/sbin/start-dfs.sh
echo "Waiting for hadoop to come out of safeMode"
local x=0
while [ $x -lt 1 ] do # try 5 times, for slow computers
  ${HADOOP_HOME}/bin/hdfs dfsadmin -safemode wait && x=5
  x=$(( $x + 1 ))
done

On Tue, Mar 22, 2016 at 2:41 PM, Josh Elser <josh.elser@gmail.com> wrote:

> Every time I use Docker, I seem to run into odd situations like this. Very
> frustrating.
>
> I would love to know what's happening and get this working reliably.
> Having an "Apache-owned" docker image for Accumulo would be nice (and would
> help us internally as well as externally, IMO).
>
>
> Andrew Hulbert wrote:
>
>> +1 for Josh's suggestion.
>>
>> Not sure if its the same problem, but I too had to add some sleeps for
>> single node accumulo startup scripts directly before and after init'ing
>> accumulo. I dug in once and it seemed that the datanode needed more time
>> to let other processes know the files in HDFS existed at /accumulo.
>> Perhaps it was awaiting the exit from safe mode? Three seconds did it
>> for me:
>>
>> https://github.com/jahhulbert-ccri/cloud-local/blob/master/bin/cloud-local.sh#L109
>>
>>
>>
>> On 03/14/2016 01:49 PM, Josh Elser wrote:
>>
>>> Well, I'm not sure wtf is going on, but I added some sleeps to your
>>> init.sh script (30seconds before starting ZooKeeper and before
>>> init'ing Accumulo) and now things worked just fine.
>>>
>>> I'm not seeing any Accumulo problems here, but rather the same Docker
>>> absurdities I run into every time I try to use it :). I also just
>>> upgrade to Docker 1.10.3 if that helps.
>>>
>>> Josh Elser wrote:
>>>
>>>> So, I got it running, and ran into the same problem you described.
>>>>
>>>> The odd thing is that both the Accumulo Master and TabletServer seemed
>>>> to be... stuck. They both claimed to be running, but they were both hung
>>>> (unable to communicate). This ultimately caused the TabletServer to lose
>>>> its ZooKeeper lock (and why the Shell didn't think it was running).
>>>>
>>>> I'm running into all sorts of docker issues ATM (trying to upgrade now,
>>>> hoping it will help), but I'd recommend trying to increase the amount of
>>>> heap for both the master and tserver from 128M to 256M in
>>>> /accumulo-1.7.1/conf/accumulo-env.sh (the -Xmx value).
>>>>
>>>> Josh Elser wrote:
>>>>
>>>>> Ok! Thanks, I'll look for that init.sh script.
>>>>>
>>>>> We're not too... docker-savvy here yet :), but we can definitely help
>>>>> out if we can find an automation script. Let me get back to you.
>>>>>
>>>>> Sven Hodapp wrote:
>>>>>
>>>>>> Hi Josh,
>>>>>>
>>>>>> thanks for your answer.
>>>>>> Currently I haven't uploaded the Dockerfile... If you want I'll upload
>>>>>> it!
>>>>>> (But currently it only is an debian with ssh, rsync, jdk7, the
>>>>>> unzipped distributions of hadoop, zookeeper and accumulo, and the
set
>>>>>> of environment variables)
>>>>>>
>>>>>> Currently, and for testing, I start all things manually.
>>>>>> The steps I do is documented in /init.sh within the docker container.
>>>>>> I've enshured that dfs and zookeeper are up before starting accumulo.
>>>>>>
>>>>>> Regards,
>>>>>> Sven
>>>>>>
>>>>>> ----- Ursprüngliche Mail -----
>>>>>>
>>>>>>> Von: "Josh Elser"<josh.elser@gmail.com>
>>>>>>> An: "user"<user@accumulo.apache.org>
>>>>>>> Gesendet: Montag, 14. März 2016 15:41:16
>>>>>>> Betreff: Re: Accumulo 1.7.1 on Docker
>>>>>>>
>>>>>>
>>>>>> Hi Sven,
>>>>>>>
>>>>>>> I can't seem to find the source for your Docker image (nor am
smart
>>>>>>> enough to figure out how to extract it). Any chance you can point
>>>>>>> us to
>>>>>>> that?
>>>>>>>
>>>>>>> That said, I did run your image, and it looks like you didn't
>>>>>>> actually
>>>>>>> start Hadoop, ZooKeeper, or Accumulo.
>>>>>>>
>>>>>>> Our documentation treats Hadoop and ZooKeeper primarily as
>>>>>>> prerequisites, so check their docs for good instructions.
>>>>>>>
>>>>>>> For Accumulo -
>>>>>>>
>>>>>>> Configuration:
>>>>>>>
>>>>>>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_installation
>>>>>>>
>>>>>>> Initialization:
>>>>>>>
>>>>>>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_initialization
>>>>>>>
>>>>>>>
>>>>>>> Starting:
>>>>>>>
>>>>>>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_starting_accumulo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sven Hodapp wrote:
>>>>>>>
>>>>>>>> Dear reader,
>>>>>>>>
>>>>>>>> I'd like to create a docker image with accumulo 1.7.1.
>>>>>>>> I've install it from scratch with hadoop-2.7.2, zookeeper-3.4.8
and
>>>>>>>> accumulo-1.7.1.
>>>>>>>> I'm going though the installation and every time I'll end
up like
>>>>>>>> that:
>>>>>>>>
>>>>>>>> root@deebd8e29683:/# accumulo shell -u root
>>>>>>>> Password: ******
>>>>>>>> 2016-03-13 13:18:31,627 [trace.DistributedTrace] INFO : SpanReceiver
>>>>>>>> org.apache.accumulo.tracer.ZooTraceClient was loaded successfully.
>>>>>>>> 2016-03-13 13:20:31,955 [impl.ServerClient] WARN : There
are no
>>>>>>>> tablet servers:
>>>>>>>> check that zookeeper and accumulo are running.
>>>>>>>>
>>>>>>>> Anybody got an idea whats wrong? I have no idea anymore...
>>>>>>>>
>>>>>>>> I'll currently share this docker image on docker hub.
>>>>>>>> If you want to try it you can simply start:
>>>>>>>>
>>>>>>>> docker run -it themerius/accumulo /bin/bash
>>>>>>>>
>>>>>>>> All binaries and configs are in there. In ./init.sh are my
setup
>>>>>>>> steps.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Sven
>>>>>>>>
>>>>>>>
>>

Mime
View raw message