accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Accumulo 1.7.1 on Docker
Date Wed, 23 Mar 2016 00:45:30 GMT
Cool beans.

If you're interested, myself (as well as others, I'm sure) would love to 
work on this.

Andrew Hulbert wrote:
> With the single-node/docker I had to add a "sleep 3" after "hdfs
> dfsadmin -safemode" returns too though before the "accumulo init"
>
> I think having an Apache-owned docker for accumulo would rock! We have
> made a few but having a canonical one would be great.
>
> On 03/22/2016 02:47 PM, Michael Wall wrote:
>> Sven, did you ever put up a Dockerfile?  I've done something like this
>> waiting for HDFS to come out of safe mode.
>>
>> echo "Starting Hadoop"
>> ${HADOOP_HOME}/sbin/start-dfs.sh
>> echo "Waiting for hadoop to come out of safeMode"
>> local x=0
>> while [ $x -lt 1 ] do # try 5 times, for slow computers
>>   ${HADOOP_HOME}/bin/hdfs dfsadmin -safemode && x=5
>>   x=$(( $x + 1 ))
>> done
>>
>> On Tue, Mar 22, 2016 at 2:41 PM, Josh Elser <josh.elser@gmail.com
>> <mailto:josh.elser@gmail.com>> wrote:
>>
>>     Every time I use Docker, I seem to run into odd situations like
>>     this. Very frustrating.
>>
>>     I would love to know what's happening and get this working
>>     reliably. Having an "Apache-owned" docker image for Accumulo would
>>     be nice (and would help us internally as well as externally, IMO).
>>
>>
>>     Andrew Hulbert wrote:
>>
>>         +1 for Josh's suggestion.
>>
>>         Not sure if its the same problem, but I too had to add some
>>         sleeps for
>>         single node accumulo startup scripts directly before and after
>>         init'ing
>>         accumulo. I dug in once and it seemed that the datanode needed
>>         more time
>>         to let other processes know the files in HDFS existed at
>>         /accumulo.
>>         Perhaps it was awaiting the exit from safe mode? Three seconds
>>         did it
>>         for me:
>>         https://github.com/jahhulbert-ccri/cloud-local/blob/master/bin/cloud-local.sh#L109
>>
>>
>>
>>         On 03/14/2016 01:49 PM, Josh Elser wrote:
>>
>>             Well, I'm not sure wtf is going on, but I added some
>>             sleeps to your
>>             init.sh script (30seconds before starting ZooKeeper and before
>>             init'ing Accumulo) and now things worked just fine.
>>
>>             I'm not seeing any Accumulo problems here, but rather the
>>             same Docker
>>             absurdities I run into every time I try to use it :). I
>>             also just
>>             upgrade to Docker 1.10.3 if that helps.
>>
>>             Josh Elser wrote:
>>
>>                 So, I got it running, and ran into the same problem
>>                 you described.
>>
>>                 The odd thing is that both the Accumulo Master and
>>                 TabletServer seemed
>>                 to be... stuck. They both claimed to be running, but
>>                 they were both hung
>>                 (unable to communicate). This ultimately caused the
>>                 TabletServer to lose
>>                 its ZooKeeper lock (and why the Shell didn't think it
>>                 was running).
>>
>>                 I'm running into all sorts of docker issues ATM
>>                 (trying to upgrade now,
>>                 hoping it will help), but I'd recommend trying to
>>                 increase the amount of
>>                 heap for both the master and tserver from 128M to 256M in
>>                 /accumulo-1.7.1/conf/accumulo-env.sh (the -Xmx value).
>>
>>                 Josh Elser wrote:
>>
>>                     Ok! Thanks, I'll look for that init.sh script.
>>
>>                     We're not too... docker-savvy here yet :), but we
>>                     can definitely help
>>                     out if we can find an automation script. Let me
>>                     get back to you.
>>
>>                     Sven Hodapp wrote:
>>
>>                         Hi Josh,
>>
>>                         thanks for your answer.
>>                         Currently I haven't uploaded the Dockerfile...
>>                         If you want I'll upload
>>                         it!
>>                         (But currently it only is an debian with ssh,
>>                         rsync, jdk7, the
>>                         unzipped distributions of hadoop, zookeeper
>>                         and accumulo, and the set
>>                         of environment variables)
>>
>>                         Currently, and for testing, I start all things
>>                         manually.
>>                         The steps I do is documented in /init.sh
>>                         within the docker container.
>>                         I've enshured that dfs and zookeeper are up
>>                         before starting accumulo.
>>
>>                         Regards,
>>                         Sven
>>
>>                         ----- Ursprüngliche Mail -----
>>
>>                             Von: "Josh
>>                             Elser"<<mailto:josh.elser@gmail.com>josh.elser@gmail.com>
>>                             An: "user"<user@accumulo.apache.org
>>                             <mailto:user@accumulo.apache.org>>
>>                             Gesendet: Montag, 14. März 2016 15:41:16
>>                             Betreff: Re: Accumulo 1.7.1 on Docker
>>
>>
>>                             Hi Sven,
>>
>>                             I can't seem to find the source for your
>>                             Docker image (nor am smart
>>                             enough to figure out how to extract it).
>>                             Any chance you can point
>>                             us to
>>                             that?
>>
>>                             That said, I did run your image, and it
>>                             looks like you didn't
>>                             actually
>>                             start Hadoop, ZooKeeper, or Accumulo.
>>
>>                             Our documentation treats Hadoop and
>>                             ZooKeeper primarily as
>>                             prerequisites, so check their docs for
>>                             good instructions.
>>
>>                             For Accumulo -
>>
>>                             Configuration:
>>                             http://accumulo.apache.org/1.7/accumulo_user_manual.html#_installation
>>
>>                             Initialization:
>>                             http://accumulo.apache.org/1.7/accumulo_user_manual.html#_initialization
>>
>>
>>                             Starting:
>>                             http://accumulo.apache.org/1.7/accumulo_user_manual.html#_starting_accumulo
>>
>>
>>
>>
>>                             Sven Hodapp wrote:
>>
>>                                 Dear reader,
>>
>>                                 I'd like to create a docker image with
>>                                 accumulo 1.7.1.
>>                                 I've install it from scratch with
>>                                 hadoop-2.7.2, zookeeper-3.4.8 and
>>                                 accumulo-1.7.1.
>>                                 I'm going though the installation and
>>                                 every time I'll end up like
>>                                 that:
>>
>>                                 root@deebd8e29683:/# accumulo shell -u
>>                                 root
>>                                 Password: ******
>>                                 2016-03-13 13:18:31,627
>>                                 [trace.DistributedTrace] INFO :
>>                                 SpanReceiver
>>                                 org.apache.accumulo.tracer.ZooTraceClient
>>                                 was loaded successfully.
>>                                 2016-03-13 13:20:31,955
>>                                 [impl.ServerClient] WARN : There are no
>>                                 tablet servers:
>>                                 check that zookeeper and accumulo are
>>                                 running.
>>
>>                                 Anybody got an idea whats wrong? I
>>                                 have no idea anymore...
>>
>>                                 I'll currently share this docker image
>>                                 on docker hub.
>>                                 If you want to try it you can simply
>>                                 start:
>>
>>                                 docker run -it themerius/accumulo
>>                                 /bin/bash
>>
>>                                 All binaries and configs are in there.
>>                                 In ./init.sh are my setup
>>                                 steps.
>>
>>                                 Regards,
>>                                 Sven
>>
>>
>>
>

Mime
View raw message