accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Hulbert <ahulb...@ccri.com>
Subject Re: Accumulo 1.7.1 on Docker
Date Tue, 22 Mar 2016 19:02:09 GMT
With the single-node/docker I had to add a "sleep 3" after "hdfs 
dfsadmin -safemode" returns too though before the "accumulo init"

I think having an Apache-owned docker for accumulo would rock! We have 
made a few but having a canonical one would be great.

On 03/22/2016 02:47 PM, Michael Wall wrote:
> Sven, did you ever put up a Dockerfile?  I've done something like this 
> waiting for HDFS to come out of safe mode.
>
> echo "Starting Hadoop"
> ${HADOOP_HOME}/sbin/start-dfs.sh
> echo "Waiting for hadoop to come out of safeMode"
> local x=0
> while [ $x -lt 1 ] do # try 5 times, for slow computers
> ${HADOOP_HOME}/bin/hdfs dfsadmin -safemode && x=5
> x=$(( $x + 1 ))
> done
>
> On Tue, Mar 22, 2016 at 2:41 PM, Josh Elser <josh.elser@gmail.com 
> <mailto:josh.elser@gmail.com>> wrote:
>
>     Every time I use Docker, I seem to run into odd situations like
>     this. Very frustrating.
>
>     I would love to know what's happening and get this working
>     reliably. Having an "Apache-owned" docker image for Accumulo would
>     be nice (and would help us internally as well as externally, IMO).
>
>
>     Andrew Hulbert wrote:
>
>         +1 for Josh's suggestion.
>
>         Not sure if its the same problem, but I too had to add some
>         sleeps for
>         single node accumulo startup scripts directly before and after
>         init'ing
>         accumulo. I dug in once and it seemed that the datanode needed
>         more time
>         to let other processes know the files in HDFS existed at
>         /accumulo.
>         Perhaps it was awaiting the exit from safe mode? Three seconds
>         did it
>         for me:
>         https://github.com/jahhulbert-ccri/cloud-local/blob/master/bin/cloud-local.sh#L109
>
>
>
>         On 03/14/2016 01:49 PM, Josh Elser wrote:
>
>             Well, I'm not sure wtf is going on, but I added some
>             sleeps to your
>             init.sh script (30seconds before starting ZooKeeper and before
>             init'ing Accumulo) and now things worked just fine.
>
>             I'm not seeing any Accumulo problems here, but rather the
>             same Docker
>             absurdities I run into every time I try to use it :). I
>             also just
>             upgrade to Docker 1.10.3 if that helps.
>
>             Josh Elser wrote:
>
>                 So, I got it running, and ran into the same problem
>                 you described.
>
>                 The odd thing is that both the Accumulo Master and
>                 TabletServer seemed
>                 to be... stuck. They both claimed to be running, but
>                 they were both hung
>                 (unable to communicate). This ultimately caused the
>                 TabletServer to lose
>                 its ZooKeeper lock (and why the Shell didn't think it
>                 was running).
>
>                 I'm running into all sorts of docker issues ATM
>                 (trying to upgrade now,
>                 hoping it will help), but I'd recommend trying to
>                 increase the amount of
>                 heap for both the master and tserver from 128M to 256M in
>                 /accumulo-1.7.1/conf/accumulo-env.sh (the -Xmx value).
>
>                 Josh Elser wrote:
>
>                     Ok! Thanks, I'll look for that init.sh script.
>
>                     We're not too... docker-savvy here yet :), but we
>                     can definitely help
>                     out if we can find an automation script. Let me
>                     get back to you.
>
>                     Sven Hodapp wrote:
>
>                         Hi Josh,
>
>                         thanks for your answer.
>                         Currently I haven't uploaded the Dockerfile...
>                         If you want I'll upload
>                         it!
>                         (But currently it only is an debian with ssh,
>                         rsync, jdk7, the
>                         unzipped distributions of hadoop, zookeeper
>                         and accumulo, and the set
>                         of environment variables)
>
>                         Currently, and for testing, I start all things
>                         manually.
>                         The steps I do is documented in /init.sh
>                         within the docker container.
>                         I've enshured that dfs and zookeeper are up
>                         before starting accumulo.
>
>                         Regards,
>                         Sven
>
>                         ----- Ursprüngliche Mail -----
>
>                             Von: "Josh Elser"<josh.elser@gmail.com
>                             <mailto:josh.elser@gmail.com>>
>                             An: "user"<user@accumulo.apache.org
>                             <mailto:user@accumulo.apache.org>>
>                             Gesendet: Montag, 14. März 2016 15:41:16
>                             Betreff: Re: Accumulo 1.7.1 on Docker
>
>
>                             Hi Sven,
>
>                             I can't seem to find the source for your
>                             Docker image (nor am smart
>                             enough to figure out how to extract it).
>                             Any chance you can point
>                             us to
>                             that?
>
>                             That said, I did run your image, and it
>                             looks like you didn't
>                             actually
>                             start Hadoop, ZooKeeper, or Accumulo.
>
>                             Our documentation treats Hadoop and
>                             ZooKeeper primarily as
>                             prerequisites, so check their docs for
>                             good instructions.
>
>                             For Accumulo -
>
>                             Configuration:
>                             http://accumulo.apache.org/1.7/accumulo_user_manual.html#_installation
>
>                             Initialization:
>                             http://accumulo.apache.org/1.7/accumulo_user_manual.html#_initialization
>
>
>                             Starting:
>                             http://accumulo.apache.org/1.7/accumulo_user_manual.html#_starting_accumulo
>
>
>
>
>                             Sven Hodapp wrote:
>
>                                 Dear reader,
>
>                                 I'd like to create a docker image with
>                                 accumulo 1.7.1.
>                                 I've install it from scratch with
>                                 hadoop-2.7.2, zookeeper-3.4.8 and
>                                 accumulo-1.7.1.
>                                 I'm going though the installation and
>                                 every time I'll end up like
>                                 that:
>
>                                 root@deebd8e29683:/# accumulo shell -u
>                                 root
>                                 Password: ******
>                                 2016-03-13 13:18:31,627
>                                 [trace.DistributedTrace] INFO :
>                                 SpanReceiver
>                                 org.apache.accumulo.tracer.ZooTraceClient
>                                 was loaded successfully.
>                                 2016-03-13 13:20:31,955
>                                 [impl.ServerClient] WARN : There are no
>                                 tablet servers:
>                                 check that zookeeper and accumulo are
>                                 running.
>
>                                 Anybody got an idea whats wrong? I
>                                 have no idea anymore...
>
>                                 I'll currently share this docker image
>                                 on docker hub.
>                                 If you want to try it you can simply
>                                 start:
>
>                                 docker run -it themerius/accumulo
>                                 /bin/bash
>
>                                 All binaries and configs are in there.
>                                 In ./init.sh are my setup
>                                 steps.
>
>                                 Regards,
>                                 Sven
>
>
>


Mime
View raw message