mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomas Barton <barton.to...@gmail.com>
Subject Re: mesos build errors
Date Thu, 24 Jul 2014 08:37:31 GMT
Depends on the nature of your tasks. Your code is written in MPI? You tasks
needs to communicate with others? One task will operate on all files, some
subset, or just on file? You might have:
     - one task per machine running on as many cores as possible
     - many smaller tasks starting in a dynamic manner depending on the data

What is the biggest bottleneck you have? disk read/write, network, CPU,
memory?

Writing own framework is possible, if you can take advantage of some
problem specific property.


On 24 July 2014 07:34, Itamar Ostricher <itamar@yowza3d.com> wrote:

> many: we have a processing pipeline with ~10 stages (one C++ program per
> stage usually), batch processing (almost-)all pairs of files in the
> dataset. the dataset contains >10K files at the moment, so a couple of
> hundreds of millions of program executions would be my definition for
> "many" in this case :-)
>
> I'll start with few machines with deploy scripts and a small subset of the
> dataset just to get the hang of it.
> It's a bit difficult to comprehend the stack, with all the possible
> options and combinations, though.
> If I have a main Python script that generates all the processing pipeline
> commands (that can be simply executed via shell), should I use a specific
> framework (like Hydra)? Or maybe use raw mesos? Or maybe I should write my
> own framework?
>
>
> On Wed, Jul 23, 2014 at 2:25 PM, Tomas Barton <barton.tomas@gmail.com>
> wrote:
>
>> Define many :) If you want to use some provisioning tools like Puppet,
>> Chef, Ansible... there are quite a few modules to do this job:
>>
>> http://mesosphere.io/learn/#tools
>>
>> If you have only a few machines, you might be fine with deploy scripts.
>>
>> An example of MPI framework is here:
>>
>> https://github.com/mesosphere/mesos-hydra
>>
>>
>>
>>
>> On 23 July 2014 12:26, Itamar Ostricher <itamar@yowza3d.com> wrote:
>>
>>> Thanks Tomas.
>>>
>>> ldconfig didn't change anything. make still failed.
>>>
>>>  But the Debian packaged installed like a charm, so I'm good :-)
>>> Now I just need to figure out how to use it...
>>> (going to start with [1], unless anyone chimes in with a better
>>> recommended starting point for a mesos-newbie who is trying to set up a
>>> cluster of GCE instances in order to distribute execution of *many* C++
>>> programs working on a large dataset that is currently stored in Google
>>> Cloud Storage.)
>>>
>>> [1] http://mesos.apache.org/documentation/latest/deploy-scripts/
>>>
>>>
>>> On Wed, Jul 23, 2014 at 11:55 AM, Tomas Barton <barton.tomas@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> that's quite strange. Try to run
>>>>
>>>> ldconfig
>>>>
>>>> and then again make.
>>>>
>>>> You can find binary packages for Debian here:
>>>> http://mesosphere.io/downloads/
>>>>
>>>> Tomas
>>>>
>>>>
>>>> On 23 July 2014 10:09, Itamar Ostricher <itamar@yowza3d.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to do a clean build of mesos for the 0.19.0 tarball.
>>>>> I was following the instructions from
>>>>> http://mesos.apache.org/gettingstarted/ step by step. Got to running
>>>>> `make`, which ran for quite a while, and exited with errors (see the
end of
>>>>> the output below).
>>>>>
>>>>> Extra env info: I'm trying to do this build on a 64-bit Debian GCE
>>>>> instance:
>>>>> itamar@mesos-test-1:/tmp/mesos-0.19.0/build$ uname -a
>>>>> Linux mesos-test-1 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64
>>>>> GNU/Linux
>>>>>
>>>>> Assistance will be much appreciated!
>>>>> Alternatively, I don't mind using precompiled binaries, if anyone can
>>>>> point me in the direction of such binaries for the GCE environment I
>>>>> described :-)
>>>>>
>>>>> tail of make output:
>>>>> ----------------------------
>>>>>
>>>>> libtool: link: warning:
>>>>> `/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/libgflags.la'
>>>>> seems to be moved
>>>>> *** Warning: Linking the shared library libmesos.la against the
>>>>> *** static library ../3rdparty/leveldb/libleveldb.a is not portable!
>>>>> libtool: link: warning:
>>>>> `/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/libgflags.la'
>>>>> seems to be moved
>>>>> libtool: link: g++  -fPIC -DPIC -shared -nostdlib
>>>>> /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o
>>>>> /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o  -Wl,--whole-archive
>>>>> ./.libs/libmesos_no_3rdparty.a ../3rdparty/libprocess/.libs/libprocess.a
>>>>> ./.libs/libjava.a -Wl,--no-whole-archive
>>>>>  ../3rdparty/libprocess/3rdparty/protobuf-2.5.0/src/.libs/libprotobuf.a
>>>>> ../3rdparty/libprocess/3rdparty/glog-0.3.3/.libs/libglog.a
>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib
>>>>> ../3rdparty/leveldb/libleveldb.a
>>>>> ../3rdparty/zookeeper-3.4.5/src/c/.libs/libzookeeper_mt.a
>>>>> /tmp/mesos-0.19.0/build/3rdparty/libprocess/3rdparty/glog-0.3.3/.libs/libglog.a
>>>>> /usr/lib/libgflags.so -lpthread
>>>>> /tmp/mesos-0.19.0/build/3rdparty/libprocess/3rdparty/libev-4.15/.libs/libev.a
>>>>> -lsasl2 /usr/lib/x86_64-linux-gnu/libcurl-nss.so -lz -lrt
>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.7
>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu
>>>>> -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu
>>>>> -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. -lstdc++
-lm
>>>>> -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o
>>>>> /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o
>>>>>  -pthread -Wl,-soname -Wl,libmesos-0.19.0.so -o .libs/
>>>>> libmesos-0.19.0.so
>>>>> libtool: link: (cd ".libs" && rm -f "libmesos.so" &&
ln -s "
>>>>> libmesos-0.19.0.so" "libmesos.so")
>>>>> libtool: link: ( cd ".libs" && rm -f "libmesos.la" &&
ln -s "../
>>>>> libmesos.la" "libmesos.la" )
>>>>> g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\"
>>>>> -DPACKAGE_VERSION=\"0.19.0\" -DPACKAGE_STRING=\"mesos\ 0.19.0\"
>>>>> -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\"
>>>>> -DVERSION=\"0.19.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1
>>>>> -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1
>>>>> -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1
>>>>> -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_PTHREAD=1 -DMESOS_HAS_JAVA=1
>>>>> -DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -DHAVE_LIBZ=1 -DHAVE_LIBCURL=1
>>>>> -DHAVE_LIBSASL2=1 -I. -I../../src   -Wall -Werror
>>>>> -DLIBDIR=\"/usr/local/lib\" -DPKGLIBEXECDIR=\"/usr/local/libexec/mesos\"
>>>>> -DPKGDATADIR=\"/usr/local/share/mesos\" -I../../include
>>>>> -I../../3rdparty/libprocess/include
>>>>> -I../../3rdparty/libprocess/3rdparty/stout/include -I../include
>>>>> -I../3rdparty/libprocess/3rdparty/boost-1.53.0
>>>>> -I../3rdparty/libprocess/3rdparty/protobuf-2.5.0/src
>>>>> -I../3rdparty/libprocess/3rdparty/picojson-4f93734
>>>>> -I../3rdparty/libprocess/3rdparty/glog-0.3.3/src
>>>>> -I../3rdparty/leveldb/include -I../3rdparty/zookeeper-3.4.5/src/c/include
>>>>> -I../3rdparty/zookeeper-3.4.5/src/c/generated   -pthread -g -g2 -O2 -MT
>>>>> local/mesos_local-main.o -MD -MP -MF local/.deps/mesos_local-main.Tpo
-c -o
>>>>> local/mesos_local-main.o `test -f 'local/main.cpp' || echo
>>>>> '../../src/'`local/main.cpp
>>>>> mv -f local/.deps/mesos_local-main.Tpo local/.deps/mesos_local-main.Po
>>>>> /bin/bash ../libtool  --tag=CXX   --mode=link g++ -pthread -g -g2 -O2
>>>>>   -o mesos-local local/mesos_local-main.o libmesos.la -lsasl2 -lcurl
>>>>> -lz  -lrt
>>>>> libtool: link: g++ -pthread -g -g2 -O2 -o .libs/mesos-local
>>>>> local/mesos_local-main.o  ./.libs/libmesos.so /usr/lib/libgflags.so
>>>>> -lpthread -lsasl2 /usr/lib/x86_64-linux-gnu/libcurl-nss.so -lz -lrt -pthread
>>>>> ./.libs/libmesos.so: error: undefined reference to 'dlopen'
>>>>> ./.libs/libmesos.so: error: undefined reference to 'dlsym'
>>>>> ./.libs/libmesos.so: error: undefined reference to 'dlerror'
>>>>> collect2: error: ld returned 1 exit status
>>>>> make[2]: *** [mesos-local] Error 1
>>>>> make[2]: Leaving directory `/tmp/mesos-0.19.0/build/src'
>>>>> make[1]: *** [all] Error 2
>>>>> make[1]: Leaving directory `/tmp/mesos-0.19.0/build/src'
>>>>> make: *** [all-recursive] Error 1
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message