tajo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinho Kim <jh...@apache.org>
Subject Re: Worker Resources
Date Wed, 17 Sep 2014 14:46:03 GMT
Hello Chris,

Looks like killed by kernel.
please update latest revision and increase the heap in tajo-env.sh.
export TAJO_WORKER_HEAPSIZE= parallel processing * 2GB or more
export TAJO_WORKER_OPTS="-XX:+UseParallelOldGC"

Worker resource is the following:
parallel  worker(512 * 3) + query master (512)

Thanks

-Jinho
Best regards


2014-09-17 21:40 GMT+09:00 Christian Schwabe <Christian.Schwabe@gmx.com>:
> Hello guys,
>
> The log has been strangely cut off, I noticed straight. Attached again the
> correct and complete log.
>
> Best regards,
>
> Chris
>
>
> Am 17.09.2014 um 13:45 schrieb Christian Schwabe
> <Christian.Schwabe@gmx.com>:
>
> Hello guys,
>
> I promised I do not pretend to solve this problem together with you. I would
> like to redeem and more questions that bring us closer to a solution,
> hopefully this promise. My Macbook Pro has 8 GB of RAM and 4 cores
> available. My configuration in tajo-site.xml looks like this:
>
> <configuration>
> <property>
>   <name>tajo.resource-tracker.heartbeat.timeout-secs</name>
>   <value>240000</value>  // or, your own longer value. default is 120*1000
> (2 minutes)
> </property>
> <property>
>   <name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
>   <value>4</value>
> </property>
>  <property>
>   <name>tajo.worker.resource.memory-mb</name>
>   <value>2048</value>
> </property>
> </configuration>
>
>
> Actually, I would now expect a fourfold parallel processing. However, I can
> only see a maximum of three tasks at once with the status"running". With
> this configuration, I can still establish the following: Tajo processed,
> judging by the progress indicator is now more data than before. Up to now,
> only about 75% possible I land now at about 90% until Tajo crashed again.
> I attached a log for this run. Also the following session variables were
> applied for this run:
> tajo> \set JOIN_TASK_INPUT_SIZE 64
> tajo> \set JOIN_PER_SHUFFLE_SIZE 64
>
> Do you have any ideas I can try?
>
>
>
> Best regards and very thankful,
> Chris
>
>
>
> <tajo-chris-master-christians-mbp.fritz.box.log><tajo-chris-worker-christians-mbp.fritz.box.log>
>
>
>

Mime
View raw message