hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madhav Sharan <msha...@usc.edu>
Subject Re: All nodes are not used
Date Tue, 09 Aug 2016 20:22:18 GMT
Hi Sunil - Thanks a lot for replying

For one job run yes some nodes don't take load at all. But if I rerun no
these are not same nodes always.

One map job takes ~3 seconds to run and till now I am not able to run my
whole job on a bigger data set so I can't say that container are short
lived.

I was doing experiments and if I split input file into N files where N =
number of cores then my job starts running on all cores. So may be I need
to look at split size. Any trick to set split size = number of cores?

I can try adjusting mapred.min.split.size manually otherwise.


--
Madhav Sharan


On Tue, Aug 9, 2016 at 8:27 AM, Sunil Govind <sunil.govind@gmail.com> wrote:

> HI Madhav
>
> Could you help to share some more information here. When u say few nodes
> are not utilized, is it always same nodes which are not utilized?
>
> also how long each of these container are running on an average, pls make
> sure you have provided enough split size to ensure the containers are not
> short running.
>
> Thanks
> Sunil
>
> On Tue, Aug 9, 2016 at 4:49 AM Madhav Sharan <msharan@usc.edu> wrote:
>
>> Hi Hadoop users,
>>
>> I am running a m/r job with an input file of 23 million records. I can
>> see all our files are not getting used.
>>
>> What can I change to utilize all nodes?
>>
>>
>> Containers Mem Used Mem Avail Vcores used Vcores avail
>> 8 11.25 GB 0 B 8 0
>> 0 0 B 11.25 GB 0 8
>> 0 0 B 11.25 GB 0 8
>> 8 11.25 GB 0 B 8 0
>> 8 11.25 GB 0 B 8 0
>> 7 11.25 GB 0 B 7 1
>> 5 7.03 GB 4.22 GB 5 3
>> 0 0 B 11.25 GB 0 8
>> 0 0 B 11.25 GB 0 8
>>
>>
>> My command looks like -
>>
>> hadoop jar target/pooled-time-series-1.0-SNAPSHOT-jar-with-dependencies.jar
>> gov.nasa.jpl.memex.pooledtimeseries.MeanChiSquareDistanceCalculation
>> /user/pts/output/MeanChiSquareAndSimilarityInput
>> /user/pts/output/MeanChiSquaredCalcOutput
>>
>> Directory - */user/pts/output/MeanChiSquareAndSimilarityInput* have a
>> input file of 23 m records. File size is ~3 GB
>>
>> Code - https://github.com/smadha/pooled_time_series/blob/master/src
>> /main/java/gov/nasa/jpl/memex/pooledtimeseries/MeanChiSquare
>> DistanceCalculation.java#L135
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_smadha_pooled-5Ftime-5Fseries_blob_master_src_main_java_gov_nasa_jpl_memex_pooledtimeseries_MeanChiSquareDistanceCalculation.java-23L135&d=DQMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=DhBa2eLkbd4gAFB01lkNgg&m=lW_zVvBUpXydqujJG2o4HChrLD-0A-mjoMCXOaKh2eI&s=TdXRcXKJ9MowW1sS7KlhX14-45SNMj0O6gVqBqoEjwg&e=>
>>
>>
>> --
>> Madhav Sharan
>>
>>

Mime
View raw message