pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Dai <da...@hortonworks.com>
Subject Re: add column number using pig UDF
Date Mon, 27 Apr 2015 17:21:43 GMT
This is the known issue with rank implementation with counter limit. We
fixed that in tez but not MR. You can either increase the limit in MR, or
try Pig on Tez.

Thanks,
Daniel

On 4/27/15, 3:07 AM, "李运田" <cumtshu@163.com> wrote:

>I use rank to add ID now ,but I always get error "
> FATAL [AsyncDispatcher event handler]
>org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
>org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
>counters: 121 max=120
>	at 
>org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:103)
>	at 
>org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:110)
>	at 
>org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(Abstr
>actCounterGroup.java:78)
>	at 
>org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(A
>bstractCounterGroup.java:95)
>"
>I get many solution in website ,but it is not resolved . perhaps there is
>something wrong with some sets like " mapreduce.job.counters.max
> mapreduce.job.counters.group.name.max
> mapreduce.job.counters.counter.name.max
> mapreduce.job.counters.groups.max"
>can you give me some advice?
>
>
>
>
>
>
>
>
>At 2015-04-24 21:56:20, "Alex Nastetsky" <alex.nastetsky@vervemobile.com>
>wrote:
>>Have you looked at the RANK function?
>>https://pig.apache.org/docs/r0.11.0/basic.html#rank
>>
>>On Fri, Apr 24, 2015 at 5:15 AM, 李运田 <cumtshu@163.com> wrote:
>>
>>> I have a big data about 10T ,I want to add column number to every
>>>column
>>> from 1 to COUNT(my data),,I  use two functions in
>>> 
>>>http://stackoverflow.com/questions/9288578/how-can-i-add-row-numbers-for
>>>-rows-in-pig-or-hive
>>> and http://www.aiuxian.com/article/p-139530.html
>>> but I always get many files. every file column number is  from 0 to
>>> ......  perhaps there are many reduces. I dont know how to add  column
>>> number   from 1 to COUNT(my data)?
>>> thank you ,give me some advice or some one can give me some advice
>>> websites?
>>> thank you very much.
>>>
>>>

Mime
View raw message