hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1641) Incorrect counters in local mode
Date Mon, 27 Sep 2010 18:15:36 GMT

    [ https://issues.apache.org/jira/browse/PIG-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915408#action_12915408
] 

Ashutosh Chauhan commented on PIG-1641:
---------------------------------------

Tested manually for local mode. Messages were same as proposed above. +1 for the commit. One
minor suggestion is to put a line at the start saying something like: "Detected Local mode.
Stats reported below may be incomplete." This will reinforce the message to users that stats
reporting is not transparent across different modes (local Vs map-reduce).

> Incorrect counters in local mode
> --------------------------------
>
>                 Key: PIG-1641
>                 URL: https://issues.apache.org/jira/browse/PIG-1641
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.0
>            Reporter: Ashutosh Chauhan
>            Assignee: Richard Ding
>             Fix For: 0.8.0
>
>         Attachments: PIG-1641.patch
>
>
> User report, not verified.
> <email>
> HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt    Features
> 0.20.2    0.8.0-SNAPSHOT    user    2010-09-21 19:25:58    2010-09-21 21:58:42    ORDER_BY
> Success!
> Job Stats (time in seconds):
> JobId    Maps    Reduces    MaxMapTime    MinMapTIme    AvgMapTime    MaxReduceTime 
  MinReduceTime    AvgReduceTime    Alias    Feature    Outputs
> job_local_0001    0    0    0    0    0    0    0    0    raw    MAP_ONLY    
> job_local_0002    0    0    0    0    0    0    0    0    rank_sort    SAMPLER    
> job_local_0003    0    0    0    0    0    0    0    0    rank_sort    ORDER_BY    Processed/user_visits_table,
> Input(s):
> Successfully read 0 records from: "Data/Raw/UserVisits.dat"
> Output(s):
> Successfully stored 0 records in: "Processed/user_visits_table"
> However, when I look in the output:
> $ ls -lh Processed/user_visits_table/CG0/
> total 15250760
> -rwxrwxrwx  1 user  _lpoperator   7.3G Sep 21 21:58 part-0*
> It read a 20G input file and generated some output...
> </email>
> Is it that in local mode counters are not available? If so, instead of printing zeros
we should print "Information Unavailable" or some such.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message