hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miles Osborne <mi...@inf.ed.ac.uk>
Subject Re: Checking if a streaming job failed
Date Fri, 03 Apr 2009 01:05:19 GMT
here is how i do it (in perl).  hadoop streaming is actually called by
a shell script, which in this case expects compressed input and
produces compressed output.  but you get the idea:

(the mailer had messed-up the formatting somewhat)
sub runStreamingCompInCompOut {
    my $mapper = shift @_;
    my $reducer = shift @_;
    my $inDir = shift @_;
    my $outDir = shift @_;
    my $numMappers = shift @_;
    my $numReducers = shift @_;
    my $jobName = $runName . ":" . shift @_;
    my $cmd = "sh runStreamingCompInCompOut.sh $mapper $reducer $inDir
$outDir $jobName $numMappers \    $numReducers &> /tmp/.trace";
print STDERR "Running: $cmd\n";
    system $cmd;
    open IN, "/tmp/.trace" or die "can't open streaming trace";
        my $line = <IN>;
        (my $date,my $time,my $status) = split(/\s+/,$line);
        if ($status eq "ERROR") {
            print STDERR "command: $cmd failed\n";

2009/4/3 Mayuran Yogarajah <mayuran.yogarajah@casalemedia.com>:
> Hello, does anyone know how I can check if a streaming job (in Perl) has
> failed or succeeded? The only way I can see at the moment is to check
> the web interface for that jobID and parse out the '*Status:*' value.
> Is it not possible to do this using 'hadoop job -status' ? I see there is a
> count
> for failed map/reduce tasks, but map/reduce tasks failing is normal (or so
> I thought).  I am under the impression that if a task fails it will simply
> be
> reassigned to a different node.  Is this not the case?  If this is normal
> then I
> can't reliably use this count to check if the job as a whole failed or
> succeeded.
> Any feedback is greatly appreciated.
> thanks,
> M

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

View raw message