hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatesh <vramanatha...@aol.com>
Subject Re: HBase map reduce job timing
Date Wed, 06 Oct 2010 03:05:55 GMT

 

 Sorry..yeah..i've to do some digging to provide some data..
What sort of data would be helpful? Would stats reported by jobtracker.jsp suffice? I've pasted
that in this email..
I can gather more jvm stats..thanks

Status: Succeeded
Started at: Tue Oct 05 21:39:58 EDT 2010
Finished at: Tue Oct 05 22:36:43 EDT 2010
Finished in: 56mins, 45sec
Job Cleanup: Successful



Kind
% Complete
Num Tasks
Pending
Running
Complete
Killed
Failed/Killed
Task Attempts

map
100.00%





565
0
0
565
0
0 / 11

reduce
100.00%





20
0
0
20
0
0 / 2

            

      


      
Counter
      
Map
      
Reduce
      
Total
    
            
                      
Job Counters 
                      
Launched reduce tasks
          
0
          
0
          
22
        
                
                    
Rack-local map tasks
          
0
          
0
          
66
        
                
                    
Launched map tasks
          
0
          
0
          
576
        
                
                    
Data-local map tasks
          
0
          
0
          
510
        
                
                      
com.JobRecords
                      
REDUCE_PHASE_RECORDS
          
0
          
597,712
          
597,712
        
                
                    
MAP_PHASE_RECORDS
          
2,534,807
          
0
          
2,534,807
        
                
                      
FileSystemCounters
                      
FILE_BYTES_READ
          
335,845,726
          
861,146,518
          
1,196,992,244
        
                
                    
FILE_BYTES_WRITTEN
          
1,197,031,156
          
861,146,518
          
2,058,177,674
        
                
                      
Map-Reduce Framework
                      
Reduce input groups
          
0
          
597,712
          
597,712
        
                
                    
Combine output records
          
0
          
0
          
0
        
                
                    
Map input records
          
2,534,807
          
0
          
2,534,807
        
                
                    
Reduce shuffle bytes
          
0
          
789,145,342
          
789,145,342
        
                
                    
Reduce output records
          
0
          
0
          
0
        
                
                    
Spilled Records
          
3,522,428
          
2,534,807
          
6,057,235
        
                
                    
Map output bytes
          
851,007,170
          
0
          
851,007,170
        
                
                    
Map output records
          
2,534,807
          
0
          
2,534,807
        
                
                    
Combine input records
          
0
          
0
          
0
        
                
                    
Reduce input records
          
0
          
2,534,807
          
2,534,807




 

 

-----Original Message-----
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Sent: Tue, Oct 5, 2010 10:53 pm
Subject: Re: HBase map reduce job timing


I'd love to give you tips, but you didn't provide any data about the
input and output of your job, the kind of hardware you're using, etc.
At this point any suggestion would be a stab in the dark, the best I
can do is pointing to the existing documentation
http://wiki.apache.org/hadoop/PerformanceTuning

J-D

On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh <vramanathan00@aol.com> wrote:
>
>
>
>  I've a mapreduce job that is taking too long..over an hour..Trying to see 
what can a tune
> to to bring it down..One thing I noticed, the job is kicking off
> - 500+ map tasks : 490 of them do not process any records..where as 10 of them 
process all the records
>  (200 K each..)..Any idea why that would be?...
>
> ..map phase takes about couple of minutes..
> ..reduce phase takes the rest..
>
> ..i'll try increasing # of reduce tasks..Open to other other suggestion for 
tunables..
>
> thanks for your input
> venkatesh
>
>
>

 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message