hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hadoopman <hadoop...@gmail.com>
Subject Re: Setting input paths
Date Wed, 06 Apr 2011 16:23:02 GMT
I have a process which  is loading data into hive hourly.  Loading data 
hourly isn't a problem however when I load historical data say 24-48 
hours I receive the below error msg.  In googling I've come across some 
suggestions that jvm memory needs to be increased.  Are there any other 
options or is that pretty much it?

I appreciate any help with this one.  To get around the problem I've 
loaded fewer historical logs which works great but isn't what I had in 
mind :-)

Thanks a bunch!

---

2011-04-05 15:33:38,400 INFO 
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 finished. closing...
2011-04-05 15:33:38,400 INFO 
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 forwarded 0 rows
2011-04-05 15:33:38,400 INFO 
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 finished. closing...
2011-04-05 15:33:38,400 INFO 
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 forwarded 0 rows
2011-04-05 15:33:38,401 WARN 
org.apache.hadoop.hive.ql.exec.GroupByOperator: Begin Hash Table flush 
at close: size = 0
2011-04-05 15:33:38,401 INFO 
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 finished. closing...
2011-04-05 15:33:38,401 INFO 
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 forwarded 0 rows
2011-04-05 15:33:38,401 INFO 
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS 
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,401 INFO 
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS
hdfs://namenoden1:9000/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/_tmp.000049_0
2011-04-05 15:33:38,401 INFO 
org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS 
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,448 INFO org.apache.hadoop.io.compress.CodecPool: 
Got brand-new compressor
2011-04-05 15:33:38,495 INFO 
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 Close done
2011-04-05 15:33:38,495 INFO 
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 Close done
2011-04-05 15:33:38,495 INFO 
org.apache.hadoop.hive.ql.exec.FilterOperator: 6 Close done
2011-04-05 15:33:38,495 INFO 
org.apache.hadoop.hive.ql.exec.SelectOperator: 1 Close done
2011-04-05 15:33:38,495 INFO 
org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done
2011-04-05 15:33:38,495 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 
13 Close done
2011-04-05 15:33:38,495 INFO ExecMapper: ExecMapper: processed 188614 
rows: used memory = 720968432
2011-04-05 15:33:38,501 FATAL org.apache.hadoop.mapred.Child: Error 
running child : java.lang.OutOfMemoryError: Java
heap space
                 at org.apache.hadoop.io.Text.setCapacity(Text.java:240)
                 at org.apache.hadoop.io.Text.append(Text.java:216)
                 at 
org.apache.hadoop.util.LineReader.readLine(LineReader.java:159)
                 at 
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
                 at 
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
                 at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
                 at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
                 at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
                 at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:202)
                 at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:186)
                 at 
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
                 at 
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:383)
                 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317)
                 at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
                 at java.security.AccessController.doPrivileged(Native 
Method)
                 at javax.security.auth.Subject.doAs(Subject.java:396)
                 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
                 at org.apache.hadoop.mapred.Child.main(Child.java:211)



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message