hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vivek Mishra <vivek.mis...@impetus.co.in>
Subject RE: MapJoin error: .hashtable file not found
Date Wed, 28 Nov 2012 07:26:57 GMT
Thanks. This is what i understood.
I am using
 set auto.convert.join=true.

As i want hive to determine smaller table to fit in memory. I did look into MapJoinOperator
and MapJoinProcessor.java , but could not find any pointer to my problem.

-Vivek
________________________________________
From: Navis류승우 [navis.ryu@nexr.com]
Sent: 28 November 2012 12:44
To: user@hive.apache.org
Subject: Re: MapJoin error: .hashtable file not found

hi,

The hashtable file you mentioned is pre-staged file for MapJoin.

If you remove MAPJOIN hint and set auto.convert.join=false, it will be disabled

2012/11/28 Vivek Mishra <vivek.mishra@impetus.co.in<mailto:vivek.mishra@impetus.co.in>>
Hi,

I am using 0.9.0 hive with Windows 2008 server.

I did try debugging into code, but no luck to understand:

"
Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason
they are not created though gz files are there?
"
-Vivek
________________________________________
From: Mark Grover [grover.markgrover@gmail.com<mailto:grover.markgrover@gmail.com>]
Sent: 28 November 2012 11:41
To: user@hive.apache.org<mailto:user@hive.apache.org>
Subject: Re: MapJoin error: .hashtable file not found

Vivek,
What version of Hive are you using? And, on what OS?

Mark

On Tue, Nov 27, 2012 at 9:34 PM, Vivek Mishra <vivek.mishra@impetus.co.in<mailto:vivek.mishra@impetus.co.in><mailto:vivek.mishra@impetus.co.in<mailto:vivek.mishra@impetus.co.in>>>
wrote:
Any pointers? Any help?

Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason
they are not created though gz files are there

-Vivek
________________________________________
From: Vivek Mishra
Sent: 24 November 2012 11:52
To: user@hive.apache.org<mailto:user@hive.apache.org><mailto:user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: MapJoin error: .hashtable file not found

Hi,

I am trying to run a MapJoin query and somehow getting below error. I can see that file is
not there in specified directory. Did some debugging but no luck as well.

Here is the error:

nOperator:        Load back 1 hashtable file from tmp file uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable
2012-11-24 11:13:45,389 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed
Cache Error
2012-11-24 11:13:45,390 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException:
c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable
(The system cannot find the file specified)
     at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
       at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
     at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321)
  at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325)
       at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325)
    at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325)
 at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495)
 at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
       at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
       at org.apache.hadoop.mapred.Child.main(Child.java:265)


Also, i have verified there is no such OOM issue as well.


Any idea?


-Vivek

________________________________

Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud
and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2.

Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

________________________________

Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud
and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2.

Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

________________________________

Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud
and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2.

Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

________________________________

Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud
and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2.

Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.
Mime
View raw message