hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Subramanian <sanjaysubraman...@yahoo.com>
Subject Re: Weird Error on Inserting in Table [ORC, MESOS, HIVE]
Date Tue, 07 Oct 2014 19:04:46 GMT
hi 
I faced a similar situation in my dev cluster CDH distribution 5.1.3
See the thread details with log files  https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/scm-users/MpcpHj5mWT8


thanks
sanjay      From: John Omernik <john@omernik.com>
 To: user@hive.apache.org 
 Sent: Tuesday, September 9, 2014 12:10 PM
 Subject: Re: Weird Error on Inserting in Table [ORC, MESOS, HIVE]
   
Well, here is me talking to myself: but in case someone else runs across this, I changed the
hive metastore connect timeout to 600 seconds (per the JIRA below for Hive 0.14) and now my
problem has gone away. It looks like the timeout was causing some craziness. 
https://issues.apache.org/jira/browse/HIVE-7140





On Tue, Sep 9, 2014 at 1:00 PM, John Omernik <john@omernik.com> wrote:

I ran with debug logging, and this is interesting, there was a loss of connection to the metastore
client RIGHT before the partition mention above... as data was looking to be moved around...
I wonder if the timing on that is bad?
14/09/09 12:47:37 [main]: INFO exec.MoveTask: Partition is: {day=null, source=null}14/09/09
12:47:38 [main]: INFO metadata.Hive: Renaming src:maprfs:/user/hive/scratch/hive-mapr/hive_2014-09-09_12-38-30_860_3555291990145206535-1/-ext-10000/day=2012-11-30/source=20121119_SWAirlines_Spam/000004_0;dest:
maprfs:/user/hive/warehouse/intel_flow.db/pcaps/day=2012-11-30/source=20121119_SWAirlines_Spam/000004_0;Status:true14/09/09
12:48:02 [main]: WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection.
Attempting to reconnect.org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException:
Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)


On Tue, Sep 9, 2014 at 11:02 AM, John Omernik <john@omernik.com> wrote:

I am doing a dynamic partition load in Hive 0.13 using ORC files. This has always worked in
the past both with MapReduce V1 and YARN. I am working with Mesos now, and trying to trouble
shoot this weird error:


Failed with exception AlreadyExistsException(message:Partition already exists

What's odd is is my insert is an insert (without Overwrite) so it's like two different reducers
have data to go into the same partition, but then there is a collision of some sort? Perhaps
there is a situation where the partition doesn't exist prior to the run, but when two reducers
have data, they both think they should be the one to create the partition? Shouldn't if a
partition already exists, the reducer just copies it's file into the partition?  I am struggling
to see why this would be an issue with Mesos, but not on Yarn, or MRv1.
Any thoughts would be welcome. 
John





  
Mime
View raw message