hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wouter de Bie <wou...@spotify.com>
Subject Re: Asynchronous query exection
Date Tue, 15 Nov 2011 19:52:02 GMT
Another way would be using Hive server. This will execute multiple queries in parallel.

--Wouter de Bie
Team Lead Analytics Infrastructure, Spotify
wouter@spotify.com (mailto:wouter@spotify.com)
+46 72 018 0777


On Tuesday, November 15, 2011 at 2:48 PM, Sam Wilson wrote:

> If you go this route, you may want to use nohup. This way your processes will continue
running even if you lose connection to your terminal session.
> 
> Other options:
> 
> 1) You can write your queries to a DB/Queue and have a process running on the Hive server
that reads from the DB/queue and runs them locally against Hive.
> 
> 2) You could use SSH and nohup to run your queries.
> 
> On Nov 15, 2011, at 2:15 PM, Mapred Learn wrote:
> > You could write your query to a file and do something like: 
> >  
> > hive -f <filename1> &
> > hive -f <filename2> &
> >  
> > etc. to invoke many instances in parallel.
> > 
> > On Tue, Nov 15, 2011 at 3:24 AM, Chinna Rao Lalam <chinnarao@huawei.com (mailto:chinnarao@huawei.com)>
wrote:
> > > Hi,
> > >  
> > >  Hive calls are blocking calls because once the query is executed it will return
the ResultSet  from that result set u will get the results.
> > >  
> > >  "hive.exec.parallel" property will helps to speed up the query execution if
the query generates more than one independent tasks. If it generates independent tasks if
this
> > > property is true it will execute the independent tasks parallely otherwise
it will execute sequentially.
> > >  
> > > Thanks&Regards,
> > > Chinna Rao Lalam 
> > >  
> > > From: Ghousia [ghousia.atheeq@gmail.com (mailto:ghousia.atheeq@gmail.com)]
> > > Sent: Tuesday, November 15, 2011 6:12 PM
> > > To: user@hive.apache.org (mailto:user@hive.apache.org)
> > > Subject: Asynchronous query exection
> > > 
> > > Hi, 
> > >  
> > > Hive queries take longer time to execute, and by default it is a blocking call.
Is there any way provided by Hive client to supports non blocking execution. 
> > >  
> > > Also, to execute jobs parallely, I tried setting the "hive.exec.parallel" to
true in hive-site.xml. But this did not work, Looking at the code, it looks like the same
flow is been followed both for serial and parallel execution. 
> > >  
> > > Any inputs would be of great help.
> > >  
> > > Thanks,
> > > Ghousia.
> > >  
> > > 
> > >   
> > >  
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> 


Mime
View raw message