hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From PaweĊ‚ Michna <pmic...@gmail.com>
Subject Neverending SQL query execution: jdbc + hive
Date Mon, 11 Aug 2014 19:16:54 GMT
Hi,

I'm executing some queries using HiveServer2 and jdbc. Once in a while my
application gets stuck with an output like this (on random query,
nondeterministic):

07:24:54.910 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.910 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.911 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.912 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.912 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.912 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.912 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
> 07:24:54.912 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> CLIENT: reading data length: 53
> 07:24:54.912 [pool-4-thread-1] DEBUG o.a.thrift.transport.TSaslTransport -
> writing data length: 100
>

and so on, until I kill it manually. My code:

private void runSingleStreamOfQueries(String hiveServerHost, int
> hiveServerPort, List<String> queryNames,
>             BenchmarkResultDTO singleStreamOfQueriesExecutionTimeDTO,
> BenchmarkResultDTO singleStreamOfQueriesErrorCountDTO,
>             List<BenchmarkResultDTO> queriesExecutionTimesDTOs, String
> databaseName) throws IOException, SQLException {
>     Connection con = null;
>     Statement stmt = null;
>     String username = "hive";
>     try {
>         con = DriverManager.getConnection("jdbc:hive2://" + hiveServerHost
> + ":" + hiveServerPort + "/" + databaseName,
>                 username, "");
>         stmt = con.createStatement();
>         int numberOfErrors = 0;
>         log.info("Starting single stream of queries...");
>         log.info("Number of queries = " + (queryNames.size()));
>         StopWatch stopwatchStream = new StopWatch();
>         StopWatch stopwatchQuery = new StopWatch();
>         stopwatchStream.start();
>         singleStreamOfQueriesExecutionTimeDTO.setTimestampStart(new
> DateTime());
>         int executedQueries = 0;
>         for (String queryName : queryNames) {
>             log.info("Executing " + queryName + "...");
>             String queryFilePath = BENCHMARK_QUERIES_DIR + File.separator
> + queryName;
>             String query = fileToString(queryFilePath);
>             log.info(query);
>             try {
>                 stopwatchQuery.start();
>                 stmt.executeQuery(query);
>                 stopwatchQuery.stop();
>                 int queryIndex = findQueryIndex(queryName,
> queriesExecutionTimesDTOs);
>                 queriesExecutionTimesDTOs.get(queryIndex).setValue(new
> BigDecimal(stopwatchQuery.getTime()/1000));
>             } catch (SQLException e) {
>                 numberOfErrors++;
>                 log.warn("SQLException on " + queryName, e);
>             } finally {
>                 stopwatchQuery.reset();
>             }
>             log.info(queryName + " executed.");
>             executedQueries++;
>             log.info(queryNames.size()-executedQueries + " queries left");
>         }
>         stopwatchStream.stop();
>         singleStreamOfQueriesExecutionTimeDTO.setValue(new
> BigDecimal(stopwatchStream.getTime()/1000));
>         singleStreamOfQueriesExecutionTimeDTO.setTimestampEnd(new
> DateTime());
>         singleStreamOfQueriesErrorCountDTO.setValue(new
> BigDecimal(numberOfErrors));
>     } finally {
>         if(null != stmt) stmt.close();
>         if(null != con) con.close();
>     }
>     log.info("Single stream of queries finished.");
> }
>

Usually for the first time it works. When I execute the method once again,
the problem arises at some nondeterministic point. When I kill the process
and execute the method again, it doesn' work from the beginning and only
restarting HiveServer2 helps.

What is the problem? Do I have memory leaks?

Regards,
Pawel

Mime
View raw message