hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajit Saha <rs...@lendingclub.com>
Subject Re: Running hive queries in different queue
Date Sun, 28 Feb 2016 10:26:01 GMT
Thanks a lot Sathi.


I also found in the Hive Execution Engine is MapReduce
set mapreduce.job.queuename=<queuename>; works

If the Hive Execution Engine is Tez
We need to do
set tez.queue.name=<queuename>;



Cheers

Rajit Saha
------------------------------------------------
Principal DevOps Engineer | BigData
LendingClub
------------------------------------------------



From: Sathi Chowdhury <sathi.chowdhury@lithium.com<mailto:sathi.chowdhury@lithium.com>>
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Date: Friday, February 26, 2016 at 6:01 PM
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Re: Running hive queries in different queue

I think  in your hive script you can do
set mapreduce.job.queuename=<queuename>;
Thanks
Sathi

From: Rajit Saha
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>"
Date: Friday, February 26, 2016 at 5:34 PM
To: "user@hive.apache.org<mailto:user@hive.apache.org>"
Subject: Running hive queries in different queue

Hi

I want to run hive query in a queue others than "default" queue from hive client command line
. Can anybody please suggest a way to do it.

Regards
Rajit

On Feb 26, 2016, at 07:36, Patrick Duin <patduin@gmail.com<mailto:patduin@gmail.com>>
wrote:

Hi Prasanth.

Thanks for the quick reply!

The logs don't show much more of the stacktrace I'm afraid:
java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:809)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


The stacktrace isn't really the issue though. The NullPointer is a symptom caused by not being
able to return any stripes, if you look at the line in the code it is  because the 'stripes'
field is null which should never happen. This, we think, is caused by failing namenode network
traffic. We would have lots of IO warning in the logs saying block's cannot be found or e.g.:
16/02/01 13:20:34 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: java.lang.InterruptedException
        at org.apache.hadoop.ipc.Client.call(Client.java:1448)
        at org.apache.hadoop.ipc.Client.call(Client.java:1400)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy32.getServerDefaults(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getServerDefaults(ClientNamenodeProtocolTranslatorPB.java:268)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy33.getServerDefaults(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getServerDefaults(DFSClient.java:1007)
        at org.apache.hadoop.hdfs.DFSClient.shouldEncryptData(DFSClient.java:2062)
        at org.apache.hadoop.hdfs.DFSClient.newDataEncryptionKey(DFSClient.java:2068)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:208)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:159)
        at org.apache.hadoop.hdfs.net.TcpPeerServer.peerFromSocketAndKey(TcpPeerServer.java:90)
        at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3123)
        at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:755)
        at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:670)
        at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:337)
        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:576)
        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:800)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:848)
        at java.io.DataInputStream.readFully(DataInputStream.java:195)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:407)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:885)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:771)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400)
        at java.util.concurrent.FutureTask.get(FutureTask.java:187)
        at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1047)
        at org.apache.hadoop.ipc.Client.call(Client.java:1442)
        ... 33 more

Our job doesn't always fail sometimes splits get calculated. We suspect when the namenode
is too busy our job maybe hits some time-outs and the whole thing fails.

Our intuition has been the same as you suggest, bigger files is better. But we see a degradation
in performance as soon as our files get bigger than the ORC block size. Keeping file size
within ORC block size sounds silly but when looking at the code (OrcInputFormat) we think
 it cuts out a bunch of code that is causing us problems. The code we are trying to hit is:
https://github.com/apache/hive/blob/release-0.14.0/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L656.
Avoiding the scheduling.

In our case we are not using any SARG but we do use column projection.

Any idea why if we query the data via Hive we don't have this issue?

Let me know if you need more information. Thanks for the insights, much appreciated.

Kind regards,
 Patrick


2016-02-25 22:20 GMT+01:00 Prasanth Jayachandran <pjayachandran@hortonworks.com<mailto:pjayachandran@hortonworks.com>>:

> On Feb 25, 2016, at 3:15 PM, Prasanth Jayachandran <pjayachandran@hortonworks.com<mailto:pjayachandran@hortonworks.com>>
wrote:
>
> Hi Patrick
>
> Can you paste entire stacktrace? Looks like NPE happened during split generation but
stack trace is incomplete to know what caused it.
>
> In Hive 0.14.0, the stripe size is changed to 64MB. The default block size for ORC files
is 256MB. 4 stripes can fit a block. ORC does padding to avoid stripes straddling HDFS blocks.
During split calculation, ORC footer which contains stripe level column statistics is read
to perform split pruning based on predicate condition specified via SARG(Search Argument).
>
> For example: Assume column ‘state’ is sorted and the predicate condition is ‘state’=“CA"
> Stripe 1: min = AZ max = FL
> Stripe 2: min = GA max = MN
> Stripe 3: min = MS max = SC
> Stripe 4: min = SD max = WY
>
> In this case, only stripe 1 satisfies the above predicate condition. So only 1 split
with stripe 1 will be created.
> So if there are huge number of small files, then footers from all files has to be read
to do split pruning. If there are few number of large files then only few footers have to
be read. Also the minimum splittable position is stripe boundary. So having fewer large files
has the advantage of reading less data during split pruning.
>
> If you can send me the full stacktrace, I can tell what is causing the exception here.
I will also let you know of any workaround/next hive version with the fix.
>
> In more recent hive versions, hive 1.2.0 onwards. OrcInputFormat is has strategies to
decided when to read footers and when not to read footers automatically. You can configure
the strategy that you want based on the workload. In case of many small files, footers will
not be read and with large files footers will be read for split pruning.

The default strategy does it automatically (choosing between when to read and when not to
footers). It is configurable as well.

>
> Thanks
> Prasanth
>
>> On Feb 25, 2016, at 7:08 AM, Patrick Duin <patduin@gmail.com<mailto:patduin@gmail.com>>
wrote:
>>
>> Hi,
>>
>> We've recently moved one of our datasets to ORC and we use Cascading and Hive to
read this data. We've had problems reading the data via Cascading, because of the generation
of splits.
>> We read in a large number of files (thousands) and they are about 1GB each. We found
that the split calculation took minutes on our cluster and often didn't succeed at all (when
our namenode was busy).
>> When digging through the code of the 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.class'
we figured out that if we make the files less then the ORC block size (256MB) the code would
avoid lots of namenode calls. We applied this solution and made our files smaller and that
solved the problem. Split calculation in our job went from 10+ mins to a couple of seconds
and always succeeds.
>> We feel it is counterintuitive as bigger files are usually better in HDFS. We've
also seen that doing a hive query on the data does not present this problem. Internally Hive
seem to take a completely different execution path and is not using the OrcInputFormat but
uses 'org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.class'.
>>
>> Can someone explain the reason for this difference or shed some light on the behaviour
we are seeing? Any help will be greatly appreciated. We are using hive-0.14.0.
>>
>> Kind regards,
>> Patrick
>>
>> Here is the stack-trace that we would see when our Cascading job failed to calculate
the splits:
>> Caused by: java.lang.RuntimeException: serious problem
>>        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$Context.waitForTasks(OrcInputFormat.java:478)
>>        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:949)
>>        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:974)
>>        at com.hotels.corc.mapred.CorcInputFormat.getSplits(CorcInputFormat.java:201)
>>        at cascading.tap.hadoop.io.MultiInputFormat.getSplits(MultiInputFormat.java:200)
>>        at cascading.tap.hadoop.io.MultiInputFormat.getSplits(MultiInputFormat.java:142)
>>        at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
>>        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
>>        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
>>        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>>        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at javax.security.auth.Subject.doAs(Subject.java:415)
>>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
>>        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:585)
>>        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:580)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at javax.security.auth.Subject.doAs(Subject.java:415)
>>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:580)
>>        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:571)
>>        at cascading.flow.hadoop.planner.HadoopFlowStepJob.internalNonBlockingStart(HadoopFlowStepJob.java:106)
>>        at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:265)
>>        at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:184)
>>        at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:146)
>>        at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:48)
>>        ... 4 more
>> Caused by: java.lang.NullPointerException
>>        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:809)
>



________________________________
DISCLAIMER: The information transmitted is intended only for the person or entity to which
it is addressed and may contain confidential and/or privileged material. Any review, re-transmission,
dissemination or other use of, or taking of any action in reliance upon this information by
persons or entities other than the intended recipient is prohibited. If you received this
in error, please contact the sender and destroy any copies of this document and any attachments.
Mime
View raw message