hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "slim bouguerra (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HIVE-21334) Eviction of blocks is major source of blockage for allocation request. Allocation path need to be lock-free.
Date Wed, 27 Feb 2019 17:43:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

slim bouguerra reassigned HIVE-21334:
-------------------------------------


> Eviction of blocks is major source of blockage for allocation request. Allocation path
need to be lock-free.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-21334
>                 URL: https://issues.apache.org/jira/browse/HIVE-21334
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: slim bouguerra
>            Assignee: slim bouguerra
>            Priority: Major
>         Attachments: lock_profile.png
>
>
> Eviction is getting in the way of memory allocation when the query fragment has no cache
entry.
> This is cause major bottleneck and waist lot of cpu cycles.
> To fix this is first we can batch the evictions to avoid taking the lock multiple times.
> The memory manager need to be able to anticipate such issue and keep some spare space
for queries that do not have any hit.
> {code}
> IO-Elevator-Thread-12  Blocked CPU usage on sample: 692ms
>   org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.evictSomeBlocks(long) LowLevelLrfuCachePolicy.java:264
>   org.apache.hadoop.hive.llap.cache.CacheContentsTracker.evictSomeBlocks(long) CacheContentsTracker.java:194
>   org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(long, boolean,
AtomicBoolean) LowLevelCacheMemoryManager.java:87
>   org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(long, AtomicBoolean)
LowLevelCacheMemoryManager.java:63
>   org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(MemoryBuffer[], int,
Allocator$BufferObjectFactory, AtomicBoolean) BuddyAllocator.java:263
>   org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.allocateMultiple(MemoryBuffer[],
int) EncodedReaderImpl.java:1295
>   org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(long,
DiskRangeList, long, long, EncodedColumnBatch$ColumnStreamData, long, long, IdentityHashMap)
EncodedReaderImpl.java:923
>   org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(int,
StripeInformation, OrcProto$RowIndex[], List, List, boolean[], boolean[], Consumer) EncodedReaderImpl.java:501
>   org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead() OrcEncodedDataReader.java:407
>   org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run() OrcEncodedDataReader.java:266
>   org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run() OrcEncodedDataReader.java:263
>   java.security.AccessController.doPrivileged(PrivilegedExceptionAction, AccessControlContext)
AccessController.java (native)
>   javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction) Subject.java:422
>   org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction) UserGroupInformation.java:1688
>   org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal() OrcEncodedDataReader.java:263
>   org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal() OrcEncodedDataReader.java:110
>   org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
>   org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call()
StatsRecordingThreadPool.java:110
>   java.util.concurrent.FutureTask.run() FutureTask.java:266
>   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1142
>   java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:617
>   java.lang.Thread.run() Thread.java:745 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message