impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darren Hoo <>
Subject impalad using 400% CPU
Date Wed, 25 Jan 2017 05:02:06 GMT
One of implad daemons in our cluster sometimes using a lot of cpu, and
won't respond to CM's health checks.
just now the daemon got stuck again,  there's one query in flight,  and it
has been running for 4 hours. Cancel this query has no effect,  unless
restart the daemonQueries

This page lists all running queries, plus any completed queries that are
archived in memory. The size of that archive is controlled with the
--query_log_size command line parameter.
1 queries in flight
UserDefault DbStatementQuery TypeStart TimeDurationScan ProgressStateLast
Event# rows fetchedResource PoolDetailsAction
default GET_TABLES QUERY 2017-01-25 04:14:06.385105000 4h4m N/A CREATED 0
waiting to be closed [?] <http://slave24:25000/queries#>
UserDefault DbStatementQuery TypeStart TimeWaiting TimeDurationScan Progress
StateLast Event# rows fetchedResource PoolDetailsAction

here's the output of perf top, it seems that it got stuck in the JVM:

Samples: 553K of event 'cycles', Event count (approx.): 81102178474

Overhead  Shared Object        Symbol

  55.39%            [.] SpinPause

  21.90%            [.] ParallelTaskTerminator::offer_termination

   6.81%  [kernel]             [k] get_page

   5.47%  [kernel]             [k] copy_pte_range

   1.16%  [kernel]             [k] smp_invalidate_interrupt

   0.52%         [.] memcpy

   0.47%  [kernel]             [k] clear_page_c

   0.47%  [kernel]             [k] __bitmap_empty

   0.44%  [kernel]             [k] copy_user_generic_string

   0.43%  [kernel]             [k] vm_normal_page

   0.42%  [kernel]             [k] default_send_IPI_mask_sequence_phys

   0.38%         [.] __memset_sse2

   0.33%  [kernel]             [k] page_fault

   0.29%  [kernel]             [k] native_set_pte_at

   0.25%  [kernel]             [k] invalidate_interrupt0

   0.19%  [kernel]             [k] _spin_lock

   0.19%  [kernel]             [k] __alloc_pages_nodemask

   0.16%  [kernel]             [k] copy_page_c

   0.16%  [kernel]             [k] flush_tlb_others_ipi

   0.14%            [.]
long, (unsigned short)1792, 131072u>, (unsigned short)256>::peek

   0.13%  [kernel]             [k] igb_poll

   0.13%  [kernel]             [k] find_next_bit

   0.11%  [kernel]             [k] invalidate_interrupt2

   0.10%  [kernel]             [k] list_del

   0.09%  [kernel]             [k] invalidate_interrupt3

   0.08%  [kernel]             [k] get_page_from_freelist

   0.07%  [kernel]             [k] do_wp_page

   0.07%  [kernel]             [k] flush_tlb_page

   0.07%  [kernel]             [k] __mem_cgroup_commit_charge

   0.06%  [kernel]             [k] schedule

   0.06%  [kernel]             [k] update_curr

   0.06%  [kernel]             [k] invalidate_interrupt1

   0.05%  [kernel]             [k] __wake_up_bit

   0.05%  [kernel]             [k] put_page

   0.05%  [kernel]             [k] kfree

   0.05%  [kernel]             [k] handle_mm_fault

   0.05%  [kernel]             [k] page_add_new_anon_rmap

   0.05%  [kernel]             [k] __alloc_skb

   0.04%  [kernel]             [k] tcp_recvmsg

   0.04%  [kernel]             [k] up_read

   0.04%  [kernel]             [k] irq_entries_start

   0.04%  [kernel]             [k] kmem_cache_free

   0.04%  [kernel]             [k] native_write_msr_safe

   0.04%  [kernel]             [k] save_args

View raw message