rk3399pro 内核硬死锁

1内核硬死锁

注意：内核死锁概念及检测方法在《内核软死锁》一文中已经介绍，所以这里不再介绍

Hard lockup比soft lockup更加严重，CPU不仅无法执行其它进程，而且不再响应中断。检测hard lockup的原理利用了PMU的NMI perf event，因为NMI中断是不可屏蔽的，在CPU不再响应中断的情况下仍然可以得到执行，它再去检查时钟中断的计数器hrtimer_interrupts是否在保持递增，如果停滞就意味着时钟中断未得到响应，也就是发生了hard lockup。

2测试Demo

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/kthread.h>
#include <linux/spinlock.h>

MODULE_LICENSE("GPL");

struct task_struct *task0;

static int
hog_thread(void *data)
{
    static DEFINE_SPINLOCK(lock);
    unsigned long flags;

    printk(KERN_INFO "Hogging a CPU now\n");
    spin_lock_irqsave(&lock, flags);
    while (1);

    /* unreached */
    return 0;
}

static int __init
hog_init(void)
{
    task0 = kthread_run(&hog_thread, NULL, "hg-thread");
    set_cpus_allowed_ptr(task0, cpumask_of(4));
    return 0;
}

module_init(hog_init);

3硬死锁log信息

rk3399pro上运行测试demo出现硬死锁log信息如下：

[ 1213.514607] hardlockup_test: loading out-of-tree module taints kernel.
[ 1213.517932] Hogging a CPU now
[ 1214.546171] rga: Rga sync pid 775 wait 1 task done timeout
[ 1214.546698] rga: interrupt = 700 status = 100
[ 1214.747994] rga: Rga sync pid 775 wait 1 task done timeout
[ 1214.748504] rga: interrupt = 700 status = 100
[ 1216.199323] rga: Rga sync pid 775 wait 1 task done timeout
[ 1216.199836] rga: interrupt = 700 status = 100
[ 1216.399275] rga: Rga sync pid 775 wait 1 task done timeout
[ 1216.399782] rga: interrupt = 700 status = 100
[ 1216.599234] rga: Rga sync pid 775 wait 1 task done timeout
[ 1216.599742] rga: interrupt = 700 status = 100
[ 1216.799245] rga: Rga sync pid 775 wait 1 task done timeout
[ 1216.799754] rga: interrupt = 700 status = 100
[ 1216.999299] rga: Rga sync pid 775 wait 1 task done timeout
[ 1216.999806] rga: interrupt = 700 status = 100
[ 1217.199315] rga: Rga sync pid 775 wait 1 task done timeout
[ 1217.199824] rga: interrupt = 700 status = 100
[ 1217.399350] rga: Rga sync pid 775 wait 1 task done timeout
[ 1217.399857] rga: interrupt = 700 status = 100
[ 1217.600337] rga: Rga sync pid 775 wait 1 task done timeout
[ 1217.600843] rga: interrupt = 700 status = 100
[ 1217.802142] rga: Rga sync pid 775 wait 1 task done timeout
[ 1217.802650] rga: interrupt = 700 status = 100
[ 1236.200103] Watchdog detected hard LOCKUP on cpu 4
[ 1236.200575] ------------[ cut here ]------------
[ 1236.201037] WARNING: at kernel/watchdog.c:352
[ 1236.201435] Modules linked in: hardlockup_test(O)
[ 1236.201897]
[ 1236.202068] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O    4.4.194 #9
[ 1236.202718] Hardware name: Rockchip RK3399pro evb v14 board for linux (DT)
[ 1236.203340] task: ffffffc0f1c4a880 task.stack: ffffffc0f1c5c000
[ 1236.203901] PC is at watchdog_timer_fn+0x170/0x2e0
[ 1236.204351] LR is at watchdog_timer_fn+0x170/0x2e0
[ 1236.204785] pc : [<ffffff800812cadc>] lr : [<ffffff800812cadc>] pstate: 600001c5
[ 1236.205447] sp : ffffffc0f6f0de40
[ 1236.205752] x29: ffffffc0f6f0de40 x28: 0000000000000004
[ 1236.206268] x27: ffffff8008f67720 x26: ffffff8008f2d240
[ 1236.206783] x25: ffffffc0f6f10258 x24: ffffffc0f1c5fd90
[ 1236.207298] x23: 0000000000000000 x22: 0000000000000000
[ 1236.207813] x21: ffffff8008f66000 x20: ffffff8008f2d2b8
[ 1236.208328] x19: ffffff8008f2d000 x18: ffffff80890f9897
[ 1236.208844] x17: 0000000000000000 x16: 0000000000000000
[ 1236.209360] x15: 0000000000000000 x14: 0000000000030da7
[ 1236.209875] x13: 000000000000000a x12: 0000000000000030
[ 1236.210391] x11: 00000000fffffffe x10: ffffff80090f989f
[ 1236.210907] x9 : 0000000005f5e0ff x8 : ffffff800835b154
[ 1236.211423] x7 : ffffff8008fce488 x6 : 0000000000000035
[ 1236.211938] x5 : 0000000000000000 x4 : 0000000000000000
[ 1236.212454] x3 : 00000040edfe3000 x2 : 00000040edfe3000
[ 1236.212970] x1 : 0000000000000003 x0 : 0000000000000026
[ 1236.213487]
[ 1236.213487] PC: 0xffffff800812ca5c:
[ 1236.213944] ca5c  f9401000 d63f0000 aa0003e1 aa1b03e2 aa1903e0 97ff571c b50003f6 91022694
[ 1236.214781] ca7c  d538d080 38606a81 34000041 38206a9f 97ffff7b a94153f3 52800020 a9425bf5
[ 1236.215613] ca9c  a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020283 f87a6840 f8636845
[ 1236.216444] cabc  91022294 eb05001f 54000b60 f8236840 f87c7b60 3834681f 17ffffda 9400d45c
[ 1236.217274] cadc  d4210000 f87c7b60 52800021 38346801 17ffffd4 97fe7624 f9438aa1 36080921
[ 1236.218105] cafc  b9401341 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841 6b160016
[ 1236.218940] cb1c  54000800 91022a99 d538d080 38796801 34000101 91024294 f8746802 d5384101
[ 1236.219772] cb3c  eb01005f 54fffa80 3839681f 17ffffd1 34000157 900081a1 52800000 910ce021
[ 1236.220605]
[ 1236.220605] LR: 0xffffff800812ca5c:
[ 1236.221061] ca5c  f9401000 d63f0000 aa0003e1 aa1b03e2 aa1903e0 97ff571c b50003f6 91022694
[ 1236.221895] ca7c  d538d080 38606a81 34000041 38206a9f 97ffff7b a94153f3 52800020 a9425bf5
[ 1236.222726] ca9c  a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020283 f87a6840 f8636845
[ 1236.223557] cabc  91022294 eb05001f 54000b60 f8236840 f87c7b60 3834681f 17ffffda 9400d45c
[ 1236.224389] cadc  d4210000 f87c7b60 52800021 38346801 17ffffd4 97fe7624 f9438aa1 36080921
[ 1236.225224] cafc  b9401341 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841 6b160016
[ 1236.226054] cb1c  54000800 91022a99 d538d080 38796801 34000101 91024294 f8746802 d5384101
[ 1236.226885] cb3c  eb01005f 54fffa80 3839681f 17ffffd1 34000157 900081a1 52800000 910ce021
[ 1236.227721]
[ 1236.227721] SP: 0xffffffc0f6f0ddc0:
[ 1236.228177] ddc0  00000000 00000000 00000000 00000000 f1c5fd90 ffffffc0 f6f10258 ffffffc0
[ 1236.229008] dde0  08f2d240 ffffff80 08f67720 ffffff80 00000004 00000000 f6f0de40 ffffffc0
[ 1236.229840] de00  0812cadc ffffff80 f6f0de40 ffffffc0 0812cadc ffffff80 600001c5 00000000
[ 1236.230671] de20  08f66710 ffffff80 0000012f 00000000 00000000 00000080 00000003 00000000
[ 1236.231505] de40  f6f0deb0 ffffffc0 08102f3c ffffff80 f6f10258 ffffffc0 f6f0fc00 ffffffc0
[ 1236.232336] de60  f6f0fc80 ffffffc0 08f27000 ffffff80 00000001 00000000 090b6a38 ffffff80
[ 1236.233167] de80  08f27000 ffffff80 0812c96c ffffff80 ca270ada 0000011f f1c4a880 ffffffc0
[ 1236.233998] dea0  f6f10258 ffffffc0 08f66710 ffffff80 f6f0df20 ffffffc0 08103558 ffffff80
[ 1236.234840]
[ 1236.234840] X7: 0xffffff8008fce408:
[ 1236.235296] e408  f1fb7600 ffffffc0 00000000 00000000 00000000 00000000 00110000 00000000
[ 1236.236127] e428  000f0000 00000000 00000000 00000000 00040000 00000000 00080000 00000000
[ 1236.236961] e448  00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000
[ 1236.237791] e468  00000000 00000001 00000000 00000001 00000002 00000001 00000001 00000000
[ 1236.238621] e488  00000000 00000000 08cf91d9 ffffff80 06060606 dead4ead ffffffff 00000000
[ 1236.239451] e4a8  ffffffff ffffffff f1840000 ffffffc0 00040000 00000000 00000001 00000000
[ 1236.240284] e4c8  00000000 dead4ead ffffffff 00000000 ffffffff ffffffff 08fce4e0 ffffff80
[ 1236.241118] e4e8  08fce4e0 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
[ 1236.241952]
[ 1236.241952] X8: 0xffffff800835b0d4:
[ 1236.242408] b0d4  9400054a aa1303e0 94000554 52800000 a94153f3 f94013f5 a8c37bfd d65f03c0
[ 1236.243246] b0f4  f9400413 17fffff6 f9400813 17fffff4 f9400c13 17fffff2 128002a0 17fffff5
[ 1236.244077] b114  71001c1f 54000161 f9405ce0 f9400c00 b4000140 a9bf7bfd aa0603e2 aa0403e1
[ 1236.244910] b134  910003fd 940004ce a8c17bfd d65f03c0 128002a0 d65f03c0 12800160 d65f03c0
[ 1236.245744] b154  a9bb7bfd 7100081f 910003fd a9025bf5 aa0403f6 a90153f3 aa0603f5 a90363f7
[ 1236.246575] b174  f9405cf3 54000181 f9400660 b4000a20 2a1503e2 aa1603e1 9400047d 52800000
[ 1236.247406] b194  a94153f3 a9425bf5 a94363f7 a8c57bfd d65f03c0 71000c1f 54000061 f9400a60
[ 1236.248237] b1b4  17fffff3 71001c1f 54000061 f9400e60 17ffffef 35000820 51000420 7100041f
[ 1236.249077]
[ 1236.249077] X10: 0xffffff80090f981f:
[ 1236.249534] 981c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1236.250365] 983c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1236.251199] 985c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1236.252029] 987c  00000000 00000000 00000000 00000000 00000000 00000001 00000000 30303020
[ 1236.252859] 989c  322e3633 35383233 39205d39 20633938 322e3633 37303934 58205d37 203a3031
[ 1236.253689] 98bc  66667830 66666666 39303038 38396630 0a3a6631 32313830 63646163 70205d3e
[ 1236.254521] 98dc  74617473 36203a65 30303030 0a356331 6f632077 6f72746e 7872206c 0a78742f
[ 1236.255351] 98fc  72700a31 7265706f 69207974 6f6e206e 2f206564 65776f70 616d2d72 6567616e
[ 1236.256185] 991c  746e656d 33666640 30303031 6f702f30 2d726577 746e6f63 6c6c6f72 66207265
[ 1236.257028]
[ 1236.257028] X18: 0xffffff80890f9817:
[ 1236.257485] 9814  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.258327] 9834  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.259158] 9854  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.259992] 9874  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.260827] 9894  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.261659] 98b4  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.262490] 98d4  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.263322] 98f4  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.264157] 9914  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.264991]
[ 1236.264991] X19: 0xffffff8008f2cf80:
[ 1236.265448] cf80  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.266283] cfa0  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.267114] cfc0  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.267946] cfe0  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.268782] d000  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.269614] d020  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.270445] d040  ******** ******** ******** ******** ******** ******** ******** ********
[ 1236.271277] d
                [ 1273.523922] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1273.524479]  4-...: (0 ticks this GP) idle=131/140000000000000/0 softirq=36325/36325 fqs=9477
[ 1273.525249]  (detected by 3, t=60004 jiffies, g=38938, c=38937, q=5427)
[ 1273.525872] Task dump for CPU 4:
[ 1273.526179] hg-thread       R  running task        0  6280      2 0x00000012
[ 1273.526842] Call trace:
[ 1273.527108] [<ffffff8008085048>] __switch_to+0x94/0xbc
[ 1273.527591] [<ffffff8008a43ccc>] schedule+0x70/0x84
[ 1444.214208] INFO: task jbd2/mmcblk0p8-:167 blocked for more than 120 seconds.
[ 1444.214873]       Tainted: G        W  O    4.4.194 #9
[ 1444.215345] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1444.216050] jbd2/mmcblk0p8- D ffffff8008085048     0   167      2 0x00000010
[ 1444.216702] Call trace:
[ 1444.216964] [<ffffff8008085048>] __switch_to+0x94/0xbc
[ 1444.217433] [<ffffff8008a4395c>] __schedule+0x324/0x624
[ 1444.217910] [<ffffff8008a43ccc>] schedule+0x70/0x84
[ 1444.218347] [<ffffff8008a46380>] schedule_timeout+0x34/0x210
[ 1444.218858] [<ffffff8008a43600>] io_schedule_timeout+0x70/0xa8
[ 1444.219384] [<ffffff8008a444a8>] bit_wait_io+0x20/0x68
[ 1444.219851] [<ffffff8008a440e8>] __wait_on_bit+0x74/0xc0
[ 1444.220330] [<ffffff8008a441a4>] out_of_line_wait_on_bit+0x70/0x80
[ 1444.220893] [<ffffff80081dc6a8>] __wait_on_buffer+0x3c/0x48
[ 1444.221399] [<ffffff800826a81c>] jbd2_journal_commit_transaction+0xc20/0x14f0
[ 1444.222049] [<ffffff800826fc54>] kjournald2+0xb8/0x224
[ 1444.222515] [<ffffff80080b8200>] kthread+0xdc/0xec
[ 1444.222960] [<ffffff8008082f00>] ret_from_fork+0x10/0x50
[ 1444.223440] Kernel panic - not syncing: hung_task: blocked tasks
[ 1444.223974] CPU: 5 PID: 39 Comm: khungtaskd Tainted: G        W  O    4.4.194 #9
[ 1444.224629] Hardware name: Rockchip RK3399pro evb v14 board for linux (DT)
[ 1444.225237] Call trace:
[ 1444.225463] [<ffffff80080882c0>] dump_backtrace+0x0/0x220
[ 1444.225940] [<ffffff8008088504>] show_stack+0x24/0x30
[ 1444.226395] [<ffffff80083b8dec>] dump_stack+0x94/0xbc
[ 1444.226851] [<ffffff8008161a5c>] panic+0xe8/0x240
[ 1444.227273] [<ffffff800812c2b0>] watchdog+0x2ec/0x304
[ 1444.227724] [<ffffff80080b8200>] kthread+0xdc/0xec
[ 1444.228152] [<ffffff8008082f00>] ret_from_fork+0x10/0x50
[ 1444.228637] CPU0: stopping
[ 1444.228921] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  O    4.4.194 #9
[ 1444.229571] Hardware name: Rockchip RK3399pro evb v14 board for linux (DT)
[ 1444.230185] Call trace:
[ 1444.230437] [<ffffff80080882c0>] dump_backtrace+0x0/0x220
[ 1444.230935] [<ffffff8008088504>] show_stack+0x24/0x30
[ 1444.231398] [<ffffff80083b8dec>] dump_stack+0x94/0xbc
[ 1444.231861] [<ffffff800808cba8>] handle_IPI+0x1e0/0x260
[ 1444.232344] [<ffffff8008080eec>] gic_handle_irq+0x12c/0x184
[ 1444.232849] Exception stack(0xffffff8008f63d60 to 0xffffff8008f63e90)
[ 1444.233440] 3d60: ffffffc0f6ed50c0 00000040edfa4000 ffffffc0f6ed50c0 001fbbf9fa2c50f3
[ 1444.234145] 3d80: 00ffffffffffffff 00000008140385b0 0000000000235056 0000000000000000
[ 1444.234849] 3da0: 00000032b5593519 ffffff8008081800 0000000000001000 0000000000000000
[ 1444.235554] 3dc0: 0000000034d5d91d 00000040edfa4000 ffffff8008f6d950 0000000000000000
[ 1444.236257] 3de0: 0000000000000000 0000000000000000 0000000030d00800 0000015038d4e80b
[ 1444.236961] 3e00: 0000000000000000 ffffffc0f13fd400 ffffff80090b6f60 0000000000000000
[ 1444.237664] 3e20: 0000015038cdb5e6 ffffff8009060d40 ffffff8008f6d950 0000000000c4923c
[ 1444.238367] 3e40: 000000000105001c ffffff8008f63e90 ffffff800877ee40 ffffff8008f63e90
[ 1444.239072] 3e60: ffffff800877ee74 0000000060000145 ffffff8008f63e90 ffffff800877ee40
[ 1444.239772] 3e80: ffffffffffffffff 0000000000000000
[ 1444.240221] [<ffffff80080827b4>] el1_irq+0xb4/0x140
[ 1444.240681] [<ffffff800877ee74>] cpuidle_enter_state+0x1cc/0x25c
[ 1444.241235] [<ffffff800877ef78>] cpuidle_enter+0x34/0x44
[ 1444.241724] [<ffffff80080e0da4>] call_cpuidle+0x6c/0x74
[ 1444.242207] [<ffffff80080e1058>] cpu_startup_entry+0x2ac/0x2bc
[ 1444.242739] [<ffffff8008a42b38>] rest_init+0x74/0x7c
[ 1444.243201] [<ffffff8008e50bc4>] start_kernel+0x3c4/0x3e8
[ 1444.243696] [<ffffff8008e501c4>] __primary_switched+0x30/0x6c
[ 1444.244223] CPU3: stopping
[ 1444.244494] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G        W  O    4.4.194 #9
[ 1444.245144] Hardware name: Rockchip RK3399pro evb v14 board for linux (DT)
[ 1444.245759] Call trace:
[ 1444.246007] [<ffffff80080882c0>] dump_backtrace+0x0/0x220
[ 1444.246505] [<ffffff8008088504>] show_stack+0x24/0x30
[ 1444.246966] [<ffffff80083b8dec>] dump_stack+0x94/0xbc
[ 1444.247427] [<ffffff800808cba8>] handle_IPI+0x1e0/0x260
[ 1444.247910] [<ffffff8008080eec>] gic_handle_irq+0x12c/0x184
[ 1444.248414] Exception stack(0xffffffc0f1c5fd90 to 0xffffffc0f1c5fec0)
[ 1444.249000] fd80:                                   0000000000000000 0000000000000003
[ 1444.249704] fda0: 00000040edfe3000 0000000000000020 00ffffffffffffff 00000008140961b0
[ 1444.250407] fdc0: 0000000000235058 0000000000000003 00000032b5593519 ffffff8008081800
[ 1444.251111] fde0: 0000000000001000 0000000000000000 0000000034d5d91d 00000040edfe3000
[ 1444.251815] fe00: ffffffc0f1c4a880 0000000000000000 0000000000000000 0000000000000000
[ 1444.252520] fe20: 0000000030d00800 0000015038d78f07 0000000000000002 ffffffc0f13fda00
[ 1444.253226] fe40: ffffff80090b6f60 0000000000000001 0000015037ed2dc2 ffffff8009060d40
[ 1444.253929] fe60: 00000000014c9000 0000000000c49244 0000000000000000 ffffffc0f1c5fec0
[ 1444.254633] fe80: ffffff800877ee70 ffffffc0f1c5fec0 ffffff800877ee74 0000000060000145
[ 1444.255338] fea0: ffffffc0f1c5fec0 ffffff800877ee70 ffffffffffffffff 0000000000000002
[ 1444.256041] [<ffffff80080827b4>] el1_irq+0xb4/0x140
[ 1444.256493] [<ffffff800877ee74>] cpuidle_enter_state+0x1cc/0x25c
[ 1444.257046] [<ffffff800877ef78>] cpuidle_enter+0x34/0x44
[ 1444.257530] [<ffffff80080e0da4>] call_cpuidle+0x6c/0x74
[ 1444.258013] [<ffffff80080e1058>] cpu_startup_entry+0x2ac/0x2bc
[ 1444.258542] [<ffffff800808c570>] secondary_start_kernel+0x190/0x1bc
[ 1444.259114] [<0000000000c49188>] 0xc49188
[ 1444.259490] CPU1: stopping
[ 1444.259761] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W  O    4.4.194 #9
[ 1444.260411] Hardware name: Rockchip RK3399pro evb v14 board for linux (DT)
[ 1444.261025] Call trace:
[ 1444.261271] [<ffffff80080882c0>] dump_backtrace+0x0/0x220
[ 1444.261769] [<ffffff8008088504>] show_stack+0x24/0x30
[ 1444.262231] [<ffffff80083b8dec>] dump_stack+0x94/0xbc
[ 1444.262691] [<ffffff800808cba8>] handle_IPI+0x1e0/0x260
[ 1444.263174] [<ffffff8008080eec>] gic_handle_irq+0x12c/0x184
[ 1444.263678] Exception stack(0xffffffc0f1c53d90 to 0xffffffc0f1c53ec0)
[ 1444.264263] 3d80:                                   0000000000000000 0000000000000001
[ 1444.264966] 3da0: 00000040edfb9000 0000000000000020 00ffffffffffffff 00000008140385b0
[ 1444.265670] 3dc0: 0000000000235056 0000000000000001 00000032b5593519 ffffff8008081800
[ 1444.266373] 3de0: 0000000000001000 0000000000000000 0000000034d5d91d 00000040edfb9000
[ 1444.267077] 3e00: ffffffc0f1c48d80 0000000000000000 0000000000000000 0000000000000000
[ 1444.267781] 3e20: 0000000030d00800 0000015038d6691b 0000000000000002 ffffffc0f13fd600
[ 1444.268484] 3e40: ffffff80090b6f60 0000000000000001 0000015037dc8fdb ffffff8009060d40
[ 1444.269188] 3e60: 00000000014c9000 0000000000c49244 0000000000000000 ffffffc0f1c53ec0
[ 1444.269892] 3e80: ffffff800877ee70 ffffffc0f1c53ec0 ffffff800877ee74 0000000060000145
[ 1444.270596] 3ea0: ffffffc0f1c53ec0 ffffff800877ee70 ffffffffffffffff 0000000000000002
[ 1444.271298] [<ffffff80080827b4>] el1_irq+0xb4/0x140
[ 1444.271749] [<ffffff800877ee74>] cpuidle_enter_state+0x1cc/0x25c
[ 1444.272302] [<ffffff800877ef78>] cpuidle_enter+0x34/0x44
[ 1444.272785] [<ffffff80080e0da4>] call_cpuidle+0x6c/0x74
[ 1444.273268] [<ffffff80080e1058>] cpu_startup_entry+0x2ac/0x2bc
[ 1444.273798] [<ffffff800808c570>] secondary_start_kernel+0x190/0x1bc
[ 1444.274369] [<0000000000c49188>] 0xc49188
[ 1444.274745] CPU2: stopping
[ 1444.275016] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W  O    4.4.194 #9
[ 1444.275667] Hardware name: Rockchip RK3399pro evb v14 board for linux (DT)
[ 1444.276281] Call trace:
[ 1444.276528] [<ffffff80080882c0>] dump_backtrace+0x0/0x220
[ 1444.277025] [<ffffff8008088504>] show_stack+0x24/0x30
[ 1444.277498] [<ffffff80083b8dec>] dump_stack+0x94/0xbc
[ 1444.277959] [<ffffff800808cba8>] handle_IPI+0x1e0/0x260
[ 1444.278441] [<ffffff8008080eec>] gic_handle_irq+0x12c/0x184
[ 1444.278945] Exception s
                          [ 1445.395292] SMP: failed to stop secondary CPUs
[ 1445.395691] CRU:
[ 1445.395865] 00000000: 00000044 00002201 80000000 00000108 00000007 00007f00 00000000 00000000
[ 1445.396616] 00000020: 00000032 00001101 80000000 00000108 00000007 00007f00 00000000 00000000
[ 1445.397369] 00000040: 00000064 00002601 8000031f 00000108 00000007 00007f00 00000000 00000000
[ 1445.398119] 00000060: 000000c0 00001302 8000031f 00000008 00000007 00007f00 00000000 00000000
[ 1445.398870] 00000080: 00000064 00001301 8000031f 00000108 00000007 00007f00 00000000 00000000
[ 1445.399620] 000000a0: 00000063 00001401 80000000 00000108 00000007 00007f00 00000000 00000000
[ 1445.400370] 000000c0: 00000024 00004201 80000000 00000108 00000007 00007f00 00000000 00000000
[ 1445.401120] 000000e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.401870] 00000100: 00000100 00000202 00000140 00006505 00000083 00008580 00008320 00000141
[ 1445.402621] 00000120: 00000181 00008181 00000141 00000141 00004141 00008343 00003185 00000201
[ 1445.403371] 00000140: 0000010f 00000300 00004a87 0000033f 00009381 00008387 00001103 00005282
[ 1445.404122] 00000160: 00008247 00000187 00001743 0000001f 00000300 00000380 00000380 00000000
[ 1445.404871] 00000180: 0000e300 00001200 00000200 00000200 00000200 00000200 00003f3f 00000081
[ 1445.405621] 000001a0: 00000000 00000000 00004141 00000421 00008700 000002dc 00000087 00000381
[ 1445.406371] 000001c0: 00000381 00000001 00000000 00000087 00000087 00000141 00000141 00008181
[ 1445.407121] 000001e0: 0000838f 00000007 00008f1f 00008f8f 00008f8f 00008787 00008787 00008787
[ 1445.407872] 00000200: 000000cf 000000cf 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.408621] 00000220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.409372] 00000240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.410122] 00000260: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.410873] 00000280: 0bb8ea60 0bb8ea60 0bb8ea60 0bb8ea60 0bb8ea60 0bb8ea60 0bb8ea60 0bb8ea60
[ 1445.411623] 000002a0: 0bb8ea60 0bb8ea60 00010014 00010014 00000000 00000000 00000000 00000000
[ 1445.412373] 000002c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.413123] 000002e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.413873] 00000300: 00000080 00000080 00000202 0000006b 00000c30 00000000 0000001a 000001c0
[ 1445.414624] 00000320: 0000fff8 0000f0ff 0000d0ff 0000c130 0000004a 0000e233 00000000 00000000
[ 1445.415374] 00000340: 00000505 00000505 00000000 00000000 00000200 0000000f 0000cfeb 00003f00
[ 1445.416124] 00000360: 0000ef60 00000020 00000ffe 00000040 0000000c 00000f86 00000d0a 00000198
[ 1445.416874] 00000380: 00002010 00000100 00000029 00000000 00000000 00000000 00000000 00000000
[ 1445.417624] 000003a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.418375] 000003c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.419124] 000003e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.419875] 00000400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.420625] 00000420: 00000000 00004070 00000000 00000014 00000000 00000000 00000000 00000000
[ 1445.421375] 00000440: 00000000 00000000 00000000 00000000 00001000 00000000 00000000 00000000
[ 1445.422126] 00000460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.422876] 00000480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.423626] 000004a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.424377] 000004c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.425126] 000004e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.425877] 00000500: 00000000 00000000 00640064 00000000 00000003 00000001 00000000 00000000
[ 1445.426627] 00000520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.427377] 00000540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.428127] 00000560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.428876] 00000580: 00000004 00000000 00000004 00000864 00000004
[ 1445.429416] PMU CRU:
[ 1445.429621] 00000000: 000000a9 00001203 80000000 00000108 00000007 00007f00 00000000 00000000
[ 1445.430373] 00000020: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.431124] 00000040: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.431876] 00000060: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.432627] 00000080: 0000060d 00002086 00000303 00000003 000002dc 00000200 0bb8ea60 0bb8ea60
[ 1445.433380] 000000a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.434132] 000000c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.434883] 000000e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.435634] 00000100: 00000efc 0000fb80 0000000f 00000000 00000024 00000000 00000000 00000000
[ 1445.436384] 00000120: 00000000 00000000 00000000 00000000 00000003
[ 1445.436927] PMU:
[ 1445.437100] 00000000: 00000000 00000000 00000000 00000000 00000000 500f8100 500f8100 00000000
[ 1445.437852] 00000020: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.438603] 00000040: 00000000 00000000 00000000 00000000 00000000 d81f833f 00000000 00000000
[ 1445.439355] 00000060: 104000f9 104000f9 104000f9 00000010 00000000 00007e00 00000000 28800000
[ 1445.440107] 00000080: 000003c0 00011940 00005dc0 000003c0 00005dc0 00000060 0000000f 00000018
[ 1445.440858] 000000a0: 00000018 00000018 00000018 00000006 00000006 00005dc0 00005dc0 00005dc0
[ 1445.441610] 000000c0: 00000000 00000000 00000000 00000000 00000000 00000000 3fffffff 00000000
[ 1445.442361] 000000e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.443109] Kernel Offset: disabled
[ 1445.443424] Memory Limit: none
[ 1445.443729] rockchip-thermal ff260000.tsadc: channal 0: temperature(47 C)
[ 1445.444327] THERMAL REGS:
[ 1445.444566] 00000000: 00000200 00030133 00000031 00000002 00000000 00000000 00000000 00000000
[ 1445.445316] 00000020: 00000229 00000227 00000000 00000000 0000024e 00000000 00000000 00000000
[ 1445.446067] 00000040: 000002a5 000002a5 00000000 00000000 00000000 00000000 00000000 00000000
[ 1445.446817] 00000060: 00000004 00000004 00000753 00000753 00000000 00000000 00000000 00000000
[ 1445.447564] 00000080: 00000000 00000000
[ 1445.461665] Rebooting in 60 seconds..
[ 1506.636160] SMP: failed to stop secondary CPUs

4.信息分析

分析几处log信息：

1.如下

[ 1236.201435] Modules linked in: hardlockup_test(O)

从上面可以看出发生hard lockup的ko模块，当然如果不是生成的ko而是直接编译到内核里面去的，这个地方将不显示出问题的ko模块。

2.如下

[ 1273.525872] Task dump for CPU 4:
[ 1273.526179] hg-thread       R  running task        0  6280      2 0x00000012

从这部分log，可以看出CPU 4核出问题，并且出现问题的线程名是 hg-thread，线程id是6280.

根据分析的结果，找到出现问题的线程，然后排查线程里面的操作，找到问题原因。