linux/kernel/hung_task.c, branch v6.14

treewide: const qualify ctl_tables where applicable

2025-01-28T12:48:37Z

Add the const qualifier to all the ctl_tables in the tree except for watchdog_hardlockup_sysctl, memory_allocation_profiling_sysctls, loadpin_sysctl_table and the ones calling register_net_sysctl (./net, drivers/inifiniband dirs). These are special cases as they use a registration function with a non-const qualified ctl_table argument or modify the arrays before passing them on to the registration function. Constifying ctl_table structs will prevent the modification of proc_handler function pointers as the arrays would reside in .rodata. This is made possible after commit 78eb4ea25cd5 ("sysctl: treewide: constify the ctl_table argument of proc_handlers") constified all the proc_handlers. Created this by running an spatch followed by a sed command: Spatch: virtual patch @ depends on !(file in "net") disable optional_qualifier @ identifier table_name != { watchdog_hardlockup_sysctl, iwcm_ctl_table, ucma_ctl_table, memory_allocation_profiling_sysctls, loadpin_sysctl_table }; @@ + const struct ctl_table table_name [] = { ... }; sed: sed --in-place \ -e "s/struct ctl_table .table = &uts_kern/const struct ctl_table *table = \&uts_kern/" \ kernel/utsname_sysctl.c Reviewed-by: Song Liu Acked-by: Steven Rostedt (Google) # for kernel/trace/ Reviewed-by: Martin K. Petersen # SCSI Reviewed-by: Darrick J. Wong # xfs Acked-by: Jani Nikula Acked-by: Corey Minyard Acked-by: Wei Liu Acked-by: Thomas Gleixner Reviewed-by: Bill O'Donnell Acked-by: Baoquan He Acked-by: Ashutosh Dixit Acked-by: Anna Schumaker Signed-off-by: Joel Granados

hung_task: add task->flags, blocked by coredump to log

2025-01-25T06:47:24Z

Resending this patch as I haven't received feedback on my initial submission https://lore.kernel.org/all/20241204182953.10854-1-oxana@cloudflare.com/ For the processes which are terminated abnormally the kernel can provide a coredump if enabled. When the coredump is performed, the process and all its threads are put into the D state (TASK_UNINTERRUPTIBLE | TASK_FREEZABLE). On the other hand, we have kernel thread khungtaskd which monitors the processes in the D state. If the task stuck in the D state more than kernel.hung_task_timeout_secs, the hung_task alert appears in the kernel log. The higher memory usage of a process, the longer it takes to create coredump, the longer tasks are in the D state. We have hung_task alerts for the processes with memory usage above 10Gb. Although, our kernel.hung_task_timeout_secs is 10 sec when the default is 120 sec. Adding additional information to the log that the task is blocked by coredump will help with monitoring. Another approach might be to completely filter out alerts for such tasks, but in that case we would lose transparency about what is putting pressure on some system resources, e.g. we saw an increase in I/O when coredump occurs due its writing to disk. Additionally, it would be helpful to have task_struct->flags in the log from the function sched_show_task(). Currently it prints task_struct->thread_info->flags, this seems misleading as the line starts with "task:xxxx". [akpm@linux-foundation.org: fix printk control string] Link: https://lkml.kernel.org/r/20250110160328.64947-1-oxana@cloudflare.com Signed-off-by: Oxana Kharitonova Cc: Al Viro Cc: Ben Segall Cc: Christian Brauner Cc: Dietmar Eggemann Cc: Ingo Molnar Cc: Jan Kara Cc: Juri Lelli Cc: Mel Gorman Cc: Peter Zijlstra (Intel) Cc: Steven Rostedt Cc: Valentin Schneider Cc: Vincent Guittot Signed-off-by: Andrew Morton

hung_task: add detect count for hung tasks

2024-11-12T01:17:03Z

Patch series "add detect count for hung tasks", v2. This patchset adds a counter, hung_task_detect_count, to track the number of times hung tasks are detected. IHMO, hung tasks are a critical metric. Currently, we detect them by periodically parsing dmesg. However, this method isn't as user-friendly as using a counter. Sometimes, a short-lived issue with NIC or hard drive can quickly decrease the hung_task_warnings to zero. Without warnings, we must directly access the node to ensure that there are no more hung tasks and that the system has recovered. After all, load average alone cannot provide a clear picture. Once this counter is in place, in a high-density deployment pattern, we plan to set hung_task_timeout_secs to a lower number to improve stability, even though this might result in false positives. And then we can set a time-based threshold: if hung tasks last beyond this duration, we will automatically migrate containers to other nodes. Based on past experience, this approach could help avoid many production disruptions. Moreover, just like other important events such as OOM that already have counters, having a dedicated counter for hung tasks makes sense ;) This patch (of 2): This commit adds a counter, hung_task_detect_count, to track the number of times hung tasks are detected. IHMO, hung tasks are a critical metric. Currently, we detect them by periodically parsing dmesg. However, this method isn't as user-friendly as using a counter. Sometimes, a short-lived issue with NIC or hard drive can quickly decrease the hung_task_warnings to zero. Without warnings, we must directly access the node to ensure that there are no more hung tasks and that the system has recovered. After all, load average alone cannot provide a clear picture. Once this counter is in place, in a high-density deployment pattern, we plan to set hung_task_timeout_secs to a lower number to improve stability, even though this might result in false positives. And then we can set a time-based threshold: if hung tasks last beyond this duration, we will automatically migrate containers to other nodes. Based on past experience, this approach could help avoid many production disruptions. Moreover, just like other important events such as OOM that already have counters, having a dedicated counter for hung tasks makes sense. [ioworker0@gmail.com: proc_doulongvec_minmax instead of proc_dointvec] Link: https://lkml.kernel.org/r/20241101114833.8377-1-ioworker0@gmail.com Link: https://lkml.kernel.org/r/20241027120747.42833-1-ioworker0@gmail.com Link: https://lkml.kernel.org/r/20241027120747.42833-2-ioworker0@gmail.com Signed-off-by: Mingzhe Yang Signed-off-by: Lance Yang Cc: Bang Li Cc: Baolin Wang Cc: David Hildenbrand Cc: Huang Cun Cc: Joel Granados Cc: Joel Granados Cc: John Siddle Cc: Kent Overstreet Cc: Ryan Roberts Cc: Thomas Weißschuh Cc: Yongliang Gao Cc: Zi Yan Signed-off-by: Andrew Morton

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24T18:59:29Z

const qualify the struct ctl_table argument in the proc_handler function signatures. This is a prerequisite to moving the static ctl_table structs into .rodata data which will ensure that proc_handler function pointers cannot be modified. This patch has been generated by the following coccinelle script: ``` virtual patch @r1@ identifier ctl, write, buffer, lenp, ppos; identifier func !~ "appldata_(timer|interval)_handler|sched_(rt|rr)_handler|rds_tcp_skbuf_handler|proc_sctp_do_(hmac_alg|rto_min|rto_max|udp_port|alpha_beta|auth|probe_interval)"; @@ int func( - struct ctl_table *ctl + const struct ctl_table *ctl ,int write, void *buffer, size_t *lenp, loff_t *ppos); @r2@ identifier func, ctl, write, buffer, lenp, ppos; @@ int func( - struct ctl_table *ctl + const struct ctl_table *ctl ,int write, void *buffer, size_t *lenp, loff_t *ppos) { ... } @r3@ identifier func; @@ int func( - struct ctl_table * + const struct ctl_table * ,int , void *, size_t *, loff_t *); @r4@ identifier func, ctl; @@ int func( - struct ctl_table *ctl + const struct ctl_table *ctl ,int , void *, size_t *, loff_t *); @r5@ identifier func, write, buffer, lenp, ppos; @@ int func( - struct ctl_table * + const struct ctl_table * ,int write, void *buffer, size_t *lenp, loff_t *ppos); ``` * Code formatting was adjusted in xfs_sysctl.c to comply with code conventions. The xfs_stats_clear_proc_handler, xfs_panic_mask_proc_handler and xfs_deprecated_dointvec_minmax where adjusted. * The ctl_table argument in proc_watchdog_common was const qualified. This is called from a proc_handler itself and is calling back into another proc_handler, making it necessary to change it as part of the proc_handler migration. Co-developed-by: Thomas Weißschuh Signed-off-by: Thomas Weißschuh Co-developed-by: Joel Granados Signed-off-by: Joel Granados

hung_task: ignore hung_task_warnings when hung_task_panic is enabled

2024-06-25T05:25:10Z

If hung_task_panic is enabled, don't consider the value of hung_task_warnings and display the information of the hung tasks. In some cases, hung_task_panic might not be initially set up, after several hung tasks occur, the hung_task_warnings count reaches zero. If hung_task_panic is set up later, it may not display any helpful hung task info in dmesg, only showing messages like: Kernel panic - not syncing: hung_task: blocked tasks CPU: 3 PID: 58 Comm: khungtaskd Not tainted 6.10.0-rc3 #19 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996) Call Trace: panic+0x2f3/0x320 watchdog+0x2dd/0x510 ? __pfx_watchdog+0x10/0x10 kthread+0xe0/0x110 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2f/0x40 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 Link: https://lkml.kernel.org/r/20240613033159.3446265-1-leonylgao@gmail.com Signed-off-by: Yongliang Gao Reviewed-by: Huang Cun Cc: Joel Granados Cc: John Siddle Cc: Kent Overstreet Signed-off-by: Andrew Morton

kernel misc: Remove the now superfluous sentinel elements from ctl_table array

2024-04-24T07:43:53Z

This commit comes at the tail end of a greater effort to remove the empty elements at the end of the ctl_table arrays (sentinels) which will reduce the overall build time size of the kernel and run time memory bloat by ~64 bytes per sentinel (further information Link : https://lore.kernel.org/all/ZO5Yx5JFogGi%2FcBo@bombadil.infradead.org/) Remove the sentinel from ctl_table arrays. Reduce by one the values used to compare the size of the adjusted arrays. Signed-off-by: Joel Granados

kernel/hung_task.c: export sysctl_hung_task_timeout_secs

2024-03-14T01:22:04Z

needed for thread_with_file; also rare but not unheard of to need this in module code, when blocking on user input. one workaround used by some code is wait_event_interruptible() - but that can be buggy if the outer context isn't expecting unwinding. Signed-off-by: Kent Overstreet Cc: Andrew Morton Cc: fuyuanli

kernel/hung_task.c: set some hung_task.c variables storage-class-specifier to static

2023-04-08T20:45:37Z

smatch reports several warnings kernel/hung_task.c:31:19: warning: symbol 'sysctl_hung_task_check_count' was not declared. Should it be static? kernel/hung_task.c:50:29: warning: symbol 'sysctl_hung_task_check_interval_secs' was not declared. Should it be static? kernel/hung_task.c:52:19: warning: symbol 'sysctl_hung_task_warnings' was not declared. Should it be static? kernel/hung_task.c:75:28: warning: symbol 'sysctl_hung_task_panic' was not declared. Should it be static? These variables are only used in hung_task.c, so they should be static Link: https://lkml.kernel.org/r/20230312164645.471259-1-trix@redhat.com Signed-off-by: Tom Rix Cc: Ben Dooks Cc: fuyuanli Cc: John Ogness Cc: Peter Zijlstra Cc: Petr Mladek Cc: Rafael J. Wysocki Cc: Rasmus Villemoes Signed-off-by: Andrew Morton

hung_task: print message when hung_task_warnings gets down to zero.

2023-02-10T01:03:20Z

It's useful to report it when hung_task_warnings gets down to zero, so that we can know if kernel log was lost or there is no hung task was detected. Link: https://lkml.kernel.org/r/20230201135416.GA6560@didi-ThinkCentre-M920t-N000 Signed-off-by: fuyuanli Reviewed-by: Petr Mladek Cc: Rasmus Villemoes Signed-off-by: Andrew Morton

sched: Fix more TASK_state comparisons

2022-09-30T14:50:39Z

Boris reported hung_task splats after commit 5aec788aeb8e ("sched: Fix TASK_state comparisons"). Upon closer consideration of that change it doesn't only exclude TASK_KILLABLE, but also TASK_IDLE. Update the comment to reflect this fact and add an additional TASK_NOLOAD test to exclude them. Additionally, remove the TASK_FREEZABLE early exit from check_hung_task(), a freezable task is not a frozen task. Fixes: 5aec788aeb8e ("sched: Fix TASK_state comparisons") Reported-by: Borislav Petkov Signed-off-by: Peter Zijlstra (Intel) Tested-by: Borislav Petkov