Out-of-bounds Read Affecting kernel-doc package, versions <0:4.18.0-553.8.1.el8_10

Note: Versions mentioned in the description apply only to the upstream kernel-doc package and not the kernel-doc package as distributed by RHEL. See How to fix? for RHEL:8 relevant fixed versions and status.

In the Linux kernel, the following vulnerability has been resolved:

perf/core: Bail out early if the request AUX area is out of bound

When perf-record with a large AUX area, e.g 4GB, it fails with:

#perf record -C 0 -m ,4G -e arm_spe_0// -- sleep 1
failed to mmap with 12 (Cannot allocate memory)

and it reveals a WARNING with __alloc_pages():

------------[ cut here ]------------
WARNING: CPU: 44 PID: 17573 at mm/page_alloc.c:5568 __alloc_pages+0x1ec/0x248
Call trace:
 __alloc_pages+0x1ec/0x248
 __kmalloc_large_node+0xc0/0x1f8
 __kmalloc_node+0x134/0x1e8
 rb_alloc_aux+0xe0/0x298
 perf_mmap+0x440/0x660
 mmap_region+0x308/0x8a8
 do_mmap+0x3c0/0x528
 vm_mmap_pgoff+0xf4/0x1b8
 ksys_mmap_pgoff+0x18c/0x218
 __arm64_sys_mmap+0x38/0x58
 invoke_syscall+0x50/0x128
 el0_svc_common.constprop.0+0x58/0x188
 do_el0_svc+0x34/0x50
 el0_svc+0x34/0x108
 el0t_64_sync_handler+0xb8/0xc0
 el0t_64_sync+0x1a4/0x1a8

'rb->aux_pages' allocated by kcalloc() is a pointer array which is used to maintains AUX trace pages. The allocated page for this array is physically contiguous (and virtually contiguous) with an order of 0..MAX_ORDER. If the size of pointer array crosses the limitation set by MAX_ORDER, it reveals a WARNING.

So bail out early with -ENOMEM if the request AUX area is out of bound, e.g.:

#perf record -C 0 -m ,4G -e arm_spe_0// -- sleep 1
failed to mmap with 12 (Cannot allocate memory)

• https://access.redhat.com/security/cve/CVE-2023-52835
• https://git.kernel.org/stable/c/1a2a4202c60fcdffbf04f259002ce9bff39edece
• https://git.kernel.org/stable/c/2424410f94a94d91230ced094062d859714c984a
• https://git.kernel.org/stable/c/2e905e608e38cf7f8dcddcf8a6036e91a78444cb
• https://git.kernel.org/stable/c/54aee5f15b83437f23b2b2469bcf21bdd9823916
• https://git.kernel.org/stable/c/788c0b3442ead737008934947730a6d1ff703734
• https://git.kernel.org/stable/c/8c504f615d7ed60ae035c51d0c789137ced6797f
• https://git.kernel.org/stable/c/9ce4e87a8efd37c85766ec08b15e885cab08553a
• https://git.kernel.org/stable/c/fd0df3f8719201dbe61a4d39083d5aecd705399a