iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

arm64: mm: Fix lockless walks with static and dynamic page-table folding

Browse Source 
Lina reports random oopsen originating from the fast GUP code when
16K pages are used with 4-level page-tables, the fourth level being
folded at runtime due to lack of LPA2.

In this configuration, the generic implementation of
p4d_offset_lockless() will return a 'p4d_t *' corresponding to the
'pgd_t' allocated on the stack of the caller, gup_fast_pgd_range().
This is normally fine, but when the fourth level of page-table is folded
at runtime, pud_offset_lockless() will offset from the address of the
'p4d_t' to calculate the address of the PUD in the same page-table page.
This results in a stray stack read when the 'p4d_t' has been allocated
on the stack and can send the walker into the weeds.

Fix the problem by providing our own definition of p4d_offset_lockless()
when CONFIG_PGTABLE_LEVELS <= 4 which returns the real page-table
pointer rather than the address of the local stack variable.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/50360968-13fb-4e6f-8f52-1725b3177215@asahilina.net
Fixes: 0dd4f60a2c76 ("arm64: mm: Add support for folding PUDs at runtime")
Reported-by: Asahi Lina <lina@asahilina.net>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20240725090345.28461-1-will@kernel.org
Signed-off-by: Will Deacon <will@kernel.org>
This commit is contained in:
Will Deacon 2024-07-25 10:03:45 +01:00
parent f3dfcd2545
commit 36639013b3
1 changed files with 22 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
arch/arm64/include/asm/pgtable.h
View File

@ -1065,6 +1065,28 @@ static inline bool pgtable_l5_enabled(void) { return false; }
#define p4d_offset_kimg(dir,addr) ((p4d_t *)dir)
static inline
p4d_t *p4d_offset_lockless_folded(pgd_t *pgdp, pgd_t pgd, unsigned long addr)
{
/*
* With runtime folding of the pud, pud_offset_lockless() passes
* the 'pgd_t *' we return here to p4d_to_folded_pud(), which
* will offset the pointer assuming that it points into
* a page-table page. However, the fast GUP path passes us a
* pgd_t allocated on the stack and so we must use the original
* pointer in 'pgdp' to construct the p4d pointer instead of
* using the generic p4d_offset_lockless() implementation.
*
* Note: reusing the original pointer means that we may
* dereference the same (live) page-table entry multiple times.
* This is safe because it is still only loaded once in the
* context of each level and the CPU guarantees same-address
* read-after-read ordering.
*/
return p4d_offset(pgdp, addr);
}
#define p4d_offset_lockless p4d_offset_lockless_folded
#endif /* CONFIG_PGTABLE_LEVELS > 4 */
#define pgd_ERROR(e) \