2018-12-28 11:31:14 +03:00
// SPDX-License-Identifier: GPL-2.0
2018-12-28 11:29:45 +03:00
/*
2020-12-22 23:00:32 +03:00
* This file contains common KASAN code .
2018-12-28 11:29:45 +03:00
*
* Copyright ( c ) 2014 Samsung Electronics Co . , Ltd .
* Author : Andrey Ryabinin < ryabinin . a . a @ gmail . com >
*
* Some code borrowed from https : //github.com/xairy/kasan-prototype by
* Andrey Konovalov < andreyknvl @ gmail . com >
*/
# include <linux/export.h>
# include <linux/init.h>
# include <linux/kasan.h>
# include <linux/kernel.h>
# include <linux/linkage.h>
# include <linux/memblock.h>
# include <linux/memory.h>
# include <linux/mm.h>
# include <linux/module.h>
# include <linux/printk.h>
# include <linux/sched.h>
kasan: record and report more information
Record and report more information to help us find the cause of the bug
and to help us correlate the error with other system events.
This patch adds recording and showing CPU number and timestamp at
allocation and free (controlled by CONFIG_KASAN_EXTRA_INFO). The
timestamps in the report use the same format and source as printk.
Error occurrence timestamp is already implicit in the printk log, and CPU
number is already shown by dump_stack_lvl, so there is no need to add it.
In order to record CPU number and timestamp at allocation and free,
corresponding members need to be added to the relevant data structures,
which will lead to increased memory consumption.
In Generic KASAN, members are added to struct kasan_track. Since in most
cases, alloc meta is stored in the redzone and free meta is stored in the
object or the redzone, memory consumption will not increase much.
In SW_TAGS KASAN and HW_TAGS KASAN, members are added to struct
kasan_stack_ring_entry. Memory consumption increases as the size of
struct kasan_stack_ring_entry increases (this part of the memory is
allocated by memblock), but since this is configurable, it is up to the
user to choose.
Link: https://lkml.kernel.org/r/VI1P193MB0752BD991325D10E4AB1913599BDA@VI1P193MB0752.EURP193.PROD.OUTLOOK.COM
Signed-off-by: Juntong Deng <juntong.deng@outlook.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-11-28 00:17:31 +03:00
# include <linux/sched/clock.h>
2018-12-28 11:29:45 +03:00
# include <linux/sched/task_stack.h>
# include <linux/slab.h>
2023-11-20 20:47:13 +03:00
# include <linux/stackdepot.h>
2018-12-28 11:29:45 +03:00
# include <linux/stacktrace.h>
# include <linux/string.h>
# include <linux/types.h>
# include <linux/bug.h>
# include "kasan.h"
# include "../slab.h"
2022-09-06 00:05:38 +03:00
struct slab * kasan_addr_to_slab ( const void * addr )
{
if ( virt_addr_valid ( addr ) )
return virt_to_slab ( addr ) ;
return NULL ;
}
2023-11-20 20:47:13 +03:00
depot_stack_handle_t kasan_save_stack ( gfp_t flags , depot_flags_t depot_flags )
2018-12-28 11:29:45 +03:00
{
unsigned long entries [ KASAN_STACK_DEPTH ] ;
2019-04-25 12:45:02 +03:00
unsigned int nr_entries ;
2018-12-28 11:29:45 +03:00
2019-04-25 12:45:02 +03:00
nr_entries = stack_trace_save ( entries , ARRAY_SIZE ( entries ) , 0 ) ;
2023-11-20 20:47:13 +03:00
return stack_depot_save_flags ( entries , nr_entries , flags , depot_flags ) ;
2018-12-28 11:29:45 +03:00
}
2020-08-07 09:24:39 +03:00
void kasan_set_track ( struct kasan_track * track , gfp_t flags )
2018-12-28 11:29:45 +03:00
{
kasan: record and report more information
Record and report more information to help us find the cause of the bug
and to help us correlate the error with other system events.
This patch adds recording and showing CPU number and timestamp at
allocation and free (controlled by CONFIG_KASAN_EXTRA_INFO). The
timestamps in the report use the same format and source as printk.
Error occurrence timestamp is already implicit in the printk log, and CPU
number is already shown by dump_stack_lvl, so there is no need to add it.
In order to record CPU number and timestamp at allocation and free,
corresponding members need to be added to the relevant data structures,
which will lead to increased memory consumption.
In Generic KASAN, members are added to struct kasan_track. Since in most
cases, alloc meta is stored in the redzone and free meta is stored in the
object or the redzone, memory consumption will not increase much.
In SW_TAGS KASAN and HW_TAGS KASAN, members are added to struct
kasan_stack_ring_entry. Memory consumption increases as the size of
struct kasan_stack_ring_entry increases (this part of the memory is
allocated by memblock), but since this is configurable, it is up to the
user to choose.
Link: https://lkml.kernel.org/r/VI1P193MB0752BD991325D10E4AB1913599BDA@VI1P193MB0752.EURP193.PROD.OUTLOOK.COM
Signed-off-by: Juntong Deng <juntong.deng@outlook.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-11-28 00:17:31 +03:00
# ifdef CONFIG_KASAN_EXTRA_INFO
u32 cpu = raw_smp_processor_id ( ) ;
u64 ts_nsec = local_clock ( ) ;
track - > cpu = cpu ;
track - > timestamp = ts_nsec > > 3 ;
# endif /* CONFIG_KASAN_EXTRA_INFO */
2018-12-28 11:29:45 +03:00
track - > pid = current - > pid ;
2023-11-20 20:47:19 +03:00
track - > stack = kasan_save_stack ( flags ,
STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_GET ) ;
2018-12-28 11:29:45 +03:00
}
2020-12-22 23:00:56 +03:00
# if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
2018-12-28 11:29:45 +03:00
void kasan_enable_current ( void )
{
current - > kasan_depth + + ;
}
2021-06-29 05:34:33 +03:00
EXPORT_SYMBOL ( kasan_enable_current ) ;
2018-12-28 11:29:45 +03:00
void kasan_disable_current ( void )
{
current - > kasan_depth - - ;
}
2021-06-29 05:34:33 +03:00
EXPORT_SYMBOL ( kasan_disable_current ) ;
2020-12-22 23:00:56 +03:00
# endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
2018-12-28 11:29:45 +03:00
2020-12-22 23:03:10 +03:00
void __kasan_unpoison_range ( const void * address , size_t size )
2020-12-22 23:00:21 +03:00
{
2021-04-30 08:59:59 +03:00
kasan_unpoison ( address , size , false ) ;
2020-12-22 23:00:21 +03:00
}
2021-04-17 01:46:00 +03:00
# ifdef CONFIG_KASAN_STACK
2018-12-28 11:29:45 +03:00
/* Unpoison the entire stack for a task. */
void kasan_unpoison_task_stack ( struct task_struct * task )
{
2020-12-22 23:02:49 +03:00
void * base = task_stack_page ( task ) ;
2021-04-30 08:59:59 +03:00
kasan_unpoison ( base , THREAD_SIZE , false ) ;
2018-12-28 11:29:45 +03:00
}
/* Unpoison the stack for the current task beyond a watermark sp value. */
asmlinkage void kasan_unpoison_task_stack_below ( const void * watermark )
{
/*
* Calculate the task stack base address . Avoid using ' current '
* because this function is called by early resume code which hasn ' t
* yet set up the percpu register ( % gs ) .
*/
void * base = ( void * ) ( ( unsigned long ) watermark & ~ ( THREAD_SIZE - 1 ) ) ;
2021-04-30 08:59:59 +03:00
kasan_unpoison ( base , watermark - base , false ) ;
2018-12-28 11:29:45 +03:00
}
2020-12-22 23:02:42 +03:00
# endif /* CONFIG_KASAN_STACK */
2018-12-28 11:29:45 +03:00
kasan: allow sampling page_alloc allocations for HW_TAGS
As Hardware Tag-Based KASAN is intended to be used in production, its
performance impact is crucial. As page_alloc allocations tend to be big,
tagging and checking all such allocations can introduce a significant
slowdown.
Add two new boot parameters that allow to alleviate that slowdown:
- kasan.page_alloc.sample, which makes Hardware Tag-Based KASAN tag only
every Nth page_alloc allocation with the order configured by the second
added parameter (default: tag every such allocation).
- kasan.page_alloc.sample.order, which makes sampling enabled by the first
parameter only affect page_alloc allocations with the order equal or
greater than the specified value (default: 3, see below).
The exact performance improvement caused by using the new parameters
depends on their values and the applied workload.
The chosen default value for kasan.page_alloc.sample.order is 3, which
matches both PAGE_ALLOC_COSTLY_ORDER and SKB_FRAG_PAGE_ORDER. This is
done for two reasons:
1. PAGE_ALLOC_COSTLY_ORDER is "the order at which allocations are deemed
costly to service", which corresponds to the idea that only large and
thus costly allocations are supposed to sampled.
2. One of the workloads targeted by this patch is a benchmark that sends
a large amount of data over a local loopback connection. Most multi-page
data allocations in the networking subsystem have the order of
SKB_FRAG_PAGE_ORDER (or PAGE_ALLOC_COSTLY_ORDER).
When running a local loopback test on a testing MTE-enabled device in sync
mode, enabling Hardware Tag-Based KASAN introduces a ~50% slowdown.
Applying this patch and setting kasan.page_alloc.sampling to a value
higher than 1 allows to lower the slowdown. The performance improvement
saturates around the sampling interval value of 10 with the default
sampling page order of 3. This lowers the slowdown to ~20%. The slowdown
in real scenarios involving the network will likely be better.
Enabling page_alloc sampling has a downside: KASAN misses bad accesses to
a page_alloc allocation that has not been tagged. This lowers the value
of KASAN as a security mitigation.
However, based on measuring the number of page_alloc allocations of
different orders during boot in a test build, sampling with the default
kasan.page_alloc.sample.order value affects only ~7% of allocations. The
rest ~93% of allocations are still checked deterministically.
Link: https://lkml.kernel.org/r/129da0614123bb85ed4dd61ae30842b2dd7c903f.1671471846.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Mark Brand <markbrand@google.com>
Cc: Peter Collingbourne <pcc@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2022-12-19 21:09:18 +03:00
bool __kasan_unpoison_pages ( struct page * page , unsigned int order , bool init )
2018-12-28 11:29:45 +03:00
{
2018-12-28 11:30:57 +03:00
u8 tag ;
unsigned long i ;
2018-12-28 11:30:50 +03:00
if ( unlikely ( PageHighMem ( page ) ) )
kasan: allow sampling page_alloc allocations for HW_TAGS
As Hardware Tag-Based KASAN is intended to be used in production, its
performance impact is crucial. As page_alloc allocations tend to be big,
tagging and checking all such allocations can introduce a significant
slowdown.
Add two new boot parameters that allow to alleviate that slowdown:
- kasan.page_alloc.sample, which makes Hardware Tag-Based KASAN tag only
every Nth page_alloc allocation with the order configured by the second
added parameter (default: tag every such allocation).
- kasan.page_alloc.sample.order, which makes sampling enabled by the first
parameter only affect page_alloc allocations with the order equal or
greater than the specified value (default: 3, see below).
The exact performance improvement caused by using the new parameters
depends on their values and the applied workload.
The chosen default value for kasan.page_alloc.sample.order is 3, which
matches both PAGE_ALLOC_COSTLY_ORDER and SKB_FRAG_PAGE_ORDER. This is
done for two reasons:
1. PAGE_ALLOC_COSTLY_ORDER is "the order at which allocations are deemed
costly to service", which corresponds to the idea that only large and
thus costly allocations are supposed to sampled.
2. One of the workloads targeted by this patch is a benchmark that sends
a large amount of data over a local loopback connection. Most multi-page
data allocations in the networking subsystem have the order of
SKB_FRAG_PAGE_ORDER (or PAGE_ALLOC_COSTLY_ORDER).
When running a local loopback test on a testing MTE-enabled device in sync
mode, enabling Hardware Tag-Based KASAN introduces a ~50% slowdown.
Applying this patch and setting kasan.page_alloc.sampling to a value
higher than 1 allows to lower the slowdown. The performance improvement
saturates around the sampling interval value of 10 with the default
sampling page order of 3. This lowers the slowdown to ~20%. The slowdown
in real scenarios involving the network will likely be better.
Enabling page_alloc sampling has a downside: KASAN misses bad accesses to
a page_alloc allocation that has not been tagged. This lowers the value
of KASAN as a security mitigation.
However, based on measuring the number of page_alloc allocations of
different orders during boot in a test build, sampling with the default
kasan.page_alloc.sample.order value affects only ~7% of allocations. The
rest ~93% of allocations are still checked deterministically.
Link: https://lkml.kernel.org/r/129da0614123bb85ed4dd61ae30842b2dd7c903f.1671471846.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Mark Brand <markbrand@google.com>
Cc: Peter Collingbourne <pcc@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2022-12-19 21:09:18 +03:00
return false ;
if ( ! kasan_sample_page_alloc ( order ) )
return false ;
2018-12-28 11:30:57 +03:00
2021-02-24 23:05:05 +03:00
tag = kasan_random_tag ( ) ;
2022-06-10 18:21:38 +03:00
kasan_unpoison ( set_tag ( page_address ( page ) , tag ) ,
PAGE_SIZE < < order , init ) ;
2018-12-28 11:30:57 +03:00
for ( i = 0 ; i < ( 1 < < order ) ; i + + )
page_kasan_tag_set ( page + i , tag ) ;
kasan: allow sampling page_alloc allocations for HW_TAGS
As Hardware Tag-Based KASAN is intended to be used in production, its
performance impact is crucial. As page_alloc allocations tend to be big,
tagging and checking all such allocations can introduce a significant
slowdown.
Add two new boot parameters that allow to alleviate that slowdown:
- kasan.page_alloc.sample, which makes Hardware Tag-Based KASAN tag only
every Nth page_alloc allocation with the order configured by the second
added parameter (default: tag every such allocation).
- kasan.page_alloc.sample.order, which makes sampling enabled by the first
parameter only affect page_alloc allocations with the order equal or
greater than the specified value (default: 3, see below).
The exact performance improvement caused by using the new parameters
depends on their values and the applied workload.
The chosen default value for kasan.page_alloc.sample.order is 3, which
matches both PAGE_ALLOC_COSTLY_ORDER and SKB_FRAG_PAGE_ORDER. This is
done for two reasons:
1. PAGE_ALLOC_COSTLY_ORDER is "the order at which allocations are deemed
costly to service", which corresponds to the idea that only large and
thus costly allocations are supposed to sampled.
2. One of the workloads targeted by this patch is a benchmark that sends
a large amount of data over a local loopback connection. Most multi-page
data allocations in the networking subsystem have the order of
SKB_FRAG_PAGE_ORDER (or PAGE_ALLOC_COSTLY_ORDER).
When running a local loopback test on a testing MTE-enabled device in sync
mode, enabling Hardware Tag-Based KASAN introduces a ~50% slowdown.
Applying this patch and setting kasan.page_alloc.sampling to a value
higher than 1 allows to lower the slowdown. The performance improvement
saturates around the sampling interval value of 10 with the default
sampling page order of 3. This lowers the slowdown to ~20%. The slowdown
in real scenarios involving the network will likely be better.
Enabling page_alloc sampling has a downside: KASAN misses bad accesses to
a page_alloc allocation that has not been tagged. This lowers the value
of KASAN as a security mitigation.
However, based on measuring the number of page_alloc allocations of
different orders during boot in a test build, sampling with the default
kasan.page_alloc.sample.order value affects only ~7% of allocations. The
rest ~93% of allocations are still checked deterministically.
Link: https://lkml.kernel.org/r/129da0614123bb85ed4dd61ae30842b2dd7c903f.1671471846.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Mark Brand <markbrand@google.com>
Cc: Peter Collingbourne <pcc@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2022-12-19 21:09:18 +03:00
return true ;
2018-12-28 11:29:45 +03:00
}
2021-06-03 02:52:28 +03:00
void __kasan_poison_pages ( struct page * page , unsigned int order , bool init )
2018-12-28 11:29:45 +03:00
{
if ( likely ( ! PageHighMem ( page ) ) )
2021-02-24 23:05:05 +03:00
kasan_poison ( page_address ( page ) , PAGE_SIZE < < order ,
2022-05-13 06:23:08 +03:00
KASAN_PAGE_FREE , init ) ;
2018-12-28 11:29:45 +03:00
}
2021-10-04 16:46:46 +03:00
void __kasan_poison_slab ( struct slab * slab )
2018-12-28 11:29:45 +03:00
{
2021-10-04 16:46:46 +03:00
struct page * page = slab_page ( slab ) ;
2018-12-28 11:30:57 +03:00
unsigned long i ;
2019-09-24 01:34:30 +03:00
for ( i = 0 ; i < compound_nr ( page ) ; i + + )
2018-12-28 11:30:57 +03:00
page_kasan_tag_reset ( page + i ) ;
2021-02-24 23:05:05 +03:00
kasan_poison ( page_address ( page ) , page_size ( page ) ,
2022-05-13 06:23:08 +03:00
KASAN_SLAB_REDZONE , false ) ;
2018-12-28 11:29:45 +03:00
}
2020-12-22 23:03:10 +03:00
void __kasan_unpoison_object_data ( struct kmem_cache * cache , void * object )
2018-12-28 11:29:45 +03:00
{
2021-04-30 08:59:59 +03:00
kasan_unpoison ( object , cache - > object_size , false ) ;
2018-12-28 11:29:45 +03:00
}
2020-12-22 23:03:10 +03:00
void __kasan_poison_object_data ( struct kmem_cache * cache , void * object )
2018-12-28 11:29:45 +03:00
{
2021-02-26 04:20:27 +03:00
kasan_poison ( object , round_up ( cache - > object_size , KASAN_GRANULE_SIZE ) ,
2022-05-13 06:23:08 +03:00
KASAN_SLAB_REDZONE , false ) ;
2018-12-28 11:29:45 +03:00
}
2018-12-28 11:30:50 +03:00
/*
2019-01-09 02:23:18 +03:00
* This function assigns a tag to an object considering the following :
* 1. A cache might have a constructor , which might save a pointer to a slab
* object somewhere ( e . g . in the object itself ) . We preassign a tag for
* each object in caches with constructors during slab creation and reuse
* the same tag each time a particular object is allocated .
* 2. A cache might be SLAB_TYPESAFE_BY_RCU , which means objects can be
* accessed after being freed . We preassign tags for objects in these
* caches as well .
* 3. For SLAB allocator we can ' t preassign tags randomly since the freelist
* is stored as an array of indexes instead of a linked list . Assign tags
* based on objects indexes , so that objects that are next to each other
* get different tags .
2018-12-28 11:30:50 +03:00
*/
2021-02-26 04:20:35 +03:00
static inline u8 assign_tag ( struct kmem_cache * cache ,
const void * object , bool init )
2018-12-28 11:30:50 +03:00
{
2020-12-22 23:03:20 +03:00
if ( IS_ENABLED ( CONFIG_KASAN_GENERIC ) )
return 0xff ;
2019-01-09 02:23:18 +03:00
/*
* If the cache neither has a constructor nor has SLAB_TYPESAFE_BY_RCU
* set , assign a tag when the object is being allocated ( init = = false ) .
*/
2018-12-28 11:30:50 +03:00
if ( ! cache - > ctor & & ! ( cache - > flags & SLAB_TYPESAFE_BY_RCU ) )
2021-02-24 23:05:05 +03:00
return init ? KASAN_TAG_KERNEL : kasan_random_tag ( ) ;
2018-12-28 11:30:50 +03:00
2019-01-09 02:23:18 +03:00
/* For caches that either have a constructor or SLAB_TYPESAFE_BY_RCU: */
2018-12-28 11:30:50 +03:00
# ifdef CONFIG_SLAB
2019-01-09 02:23:18 +03:00
/* For SLAB assign tags based on the object index in the freelist. */
2021-11-02 17:42:04 +03:00
return ( u8 ) obj_to_index ( cache , virt_to_slab ( object ) , ( void * ) object ) ;
2018-12-28 11:30:50 +03:00
# else
2019-01-09 02:23:18 +03:00
/*
* For SLUB assign a random tag during slab creation , otherwise reuse
* the already assigned tag .
*/
2021-02-24 23:05:05 +03:00
return init ? kasan_random_tag ( ) : get_tag ( object ) ;
2018-12-28 11:30:50 +03:00
# endif
}
2020-12-22 23:03:10 +03:00
void * __must_check __kasan_init_slab_obj ( struct kmem_cache * cache ,
2018-12-28 11:31:01 +03:00
const void * object )
2018-12-28 11:29:45 +03:00
{
2022-09-06 00:05:23 +03:00
/* Initialize per-object metadata if it is present. */
2022-09-06 00:05:26 +03:00
if ( kasan_requires_meta ( ) )
2022-09-06 00:05:23 +03:00
kasan_init_object_meta ( cache , object ) ;
2018-12-28 11:29:45 +03:00
2020-12-22 23:03:20 +03:00
/* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */
kasan, mm: optimize kmalloc poisoning
For allocations from kmalloc caches, kasan_kmalloc() always follows
kasan_slab_alloc(). Currenly, both of them unpoison the whole object,
which is unnecessary.
This patch provides separate implementations for both annotations:
kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only
poisons the redzone.
For generic KASAN, the redzone start might not be aligned to
KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts:
kasan_poison_last_granule() poisons the unaligned part, and then
kasan_poison() poisons the rest.
This patch also clarifies alignment guarantees of each of the poisoning
functions and drops the unnecessary round_up() call for redzone_end.
With this change, the early SLUB cache annotation needs to be changed to
kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The
number of poisoned bytes for objects in this cache stays the same, as
kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node).
Link: https://lkml.kernel.org/r/7e3961cb52be380bc412860332063f5f7ce10d13.1612546384.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Peter Collingbourne <pcc@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-02-26 04:19:59 +03:00
object = set_tag ( object , assign_tag ( cache , object , true ) ) ;
2018-12-28 11:30:50 +03:00
2018-12-28 11:29:45 +03:00
return ( void * ) object ;
}
2023-12-20 01:28:53 +03:00
static inline bool poison_slab_object ( struct kmem_cache * cache , void * object ,
unsigned long ip , bool init )
2018-12-28 11:29:45 +03:00
{
2018-12-28 11:30:50 +03:00
void * tagged_object ;
2018-12-28 11:29:45 +03:00
2021-06-29 05:40:42 +03:00
if ( ! kasan_arch_is_ready ( ) )
return false ;
2018-12-28 11:30:50 +03:00
tagged_object = object ;
2020-12-22 23:02:52 +03:00
object = kasan_reset_tag ( object ) ;
2018-12-28 11:30:50 +03:00
2021-02-26 04:19:21 +03:00
if ( is_kfence_address ( object ) )
return false ;
2023-12-20 01:28:53 +03:00
if ( unlikely ( nearest_obj ( cache , virt_to_slab ( object ) , object ) ! = object ) ) {
2022-06-15 09:22:18 +03:00
kasan_report_invalid_free ( tagged_object , ip , KASAN_REPORT_INVALID_FREE ) ;
2018-12-28 11:29:45 +03:00
return true ;
}
2023-12-20 01:28:53 +03:00
/* RCU slabs could be legally used after free within the RCU period. */
2018-12-28 11:29:45 +03:00
if ( unlikely ( cache - > flags & SLAB_TYPESAFE_BY_RCU ) )
return false ;
2021-02-24 23:05:50 +03:00
if ( ! kasan_byte_accessible ( tagged_object ) ) {
2022-06-15 09:22:18 +03:00
kasan_report_invalid_free ( tagged_object , ip , KASAN_REPORT_DOUBLE_FREE ) ;
2018-12-28 11:29:45 +03:00
return true ;
}
2021-02-26 04:20:27 +03:00
kasan_poison ( object , round_up ( cache - > object_size , KASAN_GRANULE_SIZE ) ,
2022-05-13 06:23:08 +03:00
KASAN_SLAB_FREE , init ) ;
2018-12-28 11:29:45 +03:00
2021-02-26 04:20:07 +03:00
if ( kasan_stack_collection_enabled ( ) )
2022-09-06 00:05:34 +03:00
kasan_save_free_info ( cache , tagged_object ) ;
2019-09-24 01:34:13 +03:00
2023-12-20 01:28:53 +03:00
return false ;
2018-12-28 11:29:45 +03:00
}
2021-04-30 09:00:09 +03:00
bool __kasan_slab_free ( struct kmem_cache * cache , void * object ,
unsigned long ip , bool init )
2018-12-28 11:29:45 +03:00
{
2023-12-20 01:28:53 +03:00
bool buggy_object = poison_slab_object ( cache , object , ip , init ) ;
return buggy_object ? true : kasan_quarantine_put ( cache , object ) ;
2018-12-28 11:29:45 +03:00
}
2023-12-20 01:28:48 +03:00
static inline bool check_page_allocation ( void * ptr , unsigned long ip )
2021-02-26 04:20:11 +03:00
{
kasan: fix Oops due to missing calls to kasan_arch_is_ready()
On powerpc64, you can build a kernel with KASAN as soon as you build it
with RADIX MMU support. However if the CPU doesn't have RADIX MMU, KASAN
isn't enabled at init and the following Oops is encountered.
[ 0.000000][ T0] KASAN not enabled as it requires radix!
[ 4.484295][ T26] BUG: Unable to handle kernel data access at 0xc00e000000804a04
[ 4.485270][ T26] Faulting instruction address: 0xc00000000062ec6c
[ 4.485748][ T26] Oops: Kernel access of bad area, sig: 11 [#1]
[ 4.485920][ T26] BE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[ 4.486259][ T26] Modules linked in:
[ 4.486637][ T26] CPU: 0 PID: 26 Comm: kworker/u2:2 Not tainted 6.2.0-rc3-02590-gf8a023b0a805 #249
[ 4.486907][ T26] Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1200 0xf000005 of:SLOF,HEAD pSeries
[ 4.487445][ T26] Workqueue: eval_map_wq .tracer_init_tracefs_work_func
[ 4.488744][ T26] NIP: c00000000062ec6c LR: c00000000062bb84 CTR: c0000000002ebcd0
[ 4.488867][ T26] REGS: c0000000049175c0 TRAP: 0380 Not tainted (6.2.0-rc3-02590-gf8a023b0a805)
[ 4.489028][ T26] MSR: 8000000002009032 <SF,VEC,EE,ME,IR,DR,RI> CR: 44002808 XER: 00000000
[ 4.489584][ T26] CFAR: c00000000062bb80 IRQMASK: 0
[ 4.489584][ T26] GPR00: c0000000005624d4 c000000004917860 c000000001cfc000 1800000000804a04
[ 4.489584][ T26] GPR04: c0000000003a2650 0000000000000cc0 c00000000000d3d8 c00000000000d3d8
[ 4.489584][ T26] GPR08: c0000000049175b0 a80e000000000000 0000000000000000 0000000017d78400
[ 4.489584][ T26] GPR12: 0000000044002204 c000000003790000 c00000000435003c c0000000043f1c40
[ 4.489584][ T26] GPR16: c0000000043f1c68 c0000000043501a0 c000000002106138 c0000000043f1c08
[ 4.489584][ T26] GPR20: c0000000043f1c10 c0000000043f1c20 c000000004146c40 c000000002fdb7f8
[ 4.489584][ T26] GPR24: c000000002fdb834 c000000003685e00 c000000004025030 c000000003522e90
[ 4.489584][ T26] GPR28: 0000000000000cc0 c0000000003a2650 c000000004025020 c000000004025020
[ 4.491201][ T26] NIP [c00000000062ec6c] .kasan_byte_accessible+0xc/0x20
[ 4.491430][ T26] LR [c00000000062bb84] .__kasan_check_byte+0x24/0x90
[ 4.491767][ T26] Call Trace:
[ 4.491941][ T26] [c000000004917860] [c00000000062ae70] .__kasan_kmalloc+0xc0/0x110 (unreliable)
[ 4.492270][ T26] [c0000000049178f0] [c0000000005624d4] .krealloc+0x54/0x1c0
[ 4.492453][ T26] [c000000004917990] [c0000000003a2650] .create_trace_option_files+0x280/0x530
[ 4.492613][ T26] [c000000004917a90] [c000000002050d90] .tracer_init_tracefs_work_func+0x274/0x2c0
[ 4.492771][ T26] [c000000004917b40] [c0000000001f9948] .process_one_work+0x578/0x9f0
[ 4.492927][ T26] [c000000004917c30] [c0000000001f9ebc] .worker_thread+0xfc/0x950
[ 4.493084][ T26] [c000000004917d60] [c00000000020be84] .kthread+0x1a4/0x1b0
[ 4.493232][ T26] [c000000004917e10] [c00000000000d3d8] .ret_from_kernel_thread+0x58/0x60
[ 4.495642][ T26] Code: 60000000 7cc802a6 38a00000 4bfffc78 60000000 7cc802a6 38a00001 4bfffc68 60000000 3d20a80e 7863e8c2 792907c6 <7c6348ae> 20630007 78630fe0 68630001
[ 4.496704][ T26] ---[ end trace 0000000000000000 ]---
The Oops is due to kasan_byte_accessible() not checking the readiness of
KASAN. Add missing call to kasan_arch_is_ready() and bail out when not
ready. The same problem is observed with ____kasan_kfree_large() so fix
it the same.
Also, as KASAN is not available and no shadow area is allocated for linear
memory mapping, there is no point in allocating shadow mem for vmalloc
memory as shown below in /sys/kernel/debug/kernel_page_tables
---[ kasan shadow mem start ]---
0xc00f000000000000-0xc00f00000006ffff 0x00000000040f0000 448K r w pte valid present dirty accessed
0xc00f000000860000-0xc00f00000086ffff 0x000000000ac10000 64K r w pte valid present dirty accessed
0xc00f3ffffffe0000-0xc00f3fffffffffff 0x0000000004d10000 128K r w pte valid present dirty accessed
---[ kasan shadow mem end ]---
So, also verify KASAN readiness before allocating and poisoning
shadow mem for VMAs.
Link: https://lkml.kernel.org/r/150768c55722311699fdcf8f5379e8256749f47d.1674716617.git.christophe.leroy@csgroup.eu
Fixes: 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support")
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Reported-by: Nathan Lynch <nathanl@linux.ibm.com>
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: <stable@vger.kernel.org> [5.19+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-01-26 10:04:47 +03:00
if ( ! kasan_arch_is_ready ( ) )
return false ;
2021-02-26 04:20:11 +03:00
if ( ptr ! = page_address ( virt_to_head_page ( ptr ) ) ) {
2022-06-15 09:22:18 +03:00
kasan_report_invalid_free ( ptr , ip , KASAN_REPORT_INVALID_FREE ) ;
2021-02-26 04:20:11 +03:00
return true ;
}
if ( ! kasan_byte_accessible ( ptr ) ) {
2022-06-15 09:22:18 +03:00
kasan_report_invalid_free ( ptr , ip , KASAN_REPORT_DOUBLE_FREE ) ;
2021-02-26 04:20:11 +03:00
return true ;
}
return false ;
}
void __kasan_kfree_large ( void * ptr , unsigned long ip )
{
2023-12-20 01:28:48 +03:00
check_page_allocation ( ptr , ip ) ;
/* The object will be poisoned by kasan_poison_pages(). */
2021-02-26 04:20:11 +03:00
}
2023-12-20 01:28:56 +03:00
static inline void unpoison_slab_object ( struct kmem_cache * cache , void * object ,
gfp_t flags , bool init )
{
/*
* Unpoison the whole object . For kmalloc ( ) allocations ,
* poison_kmalloc_redzone ( ) will do precise poisoning .
*/
kasan_unpoison ( object , cache - > object_size , init ) ;
/* Save alloc info (if possible) for non-kmalloc() allocations. */
if ( kasan_stack_collection_enabled ( ) & & ! is_kmalloc_cache ( cache ) )
kasan_save_alloc_info ( cache , object , flags ) ;
}
kasan, mm: optimize kmalloc poisoning
For allocations from kmalloc caches, kasan_kmalloc() always follows
kasan_slab_alloc(). Currenly, both of them unpoison the whole object,
which is unnecessary.
This patch provides separate implementations for both annotations:
kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only
poisons the redzone.
For generic KASAN, the redzone start might not be aligned to
KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts:
kasan_poison_last_granule() poisons the unaligned part, and then
kasan_poison() poisons the rest.
This patch also clarifies alignment guarantees of each of the poisoning
functions and drops the unnecessary round_up() call for redzone_end.
With this change, the early SLUB cache annotation needs to be changed to
kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The
number of poisoned bytes for objects in this cache stays the same, as
kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node).
Link: https://lkml.kernel.org/r/7e3961cb52be380bc412860332063f5f7ce10d13.1612546384.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Peter Collingbourne <pcc@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-02-26 04:19:59 +03:00
void * __must_check __kasan_slab_alloc ( struct kmem_cache * cache ,
2021-04-30 09:00:06 +03:00
void * object , gfp_t flags , bool init )
kasan, mm: optimize kmalloc poisoning
For allocations from kmalloc caches, kasan_kmalloc() always follows
kasan_slab_alloc(). Currenly, both of them unpoison the whole object,
which is unnecessary.
This patch provides separate implementations for both annotations:
kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only
poisons the redzone.
For generic KASAN, the redzone start might not be aligned to
KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts:
kasan_poison_last_granule() poisons the unaligned part, and then
kasan_poison() poisons the rest.
This patch also clarifies alignment guarantees of each of the poisoning
functions and drops the unnecessary round_up() call for redzone_end.
With this change, the early SLUB cache annotation needs to be changed to
kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The
number of poisoned bytes for objects in this cache stays the same, as
kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node).
Link: https://lkml.kernel.org/r/7e3961cb52be380bc412860332063f5f7ce10d13.1612546384.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Peter Collingbourne <pcc@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-02-26 04:19:59 +03:00
{
u8 tag ;
void * tagged_object ;
if ( gfpflags_allow_blocking ( flags ) )
kasan_quarantine_reduce ( ) ;
if ( unlikely ( object = = NULL ) )
return NULL ;
if ( is_kfence_address ( object ) )
return ( void * ) object ;
/*
* Generate and assign random tag for tag - based modes .
* Tag is ignored in set_tag ( ) for the generic mode .
*/
tag = assign_tag ( cache , object , false ) ;
tagged_object = set_tag ( object , tag ) ;
2023-12-20 01:28:56 +03:00
/* Unpoison the object and save alloc info for non-kmalloc() allocations. */
unpoison_slab_object ( cache , tagged_object , flags , init ) ;
kasan, mm: optimize kmalloc poisoning
For allocations from kmalloc caches, kasan_kmalloc() always follows
kasan_slab_alloc(). Currenly, both of them unpoison the whole object,
which is unnecessary.
This patch provides separate implementations for both annotations:
kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only
poisons the redzone.
For generic KASAN, the redzone start might not be aligned to
KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts:
kasan_poison_last_granule() poisons the unaligned part, and then
kasan_poison() poisons the rest.
This patch also clarifies alignment guarantees of each of the poisoning
functions and drops the unnecessary round_up() call for redzone_end.
With this change, the early SLUB cache annotation needs to be changed to
kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The
number of poisoned bytes for objects in this cache stays the same, as
kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node).
Link: https://lkml.kernel.org/r/7e3961cb52be380bc412860332063f5f7ce10d13.1612546384.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Peter Collingbourne <pcc@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-02-26 04:19:59 +03:00
return tagged_object ;
}
2023-12-20 01:28:54 +03:00
static inline void poison_kmalloc_redzone ( struct kmem_cache * cache ,
2021-02-26 04:20:35 +03:00
const void * object , size_t size , gfp_t flags )
2018-12-28 11:29:45 +03:00
{
unsigned long redzone_start ;
unsigned long redzone_end ;
kasan, mm: optimize kmalloc poisoning
For allocations from kmalloc caches, kasan_kmalloc() always follows
kasan_slab_alloc(). Currenly, both of them unpoison the whole object,
which is unnecessary.
This patch provides separate implementations for both annotations:
kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only
poisons the redzone.
For generic KASAN, the redzone start might not be aligned to
KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts:
kasan_poison_last_granule() poisons the unaligned part, and then
kasan_poison() poisons the rest.
This patch also clarifies alignment guarantees of each of the poisoning
functions and drops the unnecessary round_up() call for redzone_end.
With this change, the early SLUB cache annotation needs to be changed to
kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The
number of poisoned bytes for objects in this cache stays the same, as
kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node).
Link: https://lkml.kernel.org/r/7e3961cb52be380bc412860332063f5f7ce10d13.1612546384.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Peter Collingbourne <pcc@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-02-26 04:19:59 +03:00
/*
* The redzone has byte - level precision for the generic mode .
* Partially poison the last object granule to cover the unaligned
* part of the redzone .
*/
if ( IS_ENABLED ( CONFIG_KASAN_GENERIC ) )
kasan_poison_last_granule ( ( void * ) object , size ) ;
/* Poison the aligned part of the redzone. */
2018-12-28 11:29:45 +03:00
redzone_start = round_up ( ( unsigned long ) ( object + size ) ,
2020-12-22 23:00:24 +03:00
KASAN_GRANULE_SIZE ) ;
2021-02-26 04:20:27 +03:00
redzone_end = round_up ( ( unsigned long ) ( object + cache - > object_size ) ,
KASAN_GRANULE_SIZE ) ;
2021-02-24 23:05:05 +03:00
kasan_poison ( ( void * ) redzone_start , redzone_end - redzone_start ,
2022-05-13 06:23:08 +03:00
KASAN_SLAB_REDZONE , false ) ;
2018-12-28 11:29:45 +03:00
kasan, mm: optimize kmalloc poisoning
For allocations from kmalloc caches, kasan_kmalloc() always follows
kasan_slab_alloc(). Currenly, both of them unpoison the whole object,
which is unnecessary.
This patch provides separate implementations for both annotations:
kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only
poisons the redzone.
For generic KASAN, the redzone start might not be aligned to
KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts:
kasan_poison_last_granule() poisons the unaligned part, and then
kasan_poison() poisons the rest.
This patch also clarifies alignment guarantees of each of the poisoning
functions and drops the unnecessary round_up() call for redzone_end.
With this change, the early SLUB cache annotation needs to be changed to
kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The
number of poisoned bytes for objects in this cache stays the same, as
kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node).
Link: https://lkml.kernel.org/r/7e3961cb52be380bc412860332063f5f7ce10d13.1612546384.git.andreyknvl@google.com
Signed-off-by: Andrey Konovalov <andreyknvl@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Branislav Rankov <Branislav.Rankov@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Evgenii Stepanov <eugenis@google.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Peter Collingbourne <pcc@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-02-26 04:19:59 +03:00
/*
* Save alloc info ( if possible ) for kmalloc ( ) allocations .
* This also rewrites the alloc info when called from kasan_krealloc ( ) .
*/
2023-01-04 09:06:05 +03:00
if ( kasan_stack_collection_enabled ( ) & & is_kmalloc_cache ( cache ) )
2022-09-06 00:05:19 +03:00
kasan_save_alloc_info ( cache , ( void * ) object , flags ) ;
2018-12-28 11:29:45 +03:00
2019-02-21 09:19:01 +03:00
}
2020-12-22 23:03:10 +03:00
void * __must_check __kasan_kmalloc ( struct kmem_cache * cache , const void * object ,
size_t size , gfp_t flags )
2019-01-09 02:23:18 +03:00
{
2023-12-20 01:28:54 +03:00
if ( gfpflags_allow_blocking ( flags ) )
kasan_quarantine_reduce ( ) ;
if ( unlikely ( object = = NULL ) )
return NULL ;
if ( is_kfence_address ( kasan_reset_tag ( object ) ) )
return ( void * ) object ;
/* The object has already been unpoisoned by kasan_slab_alloc(). */
poison_kmalloc_redzone ( cache , object , size , flags ) ;
/* Keep the tag that was set by kasan_slab_alloc(). */
return ( void * ) object ;
2019-01-09 02:23:18 +03:00
}
2020-12-22 23:03:10 +03:00
EXPORT_SYMBOL ( __kasan_kmalloc ) ;
2018-12-28 11:29:45 +03:00
2023-12-20 01:28:55 +03:00
static inline void poison_kmalloc_large_redzone ( const void * ptr , size_t size ,
2018-12-28 11:31:01 +03:00
gfp_t flags )
2018-12-28 11:29:45 +03:00
{
unsigned long redzone_start ;
unsigned long redzone_end ;
2021-02-26 04:20:03 +03:00
/*
* The redzone has byte - level precision for the generic mode .
* Partially poison the last object granule to cover the unaligned
* part of the redzone .
*/
if ( IS_ENABLED ( CONFIG_KASAN_GENERIC ) )
kasan_poison_last_granule ( ptr , size ) ;
/* Poison the aligned part of the redzone. */
2023-12-20 01:28:55 +03:00
redzone_start = round_up ( ( unsigned long ) ( ptr + size ) , KASAN_GRANULE_SIZE ) ;
2021-02-26 04:20:03 +03:00
redzone_end = ( unsigned long ) ptr + page_size ( virt_to_page ( ptr ) ) ;
2021-02-24 23:05:05 +03:00
kasan_poison ( ( void * ) redzone_start , redzone_end - redzone_start ,
2021-04-30 08:59:59 +03:00
KASAN_PAGE_REDZONE , false ) ;
2023-12-20 01:28:55 +03:00
}
2018-12-28 11:29:45 +03:00
2023-12-20 01:28:55 +03:00
void * __must_check __kasan_kmalloc_large ( const void * ptr , size_t size ,
gfp_t flags )
{
if ( gfpflags_allow_blocking ( flags ) )
kasan_quarantine_reduce ( ) ;
if ( unlikely ( ptr = = NULL ) )
return NULL ;
/* The object has already been unpoisoned by kasan_unpoison_pages(). */
poison_kmalloc_large_redzone ( ptr , size , flags ) ;
/* Keep the tag that was set by alloc_pages(). */
2018-12-28 11:29:45 +03:00
return ( void * ) ptr ;
}
2020-12-22 23:03:10 +03:00
void * __must_check __kasan_krealloc ( const void * object , size_t size , gfp_t flags )
2018-12-28 11:29:45 +03:00
{
2021-10-04 16:46:46 +03:00
struct slab * slab ;
2018-12-28 11:29:45 +03:00
2023-12-20 01:28:55 +03:00
if ( gfpflags_allow_blocking ( flags ) )
kasan_quarantine_reduce ( ) ;
2018-12-28 11:29:45 +03:00
if ( unlikely ( object = = ZERO_SIZE_PTR ) )
return ( void * ) object ;
2023-12-20 01:28:54 +03:00
if ( is_kfence_address ( kasan_reset_tag ( object ) ) )
return ( void * ) object ;
2021-02-26 04:20:23 +03:00
/*
* Unpoison the object ' s data .
* Part of it might already have been unpoisoned , but it ' s unknown
* how big that part is .
*/
2021-04-30 08:59:59 +03:00
kasan_unpoison ( object , size , false ) ;
2021-02-26 04:20:23 +03:00
2021-10-04 16:46:46 +03:00
slab = virt_to_slab ( object ) ;
2018-12-28 11:29:45 +03:00
2021-02-26 04:20:23 +03:00
/* Piggy-back on kmalloc() instrumentation to poison the redzone. */
2021-10-04 16:46:46 +03:00
if ( unlikely ( ! slab ) )
2023-12-20 01:28:55 +03:00
poison_kmalloc_large_redzone ( object , size , flags ) ;
else
2023-12-20 01:28:54 +03:00
poison_kmalloc_redzone ( slab - > slab_cache , object , size , flags ) ;
2023-12-20 01:28:55 +03:00
return ( void * ) object ;
2018-12-28 11:29:45 +03:00
}
2023-12-20 01:28:50 +03:00
bool __kasan_mempool_poison_pages ( struct page * page , unsigned int order ,
unsigned long ip )
{
unsigned long * ptr ;
if ( unlikely ( PageHighMem ( page ) ) )
return true ;
/* Bail out if allocation was excluded due to sampling. */
if ( ! IS_ENABLED ( CONFIG_KASAN_GENERIC ) & &
page_kasan_tag ( page ) = = KASAN_TAG_KERNEL )
return true ;
ptr = page_address ( page ) ;
if ( check_page_allocation ( ptr , ip ) )
return false ;
kasan_poison ( ptr , PAGE_SIZE < < order , KASAN_PAGE_FREE , false ) ;
return true ;
}
2023-12-20 01:28:51 +03:00
void __kasan_mempool_unpoison_pages ( struct page * page , unsigned int order ,
unsigned long ip )
{
__kasan_unpoison_pages ( page , order , false ) ;
}
2023-12-20 01:28:48 +03:00
bool __kasan_mempool_poison_object ( void * ptr , unsigned long ip )
2023-12-20 01:28:46 +03:00
{
2023-12-20 01:28:52 +03:00
struct folio * folio = virt_to_folio ( ptr ) ;
struct slab * slab ;
2023-12-20 01:28:46 +03:00
/*
2023-12-20 01:28:52 +03:00
* This function can be called for large kmalloc allocation that get
* their memory from page_alloc . Thus , the folio might not be a slab .
2023-12-20 01:28:46 +03:00
*/
if ( unlikely ( ! folio_test_slab ( folio ) ) ) {
2023-12-20 01:28:48 +03:00
if ( check_page_allocation ( ptr , ip ) )
return false ;
2023-12-20 01:28:46 +03:00
kasan_poison ( ptr , folio_size ( folio ) , KASAN_PAGE_FREE , false ) ;
2023-12-20 01:28:48 +03:00
return true ;
2023-12-20 01:28:46 +03:00
}
2023-12-20 01:28:52 +03:00
slab = folio_slab ( folio ) ;
2023-12-20 01:28:53 +03:00
return ! poison_slab_object ( slab - > slab_cache , ptr , ip , false ) ;
2023-12-20 01:28:46 +03:00
}
2023-12-20 01:28:49 +03:00
void __kasan_mempool_unpoison_object ( void * ptr , size_t size , unsigned long ip )
{
2023-12-20 01:28:56 +03:00
struct slab * slab ;
gfp_t flags = 0 ; /* Might be executing under a lock. */
if ( is_kfence_address ( kasan_reset_tag ( ptr ) ) )
return ;
slab = virt_to_slab ( ptr ) ;
/*
* This function can be called for large kmalloc allocation that get
* their memory from page_alloc .
*/
if ( unlikely ( ! slab ) ) {
kasan_unpoison ( ptr , size , false ) ;
poison_kmalloc_large_redzone ( ptr , size , flags ) ;
return ;
}
/* Unpoison the object and save alloc info for non-kmalloc() allocations. */
unpoison_slab_object ( slab - > slab_cache , ptr , size , flags ) ;
/* Poison the redzone and save alloc info for kmalloc() allocations. */
if ( is_kmalloc_cache ( slab - > slab_cache ) )
poison_kmalloc_redzone ( slab - > slab_cache , ptr , size , flags ) ;
2023-12-20 01:28:49 +03:00
}
2021-02-24 23:05:50 +03:00
bool __kasan_check_byte ( const void * address , unsigned long ip )
{
if ( ! kasan_byte_accessible ( address ) ) {
kasan: use internal prototypes matching gcc-13 builtins
gcc-13 warns about function definitions for builtin interfaces that have a
different prototype, e.g.:
In file included from kasan_test.c:31:
kasan.h:574:6: error: conflicting types for built-in function '__asan_register_globals'; expected 'void(void *, long int)' [-Werror=builtin-declaration-mismatch]
574 | void __asan_register_globals(struct kasan_global *globals, size_t size);
kasan.h:577:6: error: conflicting types for built-in function '__asan_alloca_poison'; expected 'void(void *, long int)' [-Werror=builtin-declaration-mismatch]
577 | void __asan_alloca_poison(unsigned long addr, size_t size);
kasan.h:580:6: error: conflicting types for built-in function '__asan_load1'; expected 'void(void *)' [-Werror=builtin-declaration-mismatch]
580 | void __asan_load1(unsigned long addr);
kasan.h:581:6: error: conflicting types for built-in function '__asan_store1'; expected 'void(void *)' [-Werror=builtin-declaration-mismatch]
581 | void __asan_store1(unsigned long addr);
kasan.h:643:6: error: conflicting types for built-in function '__hwasan_tag_memory'; expected 'void(void *, unsigned char, long int)' [-Werror=builtin-declaration-mismatch]
643 | void __hwasan_tag_memory(unsigned long addr, u8 tag, unsigned long size);
The two problems are:
- Addresses are passes as 'unsigned long' in the kernel, but gcc-13
expects a 'void *'.
- sizes meant to use a signed ssize_t rather than size_t.
Change all the prototypes to match these. Using 'void *' consistently for
addresses gets rid of a couple of type casts, so push that down to the
leaf functions where possible.
This now passes all randconfig builds on arm, arm64 and x86, but I have
not tested it on the other architectures that support kasan, since they
tend to fail randconfig builds in other ways. This might fail if any of
the 32-bit architectures expect a 'long' instead of 'int' for the size
argument.
The __asan_allocas_unpoison() function prototype is somewhat weird, since
it uses a pointer for 'stack_top' and an size_t for 'stack_bottom'. This
looks like it is meant to be 'addr' and 'size' like the others, but the
implementation clearly treats them as 'top' and 'bottom'.
Link: https://lkml.kernel.org/r/20230509145735.9263-2-arnd@kernel.org
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Marco Elver <elver@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-05-09 17:57:21 +03:00
kasan_report ( address , 1 , false , ip ) ;
2021-02-24 23:05:50 +03:00
return false ;
}
return true ;
}