mirrors/bird
mirrors/bird
0
0
Fork 0
mirror of https://gitlab.nic.cz/labs/bird.git synced 2024-11-13 06:38:43 +00:00
Code Releases Activity
794a4eefa1
bird/sysdep/unix/alloc.c

138 lines
3.1 KiB
C
Raw Normal View History

Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
/*
* BIRD Internet Routing Daemon -- Raw allocation
*
* (c) 2020 Maria Matejka <mq@ucw.cz>
*
* Can be freely distributed and used under the terms of the GNU GPL.
*/
#include "nest/bird.h"
#include "lib/resource.h"
#include <stdlib.h>
#include <unistd.h>
Keeping un-unmmappable pages until they can be reused On Linux, munmap() may fail with ENOMEM when virtual memory is too fragmented. Working this around by just keeping such blocks for future use.
2021-11-11 15:25:59 +00:00
#include <stdatomic.h>
#include <errno.h>
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
#ifdef HAVE_MMAP
#include <sys/mman.h>
#endif
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
long page_size = 0;
Multipage allocation We can also quite simply allocate bigger blocks. Anyway, we need these blocks to be aligned to their size which needs one mmap() two times bigger and then two munmap()s returning the unaligned parts. The user can specify -B <N> on startup when <N> is the exponent of 2, setting the block size to 2^N. On most systems, N is 12, anyway if you know that your configuration is going to eat gigabytes of RAM, you are almost forced to raise your block size as you may easily get into memory fragmentation issues or you have to raise your maximum mapping count, e.g. "sysctl vm.max_map_count=(number)".
2021-09-08 09:29:49 +00:00
_Bool alloc_multipage = 0;
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
Keeping un-unmmappable pages until they can be reused On Linux, munmap() may fail with ENOMEM when virtual memory is too fragmented. Working this around by just keeping such blocks for future use.
2021-11-11 15:25:59 +00:00
static _Atomic int global_page_list_not_empty;
static list global_page_list;
static _Atomic int global_page_spinlock;
#define GLOBAL_PAGE_SPIN_LOCK for (int v = 0; !atomic_compare_exchange_weak_explicit(&global_page_spinlock, &v, 1, memory_order_acq_rel, memory_order_acquire); v = 0)
#define GLOBAL_PAGE_SPIN_UNLOCK do { int v = 1; ASSERT_DIE(atomic_compare_exchange_strong_explicit(&global_page_spinlock, &v, 0, memory_order_acq_rel, memory_order_acquire)); } while (0)
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
#ifdef HAVE_MMAP
static _Bool use_fake = 0;
#else
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
static _Bool use_fake = 1;
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
#endif
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
void resource_sys_init(void)
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
{
#ifdef HAVE_MMAP
Keeping un-unmmappable pages until they can be reused On Linux, munmap() may fail with ENOMEM when virtual memory is too fragmented. Working this around by just keeping such blocks for future use.
2021-11-11 15:25:59 +00:00
init_list(&global_page_list);
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
if (!(page_size = sysconf(_SC_PAGESIZE)))
die("System page size must be non-zero");
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
if ((u64_popcount(page_size) > 1) || (page_size > 16384))
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
#endif
Keeping un-unmmappable pages until they can be reused On Linux, munmap() may fail with ENOMEM when virtual memory is too fragmented. Working this around by just keeping such blocks for future use.
2021-11-11 15:25:59 +00:00
{
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
/* Too big or strange page, use the aligned allocator instead */
page_size = 4096;
use_fake = 1;
}
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
}
void *
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
alloc_sys_page(void)
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
{
#ifdef HAVE_MMAP
if (!use_fake)
{
Keeping un-unmmappable pages until they can be reused On Linux, munmap() may fail with ENOMEM when virtual memory is too fragmented. Working this around by just keeping such blocks for future use.
2021-11-11 15:25:59 +00:00
if (atomic_load_explicit(&global_page_list_not_empty, memory_order_relaxed))
{
GLOBAL_PAGE_SPIN_LOCK;
if (!EMPTY_LIST(global_page_list))
{
node *ret = HEAD(global_page_list);
rem_node(ret);
if (EMPTY_LIST(global_page_list))
atomic_store_explicit(&global_page_list_not_empty, 0, memory_order_relaxed);
GLOBAL_PAGE_SPIN_UNLOCK;
memset(ret, 0, sizeof(node));
return (void *) ret;
}
GLOBAL_PAGE_SPIN_UNLOCK;
}
Multipage allocation We can also quite simply allocate bigger blocks. Anyway, we need these blocks to be aligned to their size which needs one mmap() two times bigger and then two munmap()s returning the unaligned parts. The user can specify -B <N> on startup when <N> is the exponent of 2, setting the block size to 2^N. On most systems, N is 12, anyway if you know that your configuration is going to eat gigabytes of RAM, you are almost forced to raise your block size as you may easily get into memory fragmentation issues or you have to raise your maximum mapping count, e.g. "sysctl vm.max_map_count=(number)".
2021-09-08 09:29:49 +00:00
if (alloc_multipage)
{
void *big = mmap(NULL, page_size * 2, PROT_WRITE | PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (big == MAP_FAILED)
bug("mmap(%lu) failed: %m", page_size);
uintptr_t offset = ((uintptr_t) big) % page_size;
if (offset)
{
void *ret = big + page_size - offset;
munmap(big, page_size - offset);
munmap(ret + page_size, offset);
return ret;
}
else
{
munmap(big + page_size, page_size);
return big;
}
}
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
void *ret = mmap(NULL, page_size, PROT_WRITE | PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
if (ret == MAP_FAILED)
bug("mmap(%lu) failed: %m", page_size);
Multipage allocation We can also quite simply allocate bigger blocks. Anyway, we need these blocks to be aligned to their size which needs one mmap() two times bigger and then two munmap()s returning the unaligned parts. The user can specify -B <N> on startup when <N> is the exponent of 2, setting the block size to 2^N. On most systems, N is 12, anyway if you know that your configuration is going to eat gigabytes of RAM, you are almost forced to raise your block size as you may easily get into memory fragmentation issues or you have to raise your maximum mapping count, e.g. "sysctl vm.max_map_count=(number)".
2021-09-08 09:29:49 +00:00
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
return ret;
}
else
#endif
{
void *ret = aligned_alloc(page_size, page_size);
if (!ret)
bug("aligned_alloc(%lu) failed", page_size);
return ret;
}
}
void
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
free_sys_page(void *ptr)
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
{
#ifdef HAVE_MMAP
if (!use_fake)
{
Bound allocated pages to resource pools with page caches to avoid unnecessary syscalls
2021-08-31 22:46:46 +00:00
if (munmap(ptr, page_size) < 0)
Keeping un-unmmappable pages until they can be reused On Linux, munmap() may fail with ENOMEM when virtual memory is too fragmented. Working this around by just keeping such blocks for future use.
2021-11-11 15:25:59 +00:00
#ifdef ENOMEM
if (errno == ENOMEM)
{
memset(ptr, 0, page_size);
GLOBAL_PAGE_SPIN_LOCK;
add_tail(&global_page_list, (node *) ptr);
atomic_store_explicit(&global_page_list_not_empty, 1, memory_order_relaxed);
GLOBAL_PAGE_SPIN_UNLOCK;
}
else
#endif
bug("munmap(%p) failed: %m", ptr);
Slab: head now uses bitmask for used/free nodes info instead of lists From now, there are no auxiliary pointers stored in the free slab nodes. This led to strange debugging problems if use-after-free happened in slab-allocated structures, especially if the structure's first member is a next pointer. This also reduces the memory needed by 1 pointer per allocated object. OTOH, we now rely on pages being aligned to their size's multiple, which is quite common anyway.
2020-07-21 22:09:15 +00:00
}
else
#endif
free(ptr);
}
Copy Permalink