0
0
mirror of https://gitlab.nic.cz/labs/bird.git synced 2024-12-22 17:51:53 +00:00
Commit Graph

972 Commits

Author SHA1 Message Date
Maria Matejka
ab83bab9d1 Loops track also locking time 2023-04-26 19:34:29 +02:00
Maria Matejka
2ddb34c9d1 IO loops now actually measuring their time (show threads all works) 2023-04-26 19:10:52 +02:00
Maria Matejka
ce7495b49a Refactoring of domains connected to pools 2023-04-25 09:52:28 +02:00
Maria Matejka
19e79eb8ad Fix of failing show threads command 2023-04-24 11:35:05 +02:00
Maria Matejka
db2b67268c Merge commit 'd61505b039bf0aa6697e28b2a4e07907c89ba1fb' into thread-next 2023-04-24 10:40:53 +02:00
Maria Matejka
22f54eaee6 Resource pools are now bound with domains.
Memory allocation is a fragile part of BIRD and we need checking that
everybody is using the resource pools in an appropriate way. To assure
this, all the resource pools are associated with locking domains and
every resource manipulation is thoroughly checked whether the
appropriate locking domain is locked.

With transitive resource manipulation like resource dumping or mass free
operations, domains are locked and unlocked on the go, thus we require
pool domains to have higher order than their parent to allow for this
transitive operations.

Adding pool locking revealed some cases of insecure memory manipulation
and this commit fixes that as well.
2023-04-24 10:33:28 +02:00
Maria Matejka
6230d87c74 Protocols and tables now use the birdloop pools as primary 2023-04-22 21:20:19 +02:00
Maria Matejka
1141ce4e2d Resource pool closing has its dedicated function 2023-04-22 20:49:58 +02:00
Maria Matejka
b3f805ce29 Socket closing has its dedicated function 2023-04-22 20:48:42 +02:00
Luiz Amaral
d61505b039 BSD: IPv4 over IPv6 nexthop support on FreeBSD
The support for IPv4 routes with IPv6 nexthops was implemented in FreeBSD
13.1, this patch allows to import and export such routes from/to kernel.

Minor change from committer.
2023-04-21 02:03:31 +02:00
Maria Matejka
67fe3d096d Updated the version number to not include dash (breaks RPM) 2023-04-18 10:11:07 +02:00
Maria Matejka
9f0ef3045d NEWS and version update 2023-04-18 09:52:12 +02:00
Maria Matejka
787fb56da3 IO: added a specific loop pickup group for BFD; to be done better in future 2023-04-17 13:30:14 +02:00
Maria Matejka
dc75d3e305 Merge commit '231c6385' into thread-next-iface 2023-04-14 14:00:54 +02:00
Maria Matejka
6e2f00ff93 Loop: keep running the same loop for some time if there is work to do 2023-04-06 20:18:04 +02:00
Maria Matejka
836e857b30 Sockets: Unified API for main and other loops
Now sk_open() requires an explicit IO loop to open the socket in. Also
specific functions for socket RX pause / resume are added to allow for
BGP corking.

And last but not least, socket reloop is now synchronous to resolve
weird cases of the target loop stopping before actually picking up the
relooped socket. Now the caller must ensure that both loops are locked
while relooping, and this way all sockets always have their respective
loop.
2023-04-04 17:00:59 +02:00
Maria Matejka
571c4f69bf More efficient IO loop event execution to avoid long loops
If there are lots of loops in a single thread and only some of the loops
are actually active, the other loops are now kept aside and not checked
until they actually get some timers, events or active sockets.

This should help with extreme loads like 100k tables and protocols.

Also ping and loop pickup mechanism was allowing subtle race
conditions. Now properly handling collisions between loop ping and pickup.
2023-04-04 17:00:59 +02:00
Maria Matejka
d9f0f4af7d Resource dumps also write out block addresses 2023-04-04 17:00:59 +02:00
Maria Matejka
d16321686e Misc allocator fixes 2023-04-04 17:00:59 +02:00
Maria Matejka
b8d0ba36e6 Allocator fix: thread local kept pages counter must be thread local 2023-04-04 17:00:58 +02:00
Maria Matejka
98f69aa419 Propagated const through route feed routines 2023-04-04 17:00:58 +02:00
Maria Matejka
958bb84f77 Temporarily switching off time annotations of debug log
The "dump *" commands are totally garbled by the time annotations.
2023-04-04 17:00:58 +02:00
Maria Matejka
b8a230e478 Interface subsystem locking 2023-04-04 17:00:58 +02:00
Ondrej Zajicek
6b38285f58 Net: Replace runtime checks with STATIC_ASSERT() 2023-03-06 11:57:40 +01:00
Maria Matejka
a2fd889a3b Merge commit '0bb04d5390f21b0c96fc4894ba5d5510c541f0ef' into HEAD 2023-02-07 14:27:23 +01:00
Maria Matejka
c354e8f4c1 Interface updates are asynchronous
Instead of propagating interface updates as they are loaded from kernel,
they are enqueued and all the notifications are called from a
protocol-specific event. This change allows to break the locking loop
between protocols and interfaces.

Anyway, this change is based on v2 branch to keep the changes between v2
and v3 smaller.
2023-02-02 15:57:21 +01:00
Maria Matejka
e077d0e770 Moved interface list flush to device protocol cleanup hook.
The interface list must be flushed when device protocol is stopped. This
was done in a hardcoded specific hook inside generic protocol routines.
The cleanup hook was originally used for table reference counting late
cleanup, yet it can be also simply used for prettier interface list flush.
2023-02-02 14:40:00 +01:00
Maria Matejka
3ac628e0f0 Merge v2.0.12 into thread-next 2023-01-24 11:04:28 +01:00
Maria Matejka
02b2a4ecaa Merge commit '3186ffe79714a48542d5ad61a94c81216b522fd0' into thread-next 2023-01-24 09:45:40 +01:00
Maria Matejka
3d96a16ae8 Merge commit '6bb992cb04926895be57dc97e7d569ea15a07db1' into thread-next 2023-01-24 09:44:43 +01:00
Maria Matejka
21b772fd43 Merge commit '928a1cb034e6f9e8edcdd1dc07264cd703e00827' into thread-next 2023-01-23 19:51:24 +01:00
Maria Matejka
758aabd96c Merge commit '7fb23041a52d01754c53ba963e2282e524813364' into thread-next 2023-01-23 19:46:26 +01:00
Ondrej Zajicek
5437104afa NEWS and version update 2023-01-23 02:23:19 +01:00
Maria Matejka
521fec2fdc Merge commit '2b7643e1f8ecb0bd4cf9af4183b4fd53b655d19c' into thread-next 2023-01-22 13:19:22 +01:00
Maria Matejka
21c4c8eafb Merge commit '1e47b9f203aaaad0fb658d40a1670f1d0437f1f8' into thread-next 2023-01-21 23:49:52 +01:00
Maria Matejka
0f9df6d1bd Merge commit 'a80cd4707464cedb526eb72e7704b097af20beb7' into thread-next 2023-01-21 23:46:33 +01:00
Maria Matejka
7a262bc999 Adding forgotten pthread.h #include in io-loop.h (breaks at openbsd 7.1) 2023-01-21 23:42:02 +01:00
Maria Matejka
343628d8c0 Fixed various build problems on FreeBSD and/or CLang 2023-01-20 18:31:57 +01:00
Maria Matejka
1127f19a7a Merge commit '140c534fb81d0e165b7d674e869c646455ed19d1' into thread-next 2023-01-19 18:17:05 +01:00
Maria Matejka
84c298465f Decoupling loops from threads to allow fixed thread count
On large configurations, too many threads would spawn with one thread
per loop. Therefore, threads may now run multiple loops at once. The
thread count is configurable and may be changed during run. All threads
are spawned on startup.

This change helps with memory bloating. BIRD filters need large
temporary memory blocks to store their stack and also memory management
keeps its hot page storage per-thread.

Known bugs:
* Thread autobalancing is not yet implemented.
* Low latency loops are executed together with standard loops.
2023-01-19 11:13:50 +01:00
Maria Matejka
59a5bf18f9 CLI closing fix when its action is run asynchronously.
Some CLI actions, notably "show route", are run by queuing an event
somewhere else. If the user closes the socket, in case such an action is
being executed, the CLI must free the socket immediately from the error
hook but the pool must remain until the asynchronous event finishes and
cleans everything up.
2023-01-19 11:03:31 +01:00
Ondrej Zajicek
804916daa9 Alloc: Minor cleanups
- Fix THP disable on old systems
 - Failed syscalls should use die() instead of bug()
 - Our printf uses %ld for s64 instead of long
2023-01-18 13:40:21 +01:00
Maria Matejka
6bb992cb04 Merge branch 'master' of https://gitlab.nic.cz/labs/bird 2023-01-18 12:33:06 +01:00
Maria Matejka
973aa37e1e Fix memory pre-allocation
When BIRD has no free memory mapped, it allocates several pages in
advance just to be sure that there is some memory available if needed.
This hysteresis tactics works quite well to reduce memory ping-ping with
kernel.

Yet it had a subtle bug: this pre-allocation didn't take a memory
coldlist into account, therefore requesting new pages from kernel even
in cases when there were other pages available. This led to slow memory
bloating.

To demonstrate this behavior fast enough to be seen well, you may:
  * temporarily set the values in sysdep/unix/alloc.c as follows to
    exacerbate the issue:
      #define KEEP_PAGES_MAIN_MAX    4096
      #define KEEP_PAGES_MAIN_MIN    1000
      #define CLEANUP_PAGES_BULK     4096
  * create a config file with several millions of static routes
  * periodically disable all static protocols and then reload config
  * log memory consumption

This should give you a steady growth rate of about 16kB per cycle. If
you don't set the values this high, the issue happens much more slowly,
yet after 14 days of running, you are going to see an OOM kill.

After this fix, pre-allocation uses the memory coldlist to get some hot
pages and the same test as described here gets you a perfectly stable
constant memory consumption (after some initial wobbling).

Thanks to NIX-CZ for reporting and helping to investigate this issue.
Thanks to Santiago for finding the cause in the code.
2023-01-18 09:39:45 +01:00
Ondrej Zajicek
928a1cb034 Alloc: Disable transparent huge pages
The usage pattern implemented in allocator seems to be incompatible with
transparent huge pages, as memory released using madvise(MADV_DONTNEED)
with regular page size and alignment does not seem to trigger demotion
of huge pages back to regular pages, even when significant number of
pages is released. Even if demotion is triggered when system memory
is low, it still breaks memory accounting.
2023-01-17 17:13:50 +01:00
Ondrej Zajicek
f8276812e6 Minor cleanups 2023-01-13 13:32:29 +01:00
Ondrej Zajicek
7fb23041a5 BSD: Add support for kernel route metric
Add support for kernel route metric/priority, exported as krt_metric
attribute, like in Linux. This should also fix issues with overwriting
or removing system routes.
2023-01-13 13:17:46 +01:00
Mike Crute
64a2b7aaa3 Log message before aborting
Log message before aborting due to watchdog timeout. We have to use
async-safe write to debug log, as it is done in signal handler.

Minor changes from committer.
2023-01-12 17:40:53 +01:00
Ondrej Zajicek
2b7643e1f8 BSD: Use ip_mreqn on FreeBSD 12.1+ and OpenBSD 6.9+ 2022-12-18 20:23:46 +01:00
Alexander Chernikov
6825f94570 FreeBSD: use interface index instead of IP address when specifying multicast interface
Minor changes from committer.
2022-12-16 19:37:26 +01:00