четверг, 25 ноября 2021 г.

eBPF on cgroups

the long story short - they are stored in array effective and in list progs in cgroup->bpf
Below I will try to explain boring and dirty details

cgroups

This article says:
hierarchy: a set of cgroups arranged in a tree
so we need to find roots and then just traverse this trees. Roots have type cgroup_root and stored in cgroup_hierarchy_idr (synced with mutex cgroup_mutex). As usually linux lies - lets compare content of  /proc/cgroups:
#subsys_name hierarchy num_cgroups enabled
cpuset 6 1 1
cpu 5 1 1
cpuacct 5 1 1
blkio 4 1 1
memory 2 148 1
devices 9 99 1
freezer 10 1 1
net_cls 7 1 1
perf_event 8 1 1
net_prio 7 1 1
hugetlb 3 1 1
pids 11 103 1
rdma 12 1 1

with what cgroup roots actually located on this machine:

[0]  at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144
[1] systemd at 0xffff8fd6816ea000 flags 4 hierarchy_id 1 nr_cgrps 145 real_cnt 144
[2]  at 0xffff8fd68297a000 flags 0 hierarchy_id 2 nr_cgrps 148 real_cnt 147
[3]  at 0xffff8fd68297c000 flags 0 hierarchy_id 3 nr_cgrps 1 real_cnt 0
[4]  at 0xffff8fd682978000 flags 0 hierarchy_id 4 nr_cgrps 1 real_cnt 0
[5]  at 0xffff8fd68297e000 flags 0 hierarchy_id 5 nr_cgrps 1 real_cnt 0
[6]  at 0xffff8fd6854c8000 flags 0 hierarchy_id 6 nr_cgrps 1 real_cnt 0
[7]  at 0xffff8fd6854ce000 flags 0 hierarchy_id 7 nr_cgrps 1 real_cnt 0
[8]  at 0xffff8fd6854ca000 flags 0 hierarchy_id 8 nr_cgrps 1 real_cnt 0
[9]  at 0xffff8fd6854cc000 flags 0 hierarchy_id 9 nr_cgrps 99 real_cnt 98
[10]  at 0xffff8fd685e16000 flags 0 hierarchy_id 10 nr_cgrps 1 real_cnt 0
[11]  at 0xffff8fd685e12000 flags 0 hierarchy_id 11 nr_cgrps 103 real_cnt 102
[12]  at 0xffff8fd685e14000 flags 0 hierarchy_id 12 nr_cgrps 1 real_cnt 0

can you find in /proc/cgroups roots with hierarchy ID 0 and 1?

How to traverse this tree? It starts in field cgrp->self and we can use  functions css_next_descendant_pre/css_next_descendant_post etc. Strictly speaking they return pointer to cgroup_subsys_state but this is first field self  in cgroup so casting is safe

eBPF

eBPF programs are stored in prog_idr (synced with spinlock_t prog_idr_lock). Lets see what we have:

sudo ./lkmem -d -c -B ~/krnl/curr ~/krnl/System.map-5.11.0-40-generic
prog_idr at 0xffffffff8e9be540: 23
 [0] prog 0xffff9f8e809a7000 id 31 len 123 jited_len 555
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05c2254
 [1] prog 0xffff9f8e809bf000 id 32 len 1824 jited_len 8195
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05ba9e0
 [2] prog 0xffff9f8e80905000 id 33 len 1343 jited_len 6186
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc00612b0
 [3] prog 0xffff9f8e809c7000 id 34 len 1682 jited_len 7822
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc02d6040
 [4] prog 0xffff9f8e809a9000 id 35 len 1209 jited_len 5510
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05be370
 [5] prog 0xffff9f8e809cf000 id 36 len 1397 jited_len 6396
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05c46d8
 [6] prog 0xffff9f8e809d7000 id 37 len 1223 jited_len 5578
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05c7108
 [7] prog 0xffff9f8e80055000 id 38 len 267 jited_len 1237
  type: 5 BPF_PROG_TYPE_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc0059254
 [8] prog 0xffff9f8e8005d000 id 39 len 247 jited_len 1116
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05ca90c
 [9] prog 0xffff9f8e80115000 id 40 len 217 jited_len 994
  type: 5 BPF_PROG_TYPE_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05cc2c0
 [10] prog 0xffff9f8e8011d000 id 41 len 744 jited_len 3405
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05ce098
 [11] prog 0xffff9f8e809df000 id 42 len 633 jited_len 2701
  type: 5 BPF_PROG_TYPE_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05d0234
 [12] prog 0xffff9f8e808f4000 id 43 len 492 jited_len 2233
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05d262c
 [13] prog 0xffff9f8e809bb000 id 44 len 68 jited_len 312
  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05d41f4
 [14] prog 0xffff9f8e809f1000 id 55 len 2 jited_len 15
  type: 1 BPF_PROG_TYPE_SOCKET_FILTER
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05e49f4
 [15] prog 0xffff9f8e8003b000 id 89 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc04fab24
 [16] prog 0xffff9f8e80037000 id 90 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc0561468
 [17] prog 0xffff9f8e80048000 id 91 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc0563828
 [18] prog 0xffff9f8e8004f000 id 92 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc0565594
 [19] prog 0xffff9f8e80051000 id 93 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc056747c
 [20] prog 0xffff9f8e80053000 id 94 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc056923c
 [21] prog 0xffff9f8e8012f000 id 95 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc058e308
 [22] prog 0xffff9f8e80131000 id 96 len 8 jited_len 54
  type: 8 BPF_PROG_TYPE_CGROUP_SKB
  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS
   bpf_func: 0xffffffffc05900dc


Each eBPF program stored in struct bpf_prog. But wait - we can have several eBPF programs connected to the same event, right? Yeah, so for such cases there is another struct - bpf_prog_array. kernel devs were so generous that they even gave us function bpf_prog_array_length. Unfortunately it is not exported

putting it all together 

Now we know how to
  • find and traverse cgroups
  • where eBPF stored in each cgroup
  • can enum ePBF programs from bpf_prog_array
Lets see what we have:

sudo ./lkmem -d -c -g ~/krnl/curr ~/krnl/System.map-5.11.0-40-generic
[0]  at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144
 child 9:
 cgroup at 0xffff8fd6824aa000 serial_nr 80 flags 0 level 2
 cgroup BPF:
  BPF_CGROUP_INET_INGRESS: 0xffff8fd68d361300 cnt 1 flags 0
  [0] prog 0xffff9f8e80037000 id 90 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc0561468
  BPF_CGROUP_INET_EGRESS: 0xffff8fd687cf5f00 cnt 1 flags 0
  [0] prog 0xffff9f8e8003b000 id 89 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc04fab24
 child 24:
 cgroup at 0xffff8fd68864b000 serial_nr 286 flags 0 level 2
 cgroup BPF:
  BPF_CGROUP_INET_INGRESS: 0xffff8fd6858fea40 cnt 1 flags 0
  [0] prog 0xffff9f8e80053000 id 94 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc056923c
  BPF_CGROUP_INET_EGRESS: 0xffff8fd5d1fb6fc0 cnt 1 flags 0
  [0] prog 0xffff9f8e80051000 id 93 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc056747c
 child 44:
 cgroup at 0xffff8fd68310c000 serial_nr 596 flags 0 level 2
  BPF_CGROUP_INET_INGRESS: 0xffff8fd5c5be97c0 cnt 1 flags 0
  [0] prog 0xffff9f8e80131000 id 96 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc05900dc
  BPF_CGROUP_INET_EGRESS: 0xffff8fd687cf7300 cnt 1 flags 0
  [0] prog 0xffff9f8e8012f000 id 95 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc058e308
 child 94:
 cgroup at 0xffff8fd5834d7000 serial_nr 1136 flags 0 level 2
 cgroup BPF:
  BPF_CGROUP_INET_INGRESS: 0xffff8fd5c7cd1f00 cnt 1 flags 0
  [0] prog 0xffff9f8e8004f000 id 92 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc0565594
  BPF_CGROUP_INET_EGRESS: 0xffff8fd68555fac0 cnt 1 flags 0
  [0] prog 0xffff9f8e80048000 id 91 type 8 len 8 jited_len 54
   bpf_func: 0xffffffffc0563828


What can this useful information give us? Well, there is function cgroup_bpf_prog_detach. And suddenly your eBPF program may stop receiving messages

Link to source code

Комментариев нет:

Отправить комментарий