eBPF on cgroups

the long story short - they are stored in array effective and in list progs in cgroup->bpf

Below I will try to explain boring and dirty details

cgroups

This article says:

hierarchy: a set of cgroups arranged in a tree

so we need to find roots and then just traverse this trees. Roots have type cgroup_root and stored in cgroup_hierarchy_idr (synced with mutex cgroup_mutex). As usually linux lies - lets compare content of  /proc/cgroups:

#subsys_name hierarchy num_cgroups enabled

cpuset 6 1 1

cpu 5 1 1

cpuacct 5 1 1

blkio 4 1 1

memory 2 148 1

devices 9 99 1

freezer 10 1 1

net_cls 7 1 1

perf_event 8 1 1

net_prio 7 1 1

hugetlb 3 1 1

pids 11 103 1

rdma 12 1 1

with what cgroup roots actually located on this machine:

[0]  at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144

[1] systemd at 0xffff8fd6816ea000 flags 4 hierarchy_id 1 nr_cgrps 145 real_cnt 144

[2]  at 0xffff8fd68297a000 flags 0 hierarchy_id 2 nr_cgrps 148 real_cnt 147

[3]  at 0xffff8fd68297c000 flags 0 hierarchy_id 3 nr_cgrps 1 real_cnt 0

[4]  at 0xffff8fd682978000 flags 0 hierarchy_id 4 nr_cgrps 1 real_cnt 0

[5]  at 0xffff8fd68297e000 flags 0 hierarchy_id 5 nr_cgrps 1 real_cnt 0

[6]  at 0xffff8fd6854c8000 flags 0 hierarchy_id 6 nr_cgrps 1 real_cnt 0

[7]  at 0xffff8fd6854ce000 flags 0 hierarchy_id 7 nr_cgrps 1 real_cnt 0

[8]  at 0xffff8fd6854ca000 flags 0 hierarchy_id 8 nr_cgrps 1 real_cnt 0

[9]  at 0xffff8fd6854cc000 flags 0 hierarchy_id 9 nr_cgrps 99 real_cnt 98

[10]  at 0xffff8fd685e16000 flags 0 hierarchy_id 10 nr_cgrps 1 real_cnt 0

[11]  at 0xffff8fd685e12000 flags 0 hierarchy_id 11 nr_cgrps 103 real_cnt 102

[12]  at 0xffff8fd685e14000 flags 0 hierarchy_id 12 nr_cgrps 1 real_cnt 0

can you find in /proc/cgroups roots with hierarchy ID 0 and 1?

How to traverse this tree? It starts in field cgrp->self and we can use  functions css_next_descendant_pre/css_next_descendant_post etc. Strictly speaking they return pointer to cgroup_subsys_state but this is first field self  in cgroup so casting is safe

eBPF

eBPF programs are stored in prog_idr (synced with spinlock_t prog_idr_lock). Lets see what we have:

sudo ./lkmem -d -c -B ~/krnl/curr ~/krnl/System.map-5.11.0-40-generic

prog_idr at 0xffffffff8e9be540: 23

 [0] prog 0xffff9f8e809a7000 id 31 len 123 jited_len 555

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05c2254

 [1] prog 0xffff9f8e809bf000 id 32 len 1824 jited_len 8195

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05ba9e0

 [2] prog 0xffff9f8e80905000 id 33 len 1343 jited_len 6186

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc00612b0

 [3] prog 0xffff9f8e809c7000 id 34 len 1682 jited_len 7822

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc02d6040

 [4] prog 0xffff9f8e809a9000 id 35 len 1209 jited_len 5510

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05be370

 [5] prog 0xffff9f8e809cf000 id 36 len 1397 jited_len 6396

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05c46d8

 [6] prog 0xffff9f8e809d7000 id 37 len 1223 jited_len 5578

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05c7108

 [7] prog 0xffff9f8e80055000 id 38 len 267 jited_len 1237

  type: 5 BPF_PROG_TYPE_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc0059254

 [8] prog 0xffff9f8e8005d000 id 39 len 247 jited_len 1116

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05ca90c

 [9] prog 0xffff9f8e80115000 id 40 len 217 jited_len 994

  type: 5 BPF_PROG_TYPE_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05cc2c0

 [10] prog 0xffff9f8e8011d000 id 41 len 744 jited_len 3405

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05ce098

 [11] prog 0xffff9f8e809df000 id 42 len 633 jited_len 2701

  type: 5 BPF_PROG_TYPE_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05d0234

 [12] prog 0xffff9f8e808f4000 id 43 len 492 jited_len 2233

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05d262c

 [13] prog 0xffff9f8e809bb000 id 44 len 68 jited_len 312

  type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05d41f4

 [14] prog 0xffff9f8e809f1000 id 55 len 2 jited_len 15

  type: 1 BPF_PROG_TYPE_SOCKET_FILTER

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05e49f4

 [15] prog 0xffff9f8e8003b000 id 89 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc04fab24

 [16] prog 0xffff9f8e80037000 id 90 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc0561468

 [17] prog 0xffff9f8e80048000 id 91 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc0563828

 [18] prog 0xffff9f8e8004f000 id 92 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc0565594

 [19] prog 0xffff9f8e80051000 id 93 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc056747c

 [20] prog 0xffff9f8e80053000 id 94 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc056923c

 [21] prog 0xffff9f8e8012f000 id 95 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc058e308

 [22] prog 0xffff9f8e80131000 id 96 len 8 jited_len 54

  type: 8 BPF_PROG_TYPE_CGROUP_SKB

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

   bpf_func: 0xffffffffc05900dc

Each eBPF program stored in struct bpf_prog. But wait - we can have several eBPF programs connected to the same event, right? Yeah, so for such cases there is another struct - bpf_prog_array. kernel devs were so generous that they even gave us function bpf_prog_array_length. Unfortunately it is not exported

putting it all together 

Now we know how to

• find and traverse cgroups
• where eBPF stored in each cgroup
• can enum ePBF programs from bpf_prog_array

Lets see what we have:

sudo ./lkmem -d -c -g ~/krnl/curr ~/krnl/System.map-5.11.0-40-generic

[0]  at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144

 child 9:

 cgroup at 0xffff8fd6824aa000 serial_nr 80 flags 0 level 2

 cgroup BPF:

  BPF_CGROUP_INET_INGRESS: 0xffff8fd68d361300 cnt 1 flags 0

  [0] prog 0xffff9f8e80037000 id 90 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc0561468

  BPF_CGROUP_INET_EGRESS: 0xffff8fd687cf5f00 cnt 1 flags 0

  [0] prog 0xffff9f8e8003b000 id 89 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc04fab24

 child 24:

 cgroup at 0xffff8fd68864b000 serial_nr 286 flags 0 level 2

 cgroup BPF:

  BPF_CGROUP_INET_INGRESS: 0xffff8fd6858fea40 cnt 1 flags 0

  [0] prog 0xffff9f8e80053000 id 94 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc056923c

  BPF_CGROUP_INET_EGRESS: 0xffff8fd5d1fb6fc0 cnt 1 flags 0

  [0] prog 0xffff9f8e80051000 id 93 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc056747c

 child 44:

 cgroup at 0xffff8fd68310c000 serial_nr 596 flags 0 level 2

  BPF_CGROUP_INET_INGRESS: 0xffff8fd5c5be97c0 cnt 1 flags 0

  [0] prog 0xffff9f8e80131000 id 96 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc05900dc

  BPF_CGROUP_INET_EGRESS: 0xffff8fd687cf7300 cnt 1 flags 0

  [0] prog 0xffff9f8e8012f000 id 95 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc058e308

 child 94:

 cgroup at 0xffff8fd5834d7000 serial_nr 1136 flags 0 level 2

 cgroup BPF:

  BPF_CGROUP_INET_INGRESS: 0xffff8fd5c7cd1f00 cnt 1 flags 0

  [0] prog 0xffff9f8e8004f000 id 92 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc0565594

  BPF_CGROUP_INET_EGRESS: 0xffff8fd68555fac0 cnt 1 flags 0

  [0] prog 0xffff9f8e80048000 id 91 type 8 len 8 jited_len 54

   bpf_func: 0xffffffffc0563828

What can this useful information give us? Well, there is function cgroup_bpf_prog_detach. And suddenly your eBPF program may stop receiving messages

Link to source code