hidden executable pages in linux kernel, part 2

In part 1 I've described how memory managed by hardware. Now lets dig into how kernel sees memory. Not surprisingly that we should check the same structures that malicious drivers update while hiding

Modules

List of module structures with head in modules and lock modules_mutex. It has projection on file /proc/modules but sizes in those file are sloppy - function module_total_size calculates total size of driver (including discarded sections!). So we should use only some selected fields:

• on kernel >= 6.4 mem[MOD_TEXT].base & mem[MOD_TEXT].size
• on kernel < 4.5 module_core & core_text_size
• otherwise core_layout.base & core_layout.text_size
  

vmap_area_list

It has projection on file /proc/vmallocinfo and requires root access. Sure sophisticated rootkits can intercept it but that's ok since we use it for cross-scan only

 

False positives

As you can guess not every executable page belongs to some driver - there are couple exceptions

bpf_verifier_ops

Lets dissect some typical ebpf spyware. It sets up uprobes on

• SSL_read_ex
• SSL_read
• SSL_write_ex
• SSL_write
• gotls_write_register
  
• gotls_read_register
• gotls_exit_read_register

and uses bpf functions probe_read_user & probe_read_user_str to steal data and map_update_elem & ringbuf_submit to store data in bpf maps

How we can mitigate this?

Official way is to use LSM - function __sys_bpf calls security_bpf so we could register with security_add_hooks LSM hook with index bpf. This effectively prevents loading of ebpf program and sometimes is not what we want - for example in case of honeypots there is high chance that usermode program just will exits after failed ebpf program loading and you can't monitor which connections it will try to establish

Another way - is to patch bpf_func_proto for selected functions, like I did. However this is brutal method and affects all ebpf programs (I still believe that some is not spyware, he-he)

Luckily there is way to blind only some types of ebpf programs - method get_func_proto in bpf_verifier_ops. I made PoC to blind aforementioned 4 functions for BPF_PROG_TYPE_TRACING & BPF_PROG_TYPE_KPROBE only

Now we have another problem - how to check integrity of bpf_verifier_ops? I've also add this check in my lkcd. Example of output when PoC ublind is loaded looks like:

[24] type BPF_PROG_TYPE_TRACING at 0xffffffffc1357720 - ublind!s_trace_patched
  get_func_proto: 0xffffffffc13551e0 - ublind!my_func_proto
  is_valid_access: 0xffffffffaee24e20 - kernel!tracing_prog_is_valid_access



ebpf map as communication channel

Recently I've done small research to repurpose overvalued ebpf into something useful and even achieved some modest results. It seems that at least you can use ebpf maps in your old-school native drivers without writing single line of code for ebpf progs. You can ask me - c'mon, there are tons of ways to communicate with driver under linux, just to name few:

• ioctls
• read from driver/write to driver + possible employ polling
• file in kernfs
• futex in shared memory
  
• io_uring
  
• netlink like auditd or XFRM do

etc

Let's look at typical situation - your IT crew was bitten by violent adherent of the totalitarian sect of rust ebpf Witnesses and as a consequence you now have hundreds of ebpfs (and no single person who know how this pile of spaghetti code works)

Probably now is much better to integrate new drivers with ebpf, right? Considering that ebpf progs have very serious limitations (like you can't read content of linked lists like raw_notifier_head with right locking) it would be very nice to produce output of your EDR drivers directly to ebpf maps

So I made simple POC to show how you can do it. Source of driver & userland test program

Lets dive into gory details and see what other non-obvious advantages can be obtained from such heretical crossbreeding

dumping ebpf kind_ops

Very funny article about (im)possible future of ebpf. Given that right now 8 small BPF scripts with only 7 opcodes occupy 1Mb whole kernel on ebpf will require exabytes of RAM, he-he

Anyway there is another case of info hiding in linux - mentioned in article struct_ops has type btf_kind_operations and cannot be found in include (as usually). So I add today dumping of it in my lkcd. Sample of output:

PoC to blind pamspy

Lets disasm jit code from this spyware:

 [8] prog 0xffffb02dc0133000 id 160 len 46 jited_len 215 aux 0xffff8ccb58fab400 used_maps 1 used_btf 0 func_cnt 0

     tag: 0F 86 19 76 BC 37 68 B3

  stack_depth: 16

  num_exentries: 0

  type: 2 BPF_PROG_TYPE_KPROBE

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

  used maps:

   [0] 0xffff8ccbc1b1c600 - rb

...

ffffffffc07bc801 e80a38e6f1  call 0xffffffffb2620010 ; bpf_ringbuf_submit

ffffffffc07bc806 31c0        xor eax, eax

ffffffffc07bc808 415e        pop r14

ffffffffc07bc80a 415d        pop r13

ffffffffc07bc80c 5b          pop rbx

ffffffffc07bc80d c9          leave

ffffffffc07bc80e c3          ret

and in ebpf opcodes:

43 85 00 00 00 C0 CF 02 00 call 0x2CFC0 ; bpf_ringbuf_submit

44 B7 00 00 00 00 00 00 00 mov r0, 0

45 95 00 00 00 00 00 00 00 ret

Here 0x2CFC0 is offset to bpf_ringbuf_submit from __bpf_call_base

The last call submit some data to bpf map rb with type BPF_MAP_TYPE_RINGBUF. If we could patch this function no data will be passed to usermode. How are these native function addresses filled in at all?

size of ebpf jit code on different processors

it doesn't make much sense but bcs I have now several jit compilers - why not compare how much size have jitted code for different processors?

I chose 3 ebpf programs

1. simple BPF_PROG_TYPE_CGROUP_SKB with only comparison, 8 opcodes
2. BPF_PROG_TYPE_RAW_TRACEPOINT with 3 maps, 68 opcodes
3. enough complex BPF_PROG_TYPE_RAW_TRACEPOINT with 6 maps, 1824 opcodes

results

processor	1st	2nd	3rd
x64	54	312	8195
arm64	99	567	12959
powerpc	78	546	11462
risc-v	102	470	9494
s390	78	534	12622
sparc	79	482	10446

verification of jitted ebpf code

There are some projects for ebpf in usermode, but for verification purposes you need the same code which was used in kernel. So I ripped out some jit code to run it in usermode

• x64
• powerpc
• risc-v
• s390
• sparc
• sunway sw64

And now we can make verification of jitted code - we have actual generated code for some ebpf, next we run JIT for ebpf opcodes in usermode, and finally can compare them

pmu events

Some details

pmu stored in tree pmu_idr and synced with mutex pmus_lock. and as usually can be used to blind EBPF. How? Lets see:

General speaking there are usually four steps involved to attach an eBPF program to a perf event:

1. Open the perf event
2. Load the eBPF program
3. Set the eBPF program on the perf event
4. Enable the perf event

We interested in point 4 - enabling of the perf event involves calling of pmu->event_init & pmu->add methods. And worse - all pmu structures located in .data section and thus writable. So I add today some code to dump them:

ebpf opcodes patching

I made today disasm for eBPF opcodes. Lets see how they looks like:
85 00 00 00 C0 10 02 00 call 0x210C0

in jitted code this is call 0xffffffffb4c14110. ffffffffb4c14110 - 210C0 = FFFFFFFFB4BF3050, address of __bpf_call_base. Suppose that we have some paranoidal code in kernel mode and don`t want to be traced with all this ebpf black magic, what we can do on machine without JIT?

First, we could just patch first opcode to

95 00 00 00 00 00 00 00 ret

Second - we could find some empty native function in kernel (or even reuse __bpf_call_base) and patch address let`s say htab_map_update_elem to it. Can some linux ebpf-based EDR detect this?

epbf maps

As you can see from function bpf_map_alloc_id all bpf maps stored in balanced tree map_idr and synced on spinlock map_idr_lock. No surprise that you can`t view them in user-mode - there is bpf command BPF_MAP_GET_NEXT_ID but it can only enumerate ID of maps. So I add today some code to view bpf maps: lkmem -c -d -B gives output like

bpf_maps at 0xffffffff929c1880: 15

 [0] id 3 UDPrecvAge at 0xffff99e344f48000

  type: 1 BPF_MAP_TYPE_HASH

  key_size 8 value_size 8

 [1] id 4 UDPsendAge at 0xffff99e344cb4c00

  type: 1 BPF_MAP_TYPE_HASH

  key_size 38 value_size 8

also disasm of jitted ebpf code began to look better:
mov rdi, 0xffff99e344f48000 ; UDPrecvAge

call 0xffffffff90c191f0 ; __htab_map_lookup_elem




This letter explains that JIT replacing sequence of opcodes

bpf_mov r1, const_internal_map_id
bpf_call bpf_map_lookup

with direct loading of 64bit address of map (BPF_LD_IMM64 pseudo op). But this code is not optimal - every instruction occupy 10 bytes. Lets consider case where we employ constants pool and put all map addresses somewhere after function - sure this will require at least 8 bytes for each address + perhaps some space for alignment. But now we can produce code like:
mov rdi, qword [map1_addr wrt rip] ; 7 bytes

call __htab_map_lookup_elem

...

; somewhere after function

map1_addr: resq 1 ; jit should put real address of map here

if function has 3 or more reference to the same map we can have some decreasing of jitted code size

overhead of eBPF JIT

Lets try to estimate overhead of JIT compiler

I wrote simple perl script - it just counts redundant bytes for several cases:

• pair mov reg, rbp/add reg, imm (total length 7 bytes) can be replaced with lea reg, [rbp-imm] which is only 4 bytes
• pair mov reg, imm/add reg,imm can be replaced with just loading of right address so second instruction can be removed
• add reg, 1/sub reg, 1 (length 4 bytes) can be replaced to inc/dec reg (which has length 3 bytes)

etc etc

Results

jitted eBPF code

I add yesterday disasm for jitted eBPF code. To put it mildly this code is very poor

Every function has 7 bytes of nops in prolog. Comment says that this is for BPF trampoline - well, ok

Lots of code like

 mov eax, 0x1

 cmp r14, 0x2

 jnz 0xc0561497

 xor eax, eax

0xc0561497:

 ...

Somebody - tell them about cmovXX instructions

Lots of code like

mov rdi, 0xffff8fd687f3e000

add rdi, 0x110

and related to get addresses of stack var:

mov rdi, rbp

add rdi, 0xffffffffffffffe0 

Perhaps it would be preferable to use lea rdi, [rbp-XX]

Slow inc/dec:

add r8, 0x1

sub rdi, 0x1

Lots of repeated instructions:
and rdi, 0xfff

and rdi, 0xfff

it's obvious bug

And finally

eBPF on cgroups

the long story short - they are stored in array effective and in list progs in cgroup->bpf

Below I will try to explain boring and dirty details

cgroups

This article says:

hierarchy: a set of cgroups arranged in a tree

so we need to find roots and then just traverse this trees. Roots have type cgroup_root and stored in cgroup_hierarchy_idr (synced with mutex cgroup_mutex). As usually linux lies - lets compare content of  /proc/cgroups:

#subsys_name hierarchy num_cgroups enabled

cpuset 6 1 1

cpu 5 1 1

cpuacct 5 1 1

blkio 4 1 1

memory 2 148 1

devices 9 99 1

freezer 10 1 1

net_cls 7 1 1

perf_event 8 1 1

net_prio 7 1 1

hugetlb 3 1 1

pids 11 103 1

rdma 12 1 1

with what cgroup roots actually located on this machine:

[0]  at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144

[1] systemd at 0xffff8fd6816ea000 flags 4 hierarchy_id 1 nr_cgrps 145 real_cnt 144

[2]  at 0xffff8fd68297a000 flags 0 hierarchy_id 2 nr_cgrps 148 real_cnt 147

[3]  at 0xffff8fd68297c000 flags 0 hierarchy_id 3 nr_cgrps 1 real_cnt 0

[4]  at 0xffff8fd682978000 flags 0 hierarchy_id 4 nr_cgrps 1 real_cnt 0

[5]  at 0xffff8fd68297e000 flags 0 hierarchy_id 5 nr_cgrps 1 real_cnt 0

[6]  at 0xffff8fd6854c8000 flags 0 hierarchy_id 6 nr_cgrps 1 real_cnt 0

[7]  at 0xffff8fd6854ce000 flags 0 hierarchy_id 7 nr_cgrps 1 real_cnt 0

[8]  at 0xffff8fd6854ca000 flags 0 hierarchy_id 8 nr_cgrps 1 real_cnt 0

[9]  at 0xffff8fd6854cc000 flags 0 hierarchy_id 9 nr_cgrps 99 real_cnt 98

[10]  at 0xffff8fd685e16000 flags 0 hierarchy_id 10 nr_cgrps 1 real_cnt 0

[11]  at 0xffff8fd685e12000 flags 0 hierarchy_id 11 nr_cgrps 103 real_cnt 102

[12]  at 0xffff8fd685e14000 flags 0 hierarchy_id 12 nr_cgrps 1 real_cnt 0

can you find in /proc/cgroups roots with hierarchy ID 0 and 1?

How to traverse this tree? It starts in field cgrp->self and we can use  functions css_next_descendant_pre/css_next_descendant_post etc. Strictly speaking they return pointer to cgroup_subsys_state but this is first field self  in cgroup so casting is safe

blinding sysmon for linux

 Let`s see which tracepoints it using:

sudo ./lkmem -d -c -t ~/krnl/curr ~/krnl/System.map-5.11.0-37-generic

 __tracepoint_sched_process_exit at 0xffffffffa47140c0: enabled 1 cnt 1

  [0] 0xffffffffa2ed3b40 - kernel!perf_trace_sched_process_template

 __tracepoint_sys_exit at 0xffffffffa4714ae0: enabled 1 cnt 1

  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc

  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc

  [0] 0xffffffffa2f37f90 - kernel!__bpf_trace_sys_exit

 __tracepoint_sys_enter at 0xffffffffa4714b40: enabled 1 cnt 1

  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc

  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc

  [0] 0xffffffffa2f37e30 - kernel!__bpf_trace_sys_enter

1. my favorite 1bit patch - zero tracepoint->key.enabled
2. remove BPF client from funcs list
3. find trace_event_call and install your own event_filter

BPF iterators

Sure I could not get past the hype topic of BPF (overvalued mechanism to allow you just run your buggy code in kernel with low performance and lots of overhead). For access of some kernel data they add so called iterators - and maybe you even can write your own and register it with bpf_iter_reg_target (spoiler: you can`t, bcs this function is not exported. Welcome to wonderful world of open-source with unexplained and unreasonable restrictions). I was curious what BPF iterators are in the system - they stored iterators in list targets synchronized with mutex targets_mutex. It would seem what could go wrong? 

grep " targets" System.map-5.11.0-37-generic

ffffffff820ff8e0 r targets

ffffffff826e1240 d targets_mutex

ffffffff826e1260 d targets

ffffffff8277a5c0 d targets

ffffffff8286b2e8 d targets_supported

In this case, we are dealing with another mechanism for hiding information in linux kernel - using of non-unique names. I was not even lazy and wrote a script to count such names - 998 names. Top 5:

_acpi_module_name: 155

cpumask_weight.constprop.0: 47

kzalloc.constprop.0: 39

get_order: 32

kmalloc_array.constprop.0: 28

As usual the disassembler rushes to the rescue