PoC to hide kprobes list

as you may know list of kprobes has mapping on /sys in file /sys/kernel/debug/kprobes/list. And now when I have working filesystem notifications it would be extremely tempting try to make hiding content of this file. Let`s see what this inode contains:

sudo ./lkmem -s -c ~/krnl/curr ~/krnl/System.map-5.11.0-34- generic /sys/kernel/debug/kprobes/list 

res /sys/kernel/debug/kprobes/list: (nil)

 inode: 0xffff8a0448d1ae40

 s_op: 0xffffffffa5067f80 - kernel!debugfs_super_operations

 inode->i_fop: 0xffffffffa506b000 - kernel!debugfs_full_proxy_file_operations

 debugfs_real_fops: 0xffffffffa5028ce0 - kernel!kprobes_fops

 private_data: 0xffffffffa5028e00 - kernel!kprobes_sops

kprobes_sops is just struct seq_operations and the function we need is show. So idea is simple

• set notification for file /sys/kernel/debug/kprobes/list
• in fsnotify_handle_event callback check inode and mask
• if this is first opening of this file - patch kprobes_sops->show to our own function (be cautious with WP in cr0)
• if this is last closing of this file - return original handler to kprobes_sops->show
• also return original handler when driver is unloading

You may ask - why is it so difficult? It`s much easier just to patch kprobes_sops->show, right? The answer is that you minimize the risk of being discovered when patching only for some short period 

filesystem notifications in linux kernel

disclaimer

Filesystems are the most complex part of any OS. I am not a specialist in linux filesystems and even don`t commit the code to linux kernel. So all information here cannot be considered reliable, code has tons of bugs and can damage your machine and ruin the rest of your life

Usermode notifications

linux has 3 mechanisms for passing filesystem notification to user-mode:

1. dnotify
2. inotify
3. new-fashioned fanotify

all they projected to used-mode as file (analogue of FilterConnectionPorts), so you can use lsof (or even just something like "ls /proc/*/fdinfo/* | xargs grep notify") to find then and what processes do they belong to. Unfortunately (as usually) this information is not enough. Let see for example function fanotify_fdinfo. We can notice that there are 3 possible source of notifications:

1. just simple inode - for them dumped inode->i_ino & superblock s_dev - I don`t know how you can in usermode find mountpoint for this superblock
2. mount point (btw struct mount even not described in linux/include). At least knowing mnt_id you can find name in /proc/pid/mountinfo file
3. superblock - s_dev again dumping

Can you have real-time notifications about setting new xxnotify? Yes, via security_path_notify. At the same time no notifications about removing

Kernel mode notifications

Can you have the same functionality provided by xxnotify in kernel mode? Definitely yes - kernel audit uses it. I could not find any sample code to do this in your own driver so I wrote one. This is not very complex (although function fsnotify_destroy_group is not exported so you need some sort of kallsyms lookup). You can add to tracked_inode everything you want - like full filename, stat etc

And now the main question

Can you find all sources of filesystem notification?

linux kernel uprobes

Lets consider another spying mechanism in linux kernel - uprobes. They also insert int3 but this time in user-mode and can be used for example to steal TLS traffic. I made simple code to set up uprobe for /usr/bin/ls on PLT thunk getenv:

objdump -d /usr/bin/ls

...

0000000000004710 <getenv@plt>:

    4710: f3 0f 1e fa          endbr64 

    4714: f2 ff 25 5d e5 01 00 bnd jmpq *0x1e55d(%rip)        # 22c78 <getenv@GLIBC_2.2.5>

    471b: 0f 1f 44 00 00        nopl   0x0(%rax,%rax,1)

now run ls
ls -i /usr/bin/ls

1043126 /usr/bin/ls 

dmesg | tail

[258600.533089] uprobe ret_handler is executed, ip = 55EAECA62B54

[258600.533090] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533093] uprobe ret_handler is executed, ip = 55EAECA62B6C

[258600.533095] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533098] uprobe ret_handler is executed, ip = 55EAECA5861C

[258600.533099] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533102] uprobe ret_handler is executed, ip = 55EAECA57F60

[258600.533111] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533114] uprobe ret_handler is executed, ip = 55EAECA57A77

And you can`t see which uprobes are installed - file /sys/kernel/debug/tracing/uprobe_events is empty. NSA can hide their anal catheters even in opened sources, yeah. So I wrote code to dump all uprobes (stored in uprobes_tree) and consumers of each uprobe

linux kernel kprobes

without a doubt most crazy and insane spying mechanism in linux kernel is krobes

1. It`s expensive - each time when int3 occurred typical call stack looks like:
   xen_asm_exc_int3
asm_exc_int3
irq_entries_start
exc_int3
do_int3
kprobe_int3_handler

2. It makes working with kdbg (which itself is too far away from windbg) like nightmare - function do_int3 first calls kgdb_ll_trap
3. There is no mechanism to predict which functions cannot be kprobed. Let assume that your handler uses simple printk - so you can`t set kprobe on whole graph of functions called from printk (like vprintk_func, vprintk_default, vprintk_emit, __msecs_to_jiffies, arch_touch_nmi_watchdog, touch_softlockup_watchdog, __printk_safe_enter, _raw_spin_lock, vprintk_store, vscnprintf, cont_flush etc etc) and as far I know there is no way to even find them all
4. Sure you have /sys/kernel/debug/kprobes/list file so you can see which functions was hooked. But there is no way to know by whom

So I wrote dumper of installed kprobes. Sample of output:

sudo ./lkmem -k -c ~/krnl/curr ~/krnl/System.map-5.11.0-34-generic

kprobes[47]: 1

 kprobe at 0xffffffffc0605080 flags 8

  addr: 0xffffffffa4a9f040 - kernel!__do_sys_fork

  pre_handler: 0xffffffffc0603548 - lkcd

  post_handler: 0xffffffffc0603526 - lkcd

linux-kernel per-cpu vars

It`s hard to believe but linux has degraded version of KPCR on windows - so called "per-cpu variables". This is some isolated memory assigned to CPU (stored in gs segment register on x64 and in MSR register c13 on arm64) and can contains some interesting fields. Why this is important to know offsets some of this variables? Well, I suspect that linux kernel contains much more code for espionage than windows (for example trace events, tracepoints, kprobes, usb_mon_register etc etc). One of such code is function user_return_notifier_register with which you can register your own notifications. Unfortunately this list of notifications stored in per-cpu variable return_notifier_list

And as usually there is no some include file with definition of all of this per-cpu fields. Moreover this offsets depend from config for kernel building and differ in each build. Sounds like nightmare, reason to turn off the computer and go drink vodka looking at the autumn rain.

Or not? Lets see in disasm some functions using this var - like fire_user_return_notifiers:

fire_user_return_notifiers proc near

 call    __fentry__ ; another entry for spy code

 mov     rax, offset unk_29450

 add     rax, gs:this_cpu_off ; .data..percpu:0000000000011368

 mov     rdi, [rax]

In this build return_notifier_list happens to have offset 0x29450 and this_cpu_off 0x11368. 

Well, we can use disasm to get offsets to both return_notifier_list & this_cpu_off and then write code like:

; rdi - this_cpu_off

; rsi - offset

get_this_gs:

mov rax, [gs:rdi]

add rax, rsi

ret

Patch on github to extract this_cpu_off & return_notifier_list with some disasm magic