security hooks in linux kernel

This mechanism was inspired by NSA. As described all hooks stored in huge struct security_hooks_list, but it`s format is different in each version. We can determine which list belongs to what hook with disasm magic. Lets see function that calls security hooks - for example security_path_chown:

.text:FFFFFFC010496448 security_path_chown        ; CODE XREF: chown_common+104↑p

.text:FFFFFFC010496448   STP             X29, X30, [SP,#-0x18+var_18]!

.text:FFFFFFC01049644C   MOV             X29, SP

.text:FFFFFFC010496450   STP             X20, X21, [SP,#0x18+var_s0]

.text:FFFFFFC010496454   STR             X22, [SP,#0x18+var_s10]

.text:FFFFFFC010496458   MOV             X20, path

.text:FFFFFFC01049645C   MOV             W21, W1

.text:FFFFFFC010496460   MOV             path, X30

.text:FFFFFFC010496464   MOV             W22, W2

.text:FFFFFFC010496468 loc_FFFFFFC010496468  ; DATA XREF: .init.data:FFFFFFC0111474C0↓o

.text:FFFFFFC010496468   BL              _mcount

.text:FFFFFFC01049646C   LDR             X0, [path,#8]

.text:FFFFFFC010496470   LDR             X0, [X0,#0x30]

.text:FFFFFFC010496474   LDR             W0, [X0,#0xC]

.text:FFFFFFC010496478   TBNZ            W0, #9, loc_FFFFFFC0104964B4

.text:FFFFFFC01049647C   ADRP            X0, #security_hook_heads_0.path_chown@PAGE

.text:FFFFFFC010496480   STR             X19, [X29,#0x18+var_8]

.text:FFFFFFC010496484   LDR             X19, [X0,#security_hook_heads_0.path_chown@PAGEOFF]

In disasm we just search for first reference to memory near address of security_hook_heads. Some results:

what linux hiding

disclaimer

there is no doubt that the list below is incomplete, inaccurate etc - it`s just what very average programmer can find during two month of browsing linux source code

observability criteria

what I mean under "hiding"? It means that

• no kernel API to enumerate some structure
• no real-time notifications about setting some hook
• no mapping on /proc or /sys (however this method is not reliable)
• no 3rd party tools to show this. As an example I chose volatility - just bcs I readed their folio "The Art of Memory Forensics"

So you unable to see them

notification chains

very ironic that they have API like register_XXX_notifier/unregister_XXX_notifier and there is no function like enum_XXX_notifier

no mapping on /proc or /sys

volatility checks only very limited set - vt_notifier_list & keyboard_notifier_list

tracepoints

no API to enum clients

no notification about turning on some tracepoint

has mapping to /sys/kernel/tracing/events but can`t show clients of some tracepoint

volatility - no

kprobes

no API to enum consumers of some installed KPROBE

no notification about installing new kprobe. This is an extremely sad fact - for example tools like LKRG don`t knows that some memory was patched

has mapping to /sys/kernel/debug/kprobes/

volatility - no

uprobes

no API to enum consumers of some installed UPROBE

no notification about installing new uprobe

has mapping to /sys/kernel/debug/tracing/uprobe_events. Most crazy thing is that uprobes installed from kernel not shown

volatility - no

filesystem notifications

no API to enum all installed marks

for usermode events has notification via security_path_notify, for kernelmode - absolutely not

has very limited mapping to /proc/*/fdinfo/*. Again marks installed from kernel not shown

volatility - no

PoC to hide kprobes list

as you may know list of kprobes has mapping on /sys in file /sys/kernel/debug/kprobes/list. And now when I have working filesystem notifications it would be extremely tempting try to make hiding content of this file. Let`s see what this inode contains:

sudo ./lkmem -s -c ~/krnl/curr ~/krnl/System.map-5.11.0-34- generic /sys/kernel/debug/kprobes/list 

res /sys/kernel/debug/kprobes/list: (nil)

 inode: 0xffff8a0448d1ae40

 s_op: 0xffffffffa5067f80 - kernel!debugfs_super_operations

 inode->i_fop: 0xffffffffa506b000 - kernel!debugfs_full_proxy_file_operations

 debugfs_real_fops: 0xffffffffa5028ce0 - kernel!kprobes_fops

 private_data: 0xffffffffa5028e00 - kernel!kprobes_sops

kprobes_sops is just struct seq_operations and the function we need is show. So idea is simple

• set notification for file /sys/kernel/debug/kprobes/list
• in fsnotify_handle_event callback check inode and mask
• if this is first opening of this file - patch kprobes_sops->show to our own function (be cautious with WP in cr0)
• if this is last closing of this file - return original handler to kprobes_sops->show
• also return original handler when driver is unloading

You may ask - why is it so difficult? It`s much easier just to patch kprobes_sops->show, right? The answer is that you minimize the risk of being discovered when patching only for some short period 

filesystem notifications in linux kernel

disclaimer

Filesystems are the most complex part of any OS. I am not a specialist in linux filesystems and even don`t commit the code to linux kernel. So all information here cannot be considered reliable, code has tons of bugs and can damage your machine and ruin the rest of your life

Usermode notifications

linux has 3 mechanisms for passing filesystem notification to user-mode:

1. dnotify
2. inotify
3. new-fashioned fanotify

all they projected to used-mode as file (analogue of FilterConnectionPorts), so you can use lsof (or even just something like "ls /proc/*/fdinfo/* | xargs grep notify") to find then and what processes do they belong to. Unfortunately (as usually) this information is not enough. Let see for example function fanotify_fdinfo. We can notice that there are 3 possible source of notifications:

1. just simple inode - for them dumped inode->i_ino & superblock s_dev - I don`t know how you can in usermode find mountpoint for this superblock
2. mount point (btw struct mount even not described in linux/include). At least knowing mnt_id you can find name in /proc/pid/mountinfo file
3. superblock - s_dev again dumping

Can you have real-time notifications about setting new xxnotify? Yes, via security_path_notify. At the same time no notifications about removing

Kernel mode notifications

Can you have the same functionality provided by xxnotify in kernel mode? Definitely yes - kernel audit uses it. I could not find any sample code to do this in your own driver so I wrote one. This is not very complex (although function fsnotify_destroy_group is not exported so you need some sort of kallsyms lookup). You can add to tracked_inode everything you want - like full filename, stat etc

And now the main question

Can you find all sources of filesystem notification?

linux kernel uprobes

Lets consider another spying mechanism in linux kernel - uprobes. They also insert int3 but this time in user-mode and can be used for example to steal TLS traffic. I made simple code to set up uprobe for /usr/bin/ls on PLT thunk getenv:

objdump -d /usr/bin/ls

...

0000000000004710 <getenv@plt>:

    4710: f3 0f 1e fa          endbr64 

    4714: f2 ff 25 5d e5 01 00 bnd jmpq *0x1e55d(%rip)        # 22c78 <getenv@GLIBC_2.2.5>

    471b: 0f 1f 44 00 00        nopl   0x0(%rax,%rax,1)

now run ls
ls -i /usr/bin/ls

1043126 /usr/bin/ls 

dmesg | tail

[258600.533089] uprobe ret_handler is executed, ip = 55EAECA62B54

[258600.533090] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533093] uprobe ret_handler is executed, ip = 55EAECA62B6C

[258600.533095] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533098] uprobe ret_handler is executed, ip = 55EAECA5861C

[258600.533099] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533102] uprobe ret_handler is executed, ip = 55EAECA57F60

[258600.533111] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533114] uprobe ret_handler is executed, ip = 55EAECA57A77

And you can`t see which uprobes are installed - file /sys/kernel/debug/tracing/uprobe_events is empty. NSA can hide their anal catheters even in opened sources, yeah. So I wrote code to dump all uprobes (stored in uprobes_tree) and consumers of each uprobe

linux kernel kprobes

without a doubt most crazy and insane spying mechanism in linux kernel is krobes

1. It`s expensive - each time when int3 occurred typical call stack looks like:
   xen_asm_exc_int3
asm_exc_int3
irq_entries_start
exc_int3
do_int3
kprobe_int3_handler

2. It makes working with kdbg (which itself is too far away from windbg) like nightmare - function do_int3 first calls kgdb_ll_trap
3. There is no mechanism to predict which functions cannot be kprobed. Let assume that your handler uses simple printk - so you can`t set kprobe on whole graph of functions called from printk (like vprintk_func, vprintk_default, vprintk_emit, __msecs_to_jiffies, arch_touch_nmi_watchdog, touch_softlockup_watchdog, __printk_safe_enter, _raw_spin_lock, vprintk_store, vscnprintf, cont_flush etc etc) and as far I know there is no way to even find them all
4. Sure you have /sys/kernel/debug/kprobes/list file so you can see which functions was hooked. But there is no way to know by whom

So I wrote dumper of installed kprobes. Sample of output:

sudo ./lkmem -k -c ~/krnl/curr ~/krnl/System.map-5.11.0-34-generic

kprobes[47]: 1

 kprobe at 0xffffffffc0605080 flags 8

  addr: 0xffffffffa4a9f040 - kernel!__do_sys_fork

  pre_handler: 0xffffffffc0603548 - lkcd

  post_handler: 0xffffffffc0603526 - lkcd

linux-kernel per-cpu vars

It`s hard to believe but linux has degraded version of KPCR on windows - so called "per-cpu variables". This is some isolated memory assigned to CPU (stored in gs segment register on x64 and in MSR register c13 on arm64) and can contains some interesting fields. Why this is important to know offsets some of this variables? Well, I suspect that linux kernel contains much more code for espionage than windows (for example trace events, tracepoints, kprobes, usb_mon_register etc etc). One of such code is function user_return_notifier_register with which you can register your own notifications. Unfortunately this list of notifications stored in per-cpu variable return_notifier_list

And as usually there is no some include file with definition of all of this per-cpu fields. Moreover this offsets depend from config for kernel building and differ in each build. Sounds like nightmare, reason to turn off the computer and go drink vodka looking at the autumn rain.

Or not? Lets see in disasm some functions using this var - like fire_user_return_notifiers:

fire_user_return_notifiers proc near

 call    __fentry__ ; another entry for spy code

 mov     rax, offset unk_29450

 add     rax, gs:this_cpu_off ; .data..percpu:0000000000011368

 mov     rdi, [rax]

In this build return_notifier_list happens to have offset 0x29450 and this_cpu_off 0x11368. 

Well, we can use disasm to get offsets to both return_notifier_list & this_cpu_off and then write code like:

; rdi - this_cpu_off

; rsi - offset

get_this_gs:

mov rax, [gs:rdi]

add rax, rsi

ret

Patch on github to extract this_cpu_off & return_notifier_list with some disasm magic