blinding sysmon for linux

 Let`s see which tracepoints it using:

sudo ./lkmem -d -c -t ~/krnl/curr ~/krnl/System.map-5.11.0-37-generic

 __tracepoint_sched_process_exit at 0xffffffffa47140c0: enabled 1 cnt 1

  [0] 0xffffffffa2ed3b40 - kernel!perf_trace_sched_process_template

 __tracepoint_sys_exit at 0xffffffffa4714ae0: enabled 1 cnt 1

  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc

  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc

  [0] 0xffffffffa2f37f90 - kernel!__bpf_trace_sys_exit

 __tracepoint_sys_enter at 0xffffffffa4714b40: enabled 1 cnt 1

  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc

  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc

  [0] 0xffffffffa2f37e30 - kernel!__bpf_trace_sys_enter

1. my favorite 1bit patch - zero tracepoint->key.enabled
2. remove BPF client from funcs list
3. find trace_event_call and install your own event_filter

BPF iterators

Sure I could not get past the hype topic of BPF (overvalued mechanism to allow you just run your buggy code in kernel with low performance and lots of overhead). For access of some kernel data they add so called iterators - and maybe you even can write your own and register it with bpf_iter_reg_target (spoiler: you can`t, bcs this function is not exported. Welcome to wonderful world of open-source with unexplained and unreasonable restrictions). I was curious what BPF iterators are in the system - they stored iterators in list targets synchronized with mutex targets_mutex. It would seem what could go wrong? 

grep " targets" System.map-5.11.0-37-generic

ffffffff820ff8e0 r targets

ffffffff826e1240 d targets_mutex

ffffffff826e1260 d targets

ffffffff8277a5c0 d targets

ffffffff8286b2e8 d targets_supported

In this case, we are dealing with another mechanism for hiding information in linux kernel - using of non-unique names. I was not even lazy and wrote a script to count such names - 998 names. Top 5:

_acpi_module_name: 155

cpumask_weight.constprop.0: 47

kzalloc.constprop.0: 39

get_order: 32

kmalloc_array.constprop.0: 28

As usual the disassembler rushes to the rescue

security hooks in linux kernel

This mechanism was inspired by NSA. As described all hooks stored in huge struct security_hooks_list, but it`s format is different in each version. We can determine which list belongs to what hook with disasm magic. Lets see function that calls security hooks - for example security_path_chown:

.text:FFFFFFC010496448 security_path_chown        ; CODE XREF: chown_common+104↑p

.text:FFFFFFC010496448   STP             X29, X30, [SP,#-0x18+var_18]!

.text:FFFFFFC01049644C   MOV             X29, SP

.text:FFFFFFC010496450   STP             X20, X21, [SP,#0x18+var_s0]

.text:FFFFFFC010496454   STR             X22, [SP,#0x18+var_s10]

.text:FFFFFFC010496458   MOV             X20, path

.text:FFFFFFC01049645C   MOV             W21, W1

.text:FFFFFFC010496460   MOV             path, X30

.text:FFFFFFC010496464   MOV             W22, W2

.text:FFFFFFC010496468 loc_FFFFFFC010496468  ; DATA XREF: .init.data:FFFFFFC0111474C0↓o

.text:FFFFFFC010496468   BL              _mcount

.text:FFFFFFC01049646C   LDR             X0, [path,#8]

.text:FFFFFFC010496470   LDR             X0, [X0,#0x30]

.text:FFFFFFC010496474   LDR             W0, [X0,#0xC]

.text:FFFFFFC010496478   TBNZ            W0, #9, loc_FFFFFFC0104964B4

.text:FFFFFFC01049647C   ADRP            X0, #security_hook_heads_0.path_chown@PAGE

.text:FFFFFFC010496480   STR             X19, [X29,#0x18+var_8]

.text:FFFFFFC010496484   LDR             X19, [X0,#security_hook_heads_0.path_chown@PAGEOFF]

In disasm we just search for first reference to memory near address of security_hook_heads. Some results:

what linux hiding

disclaimer

there is no doubt that the list below is incomplete, inaccurate etc - it`s just what very average programmer can find during two month of browsing linux source code

observability criteria

what I mean under "hiding"? It means that

• no kernel API to enumerate some structure
• no real-time notifications about setting some hook
• no mapping on /proc or /sys (however this method is not reliable)
• no 3rd party tools to show this. As an example I chose volatility - just bcs I readed their folio "The Art of Memory Forensics"

So you unable to see them

notification chains

very ironic that they have API like register_XXX_notifier/unregister_XXX_notifier and there is no function like enum_XXX_notifier

no mapping on /proc or /sys

volatility checks only very limited set - vt_notifier_list & keyboard_notifier_list

tracepoints

no API to enum clients

no notification about turning on some tracepoint

has mapping to /sys/kernel/tracing/events but can`t show clients of some tracepoint

volatility - no

kprobes

no API to enum consumers of some installed KPROBE

no notification about installing new kprobe. This is an extremely sad fact - for example tools like LKRG don`t knows that some memory was patched

has mapping to /sys/kernel/debug/kprobes/

volatility - no

uprobes

no API to enum consumers of some installed UPROBE

no notification about installing new uprobe

has mapping to /sys/kernel/debug/tracing/uprobe_events. Most crazy thing is that uprobes installed from kernel not shown

volatility - no

filesystem notifications

no API to enum all installed marks

for usermode events has notification via security_path_notify, for kernelmode - absolutely not

has very limited mapping to /proc/*/fdinfo/*. Again marks installed from kernel not shown

volatility - no

PoC to hide kprobes list

as you may know list of kprobes has mapping on /sys in file /sys/kernel/debug/kprobes/list. And now when I have working filesystem notifications it would be extremely tempting try to make hiding content of this file. Let`s see what this inode contains:

sudo ./lkmem -s -c ~/krnl/curr ~/krnl/System.map-5.11.0-34- generic /sys/kernel/debug/kprobes/list 

res /sys/kernel/debug/kprobes/list: (nil)

 inode: 0xffff8a0448d1ae40

 s_op: 0xffffffffa5067f80 - kernel!debugfs_super_operations

 inode->i_fop: 0xffffffffa506b000 - kernel!debugfs_full_proxy_file_operations

 debugfs_real_fops: 0xffffffffa5028ce0 - kernel!kprobes_fops

 private_data: 0xffffffffa5028e00 - kernel!kprobes_sops

kprobes_sops is just struct seq_operations and the function we need is show. So idea is simple

• set notification for file /sys/kernel/debug/kprobes/list
• in fsnotify_handle_event callback check inode and mask
• if this is first opening of this file - patch kprobes_sops->show to our own function (be cautious with WP in cr0)
• if this is last closing of this file - return original handler to kprobes_sops->show
• also return original handler when driver is unloading

You may ask - why is it so difficult? It`s much easier just to patch kprobes_sops->show, right? The answer is that you minimize the risk of being discovered when patching only for some short period 

filesystem notifications in linux kernel

disclaimer

Filesystems are the most complex part of any OS. I am not a specialist in linux filesystems and even don`t commit the code to linux kernel. So all information here cannot be considered reliable, code has tons of bugs and can damage your machine and ruin the rest of your life

Usermode notifications

linux has 3 mechanisms for passing filesystem notification to user-mode:

1. dnotify
2. inotify
3. new-fashioned fanotify

all they projected to used-mode as file (analogue of FilterConnectionPorts), so you can use lsof (or even just something like "ls /proc/*/fdinfo/* | xargs grep notify") to find then and what processes do they belong to. Unfortunately (as usually) this information is not enough. Let see for example function fanotify_fdinfo. We can notice that there are 3 possible source of notifications:

1. just simple inode - for them dumped inode->i_ino & superblock s_dev - I don`t know how you can in usermode find mountpoint for this superblock
2. mount point (btw struct mount even not described in linux/include). At least knowing mnt_id you can find name in /proc/pid/mountinfo file
3. superblock - s_dev again dumping

Can you have real-time notifications about setting new xxnotify? Yes, via security_path_notify. At the same time no notifications about removing

Kernel mode notifications

Can you have the same functionality provided by xxnotify in kernel mode? Definitely yes - kernel audit uses it. I could not find any sample code to do this in your own driver so I wrote one. This is not very complex (although function fsnotify_destroy_group is not exported so you need some sort of kallsyms lookup). You can add to tracked_inode everything you want - like full filename, stat etc

And now the main question

Can you find all sources of filesystem notification?

linux kernel uprobes

Lets consider another spying mechanism in linux kernel - uprobes. They also insert int3 but this time in user-mode and can be used for example to steal TLS traffic. I made simple code to set up uprobe for /usr/bin/ls on PLT thunk getenv:

objdump -d /usr/bin/ls

...

0000000000004710 <getenv@plt>:

    4710: f3 0f 1e fa          endbr64 

    4714: f2 ff 25 5d e5 01 00 bnd jmpq *0x1e55d(%rip)        # 22c78 <getenv@GLIBC_2.2.5>

    471b: 0f 1f 44 00 00        nopl   0x0(%rax,%rax,1)

now run ls
ls -i /usr/bin/ls

1043126 /usr/bin/ls 

dmesg | tail

[258600.533089] uprobe ret_handler is executed, ip = 55EAECA62B54

[258600.533090] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533093] uprobe ret_handler is executed, ip = 55EAECA62B6C

[258600.533095] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533098] uprobe ret_handler is executed, ip = 55EAECA5861C

[258600.533099] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533102] uprobe ret_handler is executed, ip = 55EAECA57F60

[258600.533111] uprobe handler in PID 43831 executed, ip = 55eaeca56710

[258600.533114] uprobe ret_handler is executed, ip = 55EAECA57A77

And you can`t see which uprobes are installed - file /sys/kernel/debug/tracing/uprobe_events is empty. NSA can hide their anal catheters even in opened sources, yeah. So I wrote code to dump all uprobes (stored in uprobes_tree) and consumers of each uprobe