Lets try to estimate overhead of JIT compiler
I wrote simple perl script - it just counts redundant bytes for several cases:
I add yesterday disasm for jitted eBPF code. To put it mildly this code is very poor
Every function has 7 bytes of nops in prolog. Comment says that this is for BPF trampoline - well, ok
Lots of code like
mov eax, 0x1 cmp r14, 0x2 jnz 0xc0561497 xor eax, eax0xc0561497: ...mov rdi, 0xffff8fd687f3e000add rdi, 0x110mov rdi, rbpadd rdi, 0xffffffffffffffe0 and rdi, 0xfffand rdi, 0xfffhierarchy: a set of cgroups arranged in a treeso we need to find roots and then just traverse this trees. Roots have type cgroup_root and stored in cgroup_hierarchy_idr (synced with mutex cgroup_mutex). As usually linux lies - lets compare content of /proc/cgroups:
#subsys_name hierarchy num_cgroups enabledcpuset 6 1 1cpu 5 1 1cpuacct 5 1 1blkio 4 1 1memory 2 148 1devices 9 99 1freezer 10 1 1net_cls 7 1 1perf_event 8 1 1net_prio 7 1 1hugetlb 3 1 1pids 11 103 1rdma 12 1 1
[0] at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144[1] systemd at 0xffff8fd6816ea000 flags 4 hierarchy_id 1 nr_cgrps 145 real_cnt 144[2] at 0xffff8fd68297a000 flags 0 hierarchy_id 2 nr_cgrps 148 real_cnt 147[3] at 0xffff8fd68297c000 flags 0 hierarchy_id 3 nr_cgrps 1 real_cnt 0[4] at 0xffff8fd682978000 flags 0 hierarchy_id 4 nr_cgrps 1 real_cnt 0[5] at 0xffff8fd68297e000 flags 0 hierarchy_id 5 nr_cgrps 1 real_cnt 0[6] at 0xffff8fd6854c8000 flags 0 hierarchy_id 6 nr_cgrps 1 real_cnt 0[7] at 0xffff8fd6854ce000 flags 0 hierarchy_id 7 nr_cgrps 1 real_cnt 0[8] at 0xffff8fd6854ca000 flags 0 hierarchy_id 8 nr_cgrps 1 real_cnt 0[9] at 0xffff8fd6854cc000 flags 0 hierarchy_id 9 nr_cgrps 99 real_cnt 98[10] at 0xffff8fd685e16000 flags 0 hierarchy_id 10 nr_cgrps 1 real_cnt 0[11] at 0xffff8fd685e12000 flags 0 hierarchy_id 11 nr_cgrps 103 real_cnt 102[12] at 0xffff8fd685e14000 flags 0 hierarchy_id 12 nr_cgrps 1 real_cnt 0and some
Let`s see which tracepoints it using:
sudo ./lkmem -d -c -t ~/krnl/curr ~/krnl/System.map-5.11.0-37-generic __tracepoint_sched_process_exit at 0xffffffffa47140c0: enabled 1 cnt 1 [0] 0xffffffffa2ed3b40 - kernel!perf_trace_sched_process_template __tracepoint_sys_exit at 0xffffffffa4714ae0: enabled 1 cnt 1 regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc [0] 0xffffffffa2f37f90 - kernel!__bpf_trace_sys_exit __tracepoint_sys_enter at 0xffffffffa4714b40: enabled 1 cnt 1 regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc [0] 0xffffffffa2f37e30 - kernel!__bpf_trace_sys_enter
Sure I could not get past the hype topic of BPF (overvalued mechanism to allow you just run your buggy code in kernel with low performance and lots of overhead). For access of some kernel data they add so called iterators - and maybe you even can write your own and register it with bpf_iter_reg_target (spoiler: you can`t, bcs this function is not exported. Welcome to wonderful world of open-source with unexplained and unreasonable restrictions). I was curious what BPF iterators are in the system - they stored iterators in list targets synchronized with mutex targets_mutex. It would seem what could go wrong?
grep " targets" System.map-5.11.0-37-genericffffffff820ff8e0 r targetsffffffff826e1240 d targets_mutexffffffff826e1260 d targetsffffffff8277a5c0 d targetsffffffff8286b2e8 d targets_supportedThis mechanism was inspired by NSA. As described all hooks stored in huge struct security_hooks_list, but it`s format is different in each version. We can determine which list belongs to what hook with disasm magic. Lets see function that calls security hooks - for example security_path_chown:
.text:FFFFFFC010496448 security_path_chown ; CODE XREF: chown_common+104↑p.text:FFFFFFC010496448 STP X29, X30, [SP,#-0x18+var_18]!.text:FFFFFFC01049644C MOV X29, SP.text:FFFFFFC010496450 STP X20, X21, [SP,#0x18+var_s0].text:FFFFFFC010496454 STR X22, [SP,#0x18+var_s10].text:FFFFFFC010496458 MOV X20, path.text:FFFFFFC01049645C MOV W21, W1.text:FFFFFFC010496460 MOV path, X30.text:FFFFFFC010496464 MOV W22, W2.text:FFFFFFC010496468 loc_FFFFFFC010496468 ; DATA XREF: .init.data:FFFFFFC0111474C0↓o.text:FFFFFFC010496468 BL _mcount.text:FFFFFFC01049646C LDR X0, [path,#8].text:FFFFFFC010496470 LDR X0, [X0,#0x30].text:FFFFFFC010496474 LDR W0, [X0,#0xC].text:FFFFFFC010496478 TBNZ W0, #9, loc_FFFFFFC0104964B4.text:FFFFFFC01049647C ADRP X0, #security_hook_heads_0.path_chown@PAGE.text:FFFFFFC010496480 STR X19, [X29,#0x18+var_8].text:FFFFFFC010496484 LDR X19, [X0,#security_hook_heads_0.path_chown@PAGEOFF]
as you may know list of kprobes has mapping on /sys in file /sys/kernel/debug/kprobes/list. And now when I have working filesystem notifications it would be extremely tempting try to make hiding content of this file. Let`s see what this inode contains:
sudo ./lkmem -s -c ~/krnl/curr ~/krnl/System.map-5.11.0-34- generic /sys/kernel/debug/kprobes/list disclaimer
Filesystems are the most complex part of any OS. I am not a specialist in linux filesystems and even don`t commit the code to linux kernel. So all information here cannot be considered reliable, code has tons of bugs and can damage your machine and ruin the rest of your life
Usermode notifications
linux has 3 mechanisms for passing filesystem notification to user-mode:
Lets consider another spying mechanism in linux kernel - uprobes. They also insert int3 but this time in user-mode and can be used for example to steal TLS traffic. I made simple code to set up uprobe for /usr/bin/ls on PLT thunk getenv:
objdump -d /usr/bin/lswithout a doubt most crazy and insane spying mechanism in linux kernel is krobes
xen_asm_exc_int3
asm_exc_int3
irq_entries_start
exc_int3
do_int3
kprobe_int3_handler
sudo ./lkmem -k -c ~/krnl/curr ~/krnl/System.map-5.11.0-34-generic
kprobes[47]: 1 kprobe at 0xffffffffc0605080 flags 8 addr: 0xffffffffa4a9f040 - kernel!__do_sys_fork pre_handler: 0xffffffffc0603548 - lkcd post_handler: 0xffffffffc0603526 - lkcd
It`s hard to believe but linux has degraded version of KPCR on windows - so called "per-cpu variables". This is some isolated memory assigned to CPU (stored in gs segment register on x64 and in MSR register c13 on arm64) and can contains some interesting fields. Why this is important to know offsets some of this variables? Well, I suspect that linux kernel contains much more code for espionage than windows (for example trace events, tracepoints, kprobes, usb_mon_register etc etc). One of such code is function user_return_notifier_register with which you can register your own notifications. Unfortunately this list of notifications stored in per-cpu variable return_notifier_list
And as usually there is no some include file with definition of all of this per-cpu fields. Moreover this offsets depend from config for kernel building and differ in each build. Sounds like nightmare, reason to turn off the computer and go drink vodka looking at the autumn rain.It`s hard to believe but linux kernel has almost exact copy of windows ETW - event tracing. It is just as difficult to make it work, it is poorly documented, very complex and fragile. And yes, as you can guess - it also can`t show who and which parts of it in use. So I wrote some code to dump registered funcs in tracepoints and to check file ops for files in /sys/kernel/tracing/events
Lets start with tracepoints. As you see this structure has strange looked list of functions in field funcs, and calling happens in functions like event_triggers_call. How we can find this tracepoints? Well, they stored in trace_event_call->tp and array of pointers to trace_event_call located between symbols __start_ftrace_events & __stop_ftrace_events. Unfortunately all this treasures located in discardable section .init.data. But because they were all declared in the same manner we can find them by name - all symbols with prefix __tracepoint_ is what we need. So some examples (you can run lkmem -c -t vmlinux system.map to get this):
__tracepoint_sys_enter at 0xffffffff8b82e340: enabled 0 cnt 0
regfunc 0xffffffff8a192330 - kernel!syscall_regfunc
unregfunc 0xffffffff8a1923f0 - kernel!syscall_unregfunc
Well, no clients right now - cnt 0
Next about /sys/kernel/tracing/events files (this is perverted inhuman interface to manage trace events). I just dumping file->f_path.dentry->d_inode->i_fop for each such file. Sample of output (you can achieve this with lkmem -s vmlinux system.map path_to_some_sys_kernel_tracing_file):
I added today disassembler for arm64 linux kernel to search pointers. It turned out to be surprisingly difficult to do for several reasons (disasm for x64 is only 383 LOC vs 618 for arm64)
One of them is poor code produced by some gcc versions
But the main problem is arm64 opcodes. Lets see simple indirect call:I wrote simple program to estimate size of problem. Yes, I know about CFI but it seems that even on kernel 5.11 on fresh Ubuntu this mechanism is not implemented and indirect calls looks like:
mov rax, cs:XXXFirst approach is just to scan .data section - you can do this running
./lkmem path-to-unpacked-kernel path-to-System.mapThere seems to be one little-known thing in linux kernel - notification chains. So they have literal analogue of PsSetLoadImageNotifyRoutine - function register_module_notifier. And similarly they don't have a function to enumerate registered notifications - I don`t know why. Maybe they were bitten by Microsoft. Or maybe I want too much from people whose even "The Linux Kernel Module Programming Guide" contains an error in the code example. Anyway I decided to write my own (btw the last time I wrote drivers for Linux was something around 20 years ago)
How to run
git clone https://github.com/redplait/lkcd.gitI got following crash when tried to solve some trivial task:
Lets assume that we have buggy and dangerous driver (which "rely on many unexported functions and select them via pattern scans which are regularly revalidated against windows insider builds", he-he). Sure we want restrict access to it, for example like ProcessHacker do
Unfortunately the latter uses CNG and cannot work on xp/w2k3. So I made fork of libecc to use this library with WDK7. Test driver and client also includedec_utils.exe gen_keys BRAINPOOLP512R1 ECRDSA mykeypairec_utils.exe sign BRAINPOOLP512R1 ECRDSA SHA3_512 testclnt.exe mykeypair_private_key.bin testclnt.sigIOCTL_TEST_IOCTL return 1testclnt.exe -u
Lets see what generates gcc for arm64 - for example gcc7.5 and linux kernel
Function do_sysinstr:
process_all_poor_gcc_functions poorgcc64 0 1I already described how you can extract address of GlobalRpcServer and offset to some RPC_SERVER_T fields. Lets do it for arm64 in declarative manner using FSM
Start again with I_RpcServerRegisterForwardFunction function - we can get address of RpcHasBeenInitialized (will be stored with index 1), GlobalRpcServer (with index 2) and RPC_SERVER_T.pRpcForwardFunction offset (with index 3):
In previous post I described declarative way to find non-exported data and functions using FSM. But often you also need to know offsets to some fields in structures - they can be changed in different versions of Windows. So let see if this can be done in the same declarative manner
Perhaps most safe way is to track registers contained arguments to some function (btw not necessary exported). So I added yet two states to FSM
I added saving and loading of FSM rules in file - so now you can edit them (or perhaps even write new manually) and then apply with new tool afsm. So lets see how it works
As expected results of auto-derived FSM for usermode dlls are much worse - for example on rpcrt4.dll can be found only 76 symbols from 228. It's because code in usermode contains much fewer unique constants (like NTSTATUS or allocation tags in kernel). So we need to use some additional data to make edges more distinguishable. Lets consider several candidates
It seems that MS cut off whole apfnSimpleCall dispatching - no more functions
Let`s see what we can do with our auto-derived state-machines. All source code in my github repo
ldr.exe -se -t 8 -der D:\work\kernel\w10\18346\arm\ntoskrnl.exe 37CC18found at 0076D850 - KdSystemDebugControl ldrb exorted KdDebuggerEnabled ldrbapply return 37CC18, must_be 37CC18
ldr.exe -se -t 8 -der D:\work\kernel\w10\18346\arm\ntoskrnl.exe 37CC18 -T d:\work\kernel\w10\rtm\2004\arm\ntoskrnl.exe ldrb exorted KdDebuggerEnabled ldrbTest[0]: C3F639
// pubsym <rva 0xc3f639> KdLocalDebugEnabledIt`s time to measure how effective this state-machines. I made today simple perl script to measure how much symbols (located in sections .data, ALMOSTRO and PAGEDATA) can be found for arm64 windows kernel. The conditions for success are
Several days ago I made PoC to extract addresses of WSK data from windows 10 arm64 afd.sys - specifically AfdWskClientListHead and lock AfdWskClientSpinLock. Nothing special except fact that afd.sys has no exported functions. So you must find some rare constant, then find functions which use it and only then do some disasm applying state machine to each code block (see lambda passed to traverse_simple_state_graph)
While I was writing this code, I was not left with a question whether it is possible to employ computer to build such state machines. And now I know that this is possible (at least for code on plain C for RISC-like asm with predictable addresses of instructions etc etc)
Lets see how such algo can be arranged:
1) you must find all cross-refs to desired variable and collect list of functions which use it (exactly what deriv_hack::find_xrefs method does)
2) then you must disasm each such function and try to get some primitives - like loading of constants, calling imported/exported functions etc - see deriv_hack::make_path method. Sure set of this primitives will be different for each processor and perhaps will depends from your tasks
Results for afd.sys!AfdWskClientListHead: