среда, 1 декабря 2021 г.

jitted eBPF code

I add yesterday disasm for jitted eBPF code. To put it mildly this code is very poor

Every function has 7 bytes of nops in prolog. Comment says that this is for BPF trampoline - well, ok

Lots of code like

 mov eax, 0x1
 cmp r14, 0x2
 jnz 0xc0561497
 xor eax, eax
0xc0561497:
 ...
Somebody - tell them about cmovXX instructions

Lots of code like
mov rdi, 0xffff8fd687f3e000
add rdi, 0x110

Lots of repeated instructions:
and rdi, 0xfff
and rdi, 0xfff

it's obvious bug

And finally

четверг, 25 ноября 2021 г.

eBPF on cgroups

the long story short - they are stored in array effective and in list progs in cgroup->bpf
Below I will try to explain boring and dirty details

cgroups

This article says:
hierarchy: a set of cgroups arranged in a tree
so we need to find roots and then just traverse this trees. Roots have type cgroup_root and stored in cgroup_hierarchy_idr (synced with mutex cgroup_mutex). As usually linux lies - lets compare content of  /proc/cgroups:
#subsys_name hierarchy num_cgroups enabled
cpuset 6 1 1
cpu 5 1 1
cpuacct 5 1 1
blkio 4 1 1
memory 2 148 1
devices 9 99 1
freezer 10 1 1
net_cls 7 1 1
perf_event 8 1 1
net_prio 7 1 1
hugetlb 3 1 1
pids 11 103 1
rdma 12 1 1

with what cgroup roots actually located on this machine:

[0]  at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144
[1] systemd at 0xffff8fd6816ea000 flags 4 hierarchy_id 1 nr_cgrps 145 real_cnt 144
[2]  at 0xffff8fd68297a000 flags 0 hierarchy_id 2 nr_cgrps 148 real_cnt 147
[3]  at 0xffff8fd68297c000 flags 0 hierarchy_id 3 nr_cgrps 1 real_cnt 0
[4]  at 0xffff8fd682978000 flags 0 hierarchy_id 4 nr_cgrps 1 real_cnt 0
[5]  at 0xffff8fd68297e000 flags 0 hierarchy_id 5 nr_cgrps 1 real_cnt 0
[6]  at 0xffff8fd6854c8000 flags 0 hierarchy_id 6 nr_cgrps 1 real_cnt 0
[7]  at 0xffff8fd6854ce000 flags 0 hierarchy_id 7 nr_cgrps 1 real_cnt 0
[8]  at 0xffff8fd6854ca000 flags 0 hierarchy_id 8 nr_cgrps 1 real_cnt 0
[9]  at 0xffff8fd6854cc000 flags 0 hierarchy_id 9 nr_cgrps 99 real_cnt 98
[10]  at 0xffff8fd685e16000 flags 0 hierarchy_id 10 nr_cgrps 1 real_cnt 0
[11]  at 0xffff8fd685e12000 flags 0 hierarchy_id 11 nr_cgrps 103 real_cnt 102
[12]  at 0xffff8fd685e14000 flags 0 hierarchy_id 12 nr_cgrps 1 real_cnt 0

can you find in /proc/cgroups roots with hierarchy ID 0 and 1?

How to traverse this tree? It starts in field cgrp->self and we can use  functions css_next_descendant_pre/css_next_descendant_post etc. Strictly speaking they return pointer to cgroup_subsys_state but this is first field self  in cgroup so casting is safe

четверг, 11 ноября 2021 г.

slides from our talk at Black Hat EU 2021

link

and some

afterword

all presented attacks caused by misuse of Windows logging mechanism for ETW-based EDRs. And I see bad sign when the same thing happens with eBPF on Linux. So who knows - maybe my next paper will be called "blinding eBPF-based EDRs on Linux" :-)

пятница, 15 октября 2021 г.

blinding sysmon for linux

 Let`s see which tracepoints it using:


sudo ./lkmem -d -c -t ~/krnl/curr ~/krnl/System.map-5.11.0-37-generic
 __tracepoint_sched_process_exit at 0xffffffffa47140c0: enabled 1 cnt 1
  [0] 0xffffffffa2ed3b40 - kernel!perf_trace_sched_process_template
 __tracepoint_sys_exit at 0xffffffffa4714ae0: enabled 1 cnt 1
  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc
  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc
  [0] 0xffffffffa2f37f90 - kernel!__bpf_trace_sys_exit
 __tracepoint_sys_enter at 0xffffffffa4714b40: enabled 1 cnt 1
  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc
  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc
  [0] 0xffffffffa2f37e30 - kernel!__bpf_trace_sys_enter

  1. my favorite 1bit patch - zero tracepoint->key.enabled
  2. remove BPF client from funcs list
  3. find trace_event_call and install your own event_filter

понедельник, 11 октября 2021 г.

BPF iterators

Sure I could not get past the hype topic of BPF (overvalued mechanism to allow you just run your buggy code in kernel with low performance and lots of overhead). For access of some kernel data they add so called iterators - and maybe you even can write your own and register it with bpf_iter_reg_target (spoiler: you can`t, bcs this function is not exported. Welcome to wonderful world of open-source with unexplained and unreasonable restrictions). I was curious what BPF iterators are in the system - they stored iterators in list targets synchronized with mutex targets_mutex. It would seem what could go wrong? 

grep " targets" System.map-5.11.0-37-generic
ffffffff820ff8e0 r targets
ffffffff826e1240 d targets_mutex
ffffffff826e1260 d targets
ffffffff8277a5c0 d targets
ffffffff8286b2e8 d targets_supported

In this case, we are dealing with another mechanism for hiding information in linux kernel - using of non-unique names. I was not even lazy and wrote a script to count such names - 998 names. Top 5:

_acpi_module_name: 155
cpumask_weight.constprop.0: 47
kzalloc.constprop.0: 39
get_order: 32
kmalloc_array.constprop.0: 28

As usual the disassembler rushes to the rescue

понедельник, 4 октября 2021 г.

security hooks in linux kernel

This mechanism was inspired by NSA. As described all hooks stored in huge struct security_hooks_list, but it`s format is different in each version. We can determine which list belongs to what hook with disasm magic. Lets see function that calls security hooks - for example security_path_chown:

.text:FFFFFFC010496448 security_path_chown        ; CODE XREF: chown_common+104↑p
.text:FFFFFFC010496448   STP             X29, X30, [SP,#-0x18+var_18]!
.text:FFFFFFC01049644C   MOV             X29, SP
.text:FFFFFFC010496450   STP             X20, X21, [SP,#0x18+var_s0]
.text:FFFFFFC010496454   STR             X22, [SP,#0x18+var_s10]
.text:FFFFFFC010496458   MOV             X20, path
.text:FFFFFFC01049645C   MOV             W21, W1
.text:FFFFFFC010496460   MOV             path, X30
.text:FFFFFFC010496464   MOV             W22, W2
.text:FFFFFFC010496468 loc_FFFFFFC010496468  ; DATA XREF: .init.data:FFFFFFC0111474C0↓o
.text:FFFFFFC010496468   BL              _mcount
.text:FFFFFFC01049646C   LDR             X0, [path,#8]
.text:FFFFFFC010496470   LDR             X0, [X0,#0x30]
.text:FFFFFFC010496474   LDR             W0, [X0,#0xC]
.text:FFFFFFC010496478   TBNZ            W0, #9, loc_FFFFFFC0104964B4
.text:FFFFFFC01049647C   ADRP            X0, #security_hook_heads_0.path_chown@PAGE
.text:FFFFFFC010496480   STR             X19, [X29,#0x18+var_8]
.text:FFFFFFC010496484   LDR             X19, [X0,#security_hook_heads_0.path_chown@PAGEOFF]


In disasm we just search for first reference to memory near address of security_hook_heads. Some results:

воскресенье, 3 октября 2021 г.

what linux hiding

disclaimer
there is no doubt that the list below is incomplete, inaccurate etc - it`s just what very average programmer can find during two month of browsing linux source code

observability criteria
what I mean under "hiding"? It means that
  • no kernel API to enumerate some structure
  • no real-time notifications about setting some hook
  • no mapping on /proc or /sys (however this method is not reliable)
  • no 3rd party tools to show this. As an example I chose volatility - just bcs I readed their folio "The Art of Memory Forensics"
So you unable to see them

notification chains
very ironic that they have API like register_XXX_notifier/unregister_XXX_notifier and there is no function like enum_XXX_notifier
no mapping on /proc or /sys
volatility checks only very limited set - vt_notifier_list & keyboard_notifier_list

tracepoints
no API to enum clients
no notification about turning on some tracepoint
has mapping to /sys/kernel/tracing/events but can`t show clients of some tracepoint
volatility - no

kprobes
no API to enum consumers of some installed KPROBE
no notification about installing new kprobe. This is an extremely sad fact - for example tools like LKRG don`t knows that some memory was patched
has mapping to /sys/kernel/debug/kprobes/
volatility - no

uprobes
no API to enum consumers of some installed UPROBE
no notification about installing new uprobe
has mapping to /sys/kernel/debug/tracing/uprobe_events. Most crazy thing is that uprobes installed from kernel not shown
volatility - no

filesystem notifications
no API to enum all installed marks
for usermode events has notification via security_path_notify, for kernelmode - absolutely not
has very limited mapping to /proc/*/fdinfo/*. Again marks installed from kernel not shown
volatility - no