plugin for Binary Ninja

due to the sad fact that IDA Pro moving to cloud (just think about confidentiality) I decided to look at some alternatives - Binary Ninja. First impression was terrible

• totally unknown API, guys - why not make some compatibility layer with IDAPython?
• counterintuitive types in LLIL - for example constant ptr has type RegisterValue. whut?
• I found bug in LLIL types conversion to python types (and suspect it is not alone)

anyway after couple of weeks I was able to write some simple plugin for checking functions who left some linux kernel resource locked. Perhaps it can be remastered for windows kernel too

overhead of eBPF JIT

Lets try to estimate overhead of JIT compiler

I wrote simple perl script - it just counts redundant bytes for several cases:

• pair mov reg, rbp/add reg, imm (total length 7 bytes) can be replaced with lea reg, [rbp-imm] which is only 4 bytes
• pair mov reg, imm/add reg,imm can be replaced with just loading of right address so second instruction can be removed
• add reg, 1/sub reg, 1 (length 4 bytes) can be replaced to inc/dec reg (which has length 3 bytes)

etc etc

Results

jitted eBPF code

I add yesterday disasm for jitted eBPF code. To put it mildly this code is very poor

Every function has 7 bytes of nops in prolog. Comment says that this is for BPF trampoline - well, ok

Lots of code like

 mov eax, 0x1

 cmp r14, 0x2

 jnz 0xc0561497

 xor eax, eax

0xc0561497:

 ...

Somebody - tell them about cmovXX instructions

Lots of code like

mov rdi, 0xffff8fd687f3e000

add rdi, 0x110

and related to get addresses of stack var:

mov rdi, rbp

add rdi, 0xffffffffffffffe0 

Perhaps it would be preferable to use lea rdi, [rbp-XX]

Slow inc/dec:

add r8, 0x1

sub rdi, 0x1

Lots of repeated instructions:
and rdi, 0xfff

and rdi, 0xfff

it's obvious bug

And finally

eBPF on cgroups

the long story short - they are stored in array effective and in list progs in cgroup->bpf

Below I will try to explain boring and dirty details

cgroups

This article says:

hierarchy: a set of cgroups arranged in a tree

so we need to find roots and then just traverse this trees. Roots have type cgroup_root and stored in cgroup_hierarchy_idr (synced with mutex cgroup_mutex). As usually linux lies - lets compare content of  /proc/cgroups:

#subsys_name hierarchy num_cgroups enabled

cpuset 6 1 1

cpu 5 1 1

cpuacct 5 1 1

blkio 4 1 1

memory 2 148 1

devices 9 99 1

freezer 10 1 1

net_cls 7 1 1

perf_event 8 1 1

net_prio 7 1 1

hugetlb 3 1 1

pids 11 103 1

rdma 12 1 1

with what cgroup roots actually located on this machine:

[0]  at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144

[1] systemd at 0xffff8fd6816ea000 flags 4 hierarchy_id 1 nr_cgrps 145 real_cnt 144

[2]  at 0xffff8fd68297a000 flags 0 hierarchy_id 2 nr_cgrps 148 real_cnt 147

[3]  at 0xffff8fd68297c000 flags 0 hierarchy_id 3 nr_cgrps 1 real_cnt 0

[4]  at 0xffff8fd682978000 flags 0 hierarchy_id 4 nr_cgrps 1 real_cnt 0

[5]  at 0xffff8fd68297e000 flags 0 hierarchy_id 5 nr_cgrps 1 real_cnt 0

[6]  at 0xffff8fd6854c8000 flags 0 hierarchy_id 6 nr_cgrps 1 real_cnt 0

[7]  at 0xffff8fd6854ce000 flags 0 hierarchy_id 7 nr_cgrps 1 real_cnt 0

[8]  at 0xffff8fd6854ca000 flags 0 hierarchy_id 8 nr_cgrps 1 real_cnt 0

[9]  at 0xffff8fd6854cc000 flags 0 hierarchy_id 9 nr_cgrps 99 real_cnt 98

[10]  at 0xffff8fd685e16000 flags 0 hierarchy_id 10 nr_cgrps 1 real_cnt 0

[11]  at 0xffff8fd685e12000 flags 0 hierarchy_id 11 nr_cgrps 103 real_cnt 102

[12]  at 0xffff8fd685e14000 flags 0 hierarchy_id 12 nr_cgrps 1 real_cnt 0

can you find in /proc/cgroups roots with hierarchy ID 0 and 1?

How to traverse this tree? It starts in field cgrp->self and we can use  functions css_next_descendant_pre/css_next_descendant_post etc. Strictly speaking they return pointer to cgroup_subsys_state but this is first field self  in cgroup so casting is safe

slides from our talk at Black Hat EU 2021

link

and some

afterword

all presented attacks caused by misuse of Windows logging mechanism for ETW-based EDRs. And I see bad sign when the same thing happens with eBPF on Linux. So who knows - maybe my next paper will be called "blinding eBPF-based EDRs on Linux" :-)

blinding sysmon for linux

 Let`s see which tracepoints it using:

sudo ./lkmem -d -c -t ~/krnl/curr ~/krnl/System.map-5.11.0-37-generic

 __tracepoint_sched_process_exit at 0xffffffffa47140c0: enabled 1 cnt 1

  [0] 0xffffffffa2ed3b40 - kernel!perf_trace_sched_process_template

 __tracepoint_sys_exit at 0xffffffffa4714ae0: enabled 1 cnt 1

  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc

  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc

  [0] 0xffffffffa2f37f90 - kernel!__bpf_trace_sys_exit

 __tracepoint_sys_enter at 0xffffffffa4714b40: enabled 1 cnt 1

  regfunc: 0xffffffffa2fa3350 - kernel!syscall_regfunc

  unregfunc: 0xffffffffa2fa3410 - kernel!syscall_unregfunc

  [0] 0xffffffffa2f37e30 - kernel!__bpf_trace_sys_enter

1. my favorite 1bit patch - zero tracepoint->key.enabled
2. remove BPF client from funcs list
3. find trace_event_call and install your own event_filter

BPF iterators

Sure I could not get past the hype topic of BPF (overvalued mechanism to allow you just run your buggy code in kernel with low performance and lots of overhead). For access of some kernel data they add so called iterators - and maybe you even can write your own and register it with bpf_iter_reg_target (spoiler: you can`t, bcs this function is not exported. Welcome to wonderful world of open-source with unexplained and unreasonable restrictions). I was curious what BPF iterators are in the system - they stored iterators in list targets synchronized with mutex targets_mutex. It would seem what could go wrong? 

grep " targets" System.map-5.11.0-37-generic

ffffffff820ff8e0 r targets

ffffffff826e1240 d targets_mutex

ffffffff826e1260 d targets

ffffffff8277a5c0 d targets

ffffffff8286b2e8 d targets_supported

In this case, we are dealing with another mechanism for hiding information in linux kernel - using of non-unique names. I was not even lazy and wrote a script to count such names - 998 names. Top 5:

_acpi_module_name: 155

cpumask_weight.constprop.0: 47

kzalloc.constprop.0: 39

get_order: 32

kmalloc_array.constprop.0: 28

As usual the disassembler rushes to the rescue