linux drivers cross-compilation

Just reminder for myself how to build driver for arm64 having x64 based machine with ubuntu

Install right gcc

for arm64 we need gcc-aarch64-linux-gnu:

sudo apt-get install gcc-aarch64-linux-gnu

Build Kernel

You cannot use installed kernel and must build one for appropriate architecture - in my case for arm64 (note - gcc has prefix aarch64, C - consistency). Clone or unpack kernel source tree to some directory KROOT and then

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules

Patch Makefile

Usual trick is to use something like

MACHINE ?= $(shell uname -m)
ifeq ($(MACHINE),x86_64)

but this gives you arch of host machine, so you must rewrite all such cases to use ARCH variable (and to setup make -C $(KROOT))

Building

and finally you can cross-compile your driver with something like

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -f Makefile.arm64

BTI incompatible exported functions in kernel 5.15.0-53

if BTI is enabled, the first instruction encountered after an indirect jump must be a special BTI instruction

from here

I downloaded Ubuntu for arm64 (jammy-desktop-arm64.iso) and decided to check if there are some functions with don`t contain BTI c at start

17804 such functions. System.map-5.15.0-53-generic contains 62819 functions in total. Next I just intersected them with exported - 1269

This is obvious bug - maybe in gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0

at least some of this functions are really important - like register_ftrace_function

dirty secrets of ld.so

As you can know you can set library path under linux with several ways:

• envvar LD_LIBRARY_PATH, but it can be removed somewhere inside program so /proc/pid/environ is useless (as usually they expose via official API only useless trash but carefully hiding any really important things)
• via option --library-path to ld.so - like /lib64/ld-linux-x86-64.so.2 --library-path path someprogram Again command line can be patched
• via /etc/ld.so.conf - this file also can be patched after your program was launched

So good question is "is there some trusted source to see what library path was installed for some running program?"

Yes, this is ld,so itself - because it uses this data while dynamically loading some modules, So long story short: value from --library-path & LD_LIBRARY_PATH stored in variable library_path and whole directory set in rtld_search_dirs

Bad news - they are not exported and even worse - they are hard to find even using disassembler

PoC to blind pamspy

Lets disasm jit code from this spyware:

 [8] prog 0xffffb02dc0133000 id 160 len 46 jited_len 215 aux 0xffff8ccb58fab400 used_maps 1 used_btf 0 func_cnt 0

     tag: 0F 86 19 76 BC 37 68 B3

  stack_depth: 16

  num_exentries: 0

  type: 2 BPF_PROG_TYPE_KPROBE

  expected_attach_type: 0 BPF_CGROUP_INET_INGRESS

  used maps:

   [0] 0xffff8ccbc1b1c600 - rb

...

ffffffffc07bc801 e80a38e6f1  call 0xffffffffb2620010 ; bpf_ringbuf_submit

ffffffffc07bc806 31c0        xor eax, eax

ffffffffc07bc808 415e        pop r14

ffffffffc07bc80a 415d        pop r13

ffffffffc07bc80c 5b          pop rbx

ffffffffc07bc80d c9          leave

ffffffffc07bc80e c3          ret

and in ebpf opcodes:

43 85 00 00 00 C0 CF 02 00 call 0x2CFC0 ; bpf_ringbuf_submit

44 B7 00 00 00 00 00 00 00 mov r0, 0

45 95 00 00 00 00 00 00 00 ret

Here 0x2CFC0 is offset to bpf_ringbuf_submit from __bpf_call_base

The last call submit some data to bpf map rb with type BPF_MAP_TYPE_RINGBUF. If we could patch this function no data will be passed to usermode. How are these native function addresses filled in at all?

size of ebpf jit code on different processors

it doesn't make much sense but bcs I have now several jit compilers - why not compare how much size have jitted code for different processors?

I chose 3 ebpf programs

1. simple BPF_PROG_TYPE_CGROUP_SKB with only comparison, 8 opcodes
2. BPF_PROG_TYPE_RAW_TRACEPOINT with 3 maps, 68 opcodes
3. enough complex BPF_PROG_TYPE_RAW_TRACEPOINT with 6 maps, 1824 opcodes

results

processor	1st	2nd	3rd
x64	54	312	8195
arm64	99	567	12959
powerpc	78	546	11462
risc-v	102	470	9494
s390	78	534	12622
sparc	79	482	10446

verification of jitted ebpf code

There are some projects for ebpf in usermode, but for verification purposes you need the same code which was used in kernel. So I ripped out some jit code to run it in usermode

• x64
• powerpc
• risc-v
• s390
• sparc
• sunway sw64

And now we can make verification of jitted code - we have actual generated code for some ebpf, next we run JIT for ebpf opcodes in usermode, and finally can compare them

pmu events

Some details

pmu stored in tree pmu_idr and synced with mutex pmus_lock. and as usually can be used to blind EBPF. How? Lets see:

General speaking there are usually four steps involved to attach an eBPF program to a perf event:

1. Open the perf event
2. Load the eBPF program
3. Set the eBPF program on the perf event
4. Enable the perf event

We interested in point 4 - enabling of the perf event involves calling of pmu->event_init & pmu->add methods. And worse - all pmu structures located in .data section and thus writable. So I add today some code to dump them: