linux kernel tracing

It`s hard to believe but linux kernel has almost exact copy of windows ETW - event tracing. It is just as difficult to make it work, it is poorly documented, very complex and fragile. And yes, as you can guess - it also can`t show who and which parts of it in use. So I wrote some code to dump registered funcs in tracepoints and to check file ops for files in /sys/kernel/tracing/events

Lets start with tracepoints. As you see this structure has strange looked list of functions in field funcs, and calling happens in functions like event_triggers_call. How we can find this tracepoints? Well,  they stored in trace_event_call->tp and array of pointers to trace_event_call located between symbols __start_ftrace_events & __stop_ftrace_events. Unfortunately all this treasures located in discardable section .init.data. But because they were all declared in the same manner we can find them by name - all symbols with prefix __tracepoint_ is what we need. So some examples (you can run lkmem -c -t vmlinux system.map to get this):

 __tracepoint_sys_enter at 0xffffffff8b82e340: enabled 0 cnt 0
  regfunc 0xffffffff8a192330 - kernel!syscall_regfunc
  unregfunc 0xffffffff8a1923f0 - kernel!syscall_unregfunc

Well, no clients right now - cnt 0

Next about /sys/kernel/tracing/events files (this is perverted inhuman interface to manage trace events). I just dumping file->f_path.dentry->d_inode->i_fop for each such file. Sample of output (you can achieve this with lkmem -s vmlinux system.map path_to_some_sys_kernel_tracing_file):

arm64 disasm for linux kernel

I added today disassembler for arm64 linux kernel to search pointers. It turned out to be surprisingly difficult to do for several reasons (disasm for x64 is only 383 LOC vs 618 for arm64)

One of them is poor code produced by some gcc versions

But the main problem is arm64 opcodes. Lets see simple indirect call:

  ADRP            X27, #mh_filter@PAGE

  CMP             W22, #0x3A ; ':'

  B.EQ            loc_FFFFFFC010CC7140

  CMP             W22, #0x87

  B.NE            loc_FFFFFFC010CC7188

  LDR             X2, [X27,#mh_filter@PAGEOFF]

  CBZ             X2, loc_FFFFFFC010CC7188

  MOV             X1, skb

  MOV             X0, X28

  BLR             X2

    

compare this with code to call list of funcs from tracepoints:

  ADRP            __data, #__tracepoint_cpu_idle@PAGE

  ADD             X0, X0, #__tracepoint_cpu_idle@PAGEOFF

  MOV             X29, SP

  STR             X19, [SP,#var_s10]

  LDR             X19, [X0,#(__tracepoint_powernv_throttle.funcs - 0xFFFFFFC011A562C0)]

 ...

loc_FFFFFFC01011FC60:

  LDR             X4, [X19]

  MOV             W3, W20

  LDR             X0, [X19,#8]

  MOV             X2, X21

  MOV             W1, W22

  BLR             X4

  LDR             X0, [X19,#0x18]!

  CBNZ            X0, loc_FFFFFFC01011FC60

In second case register X4 was loaded from X19, which in turn was loaded from some memory, so I need to track how many times content of register was loaded

Anyway results is +34 newly discovered functions pointers

functions pointers in linux kernel data sections

I wrote simple program to estimate size of problem. Yes, I know about CFI but it seems that even on kernel 5.11 on fresh Ubuntu this mechanism is not implemented and indirect calls looks like:

  mov     rax, cs:XXX

  call    __x86_indirect_thunk_rax

__x86_indirect_thunk_rax proc near: 

  jmp     rax

First approach is just to scan .data section - you can do this running

./lkmem path-to-unpacked-kernel path-to-System.map

Some results:

• arm64 5.11.0: 9893
• x64 5.8-53: 10698
• x64 5.11.0: 13414
• x64 4.18: 16224

Ok, how about not yet inited pointers (or pointers in .bss section)? We need use disassembler - just disasm all functions in .text and find indirect calls and calls to __x86_indirect_thunk_XXX. Results (with -d option):

• x64 4.18: +42
• x64 5.8-53: +52
• x64 5.11.0: +45

and with .bss section (option -b):

• x64 4.18: +99
• x64 5.8-53: +120
• x64 5.11.0: +109

dumper of linux kernel notification chains

There seems to be one little-known thing in linux kernel - notification chains. So they have literal analogue of PsSetLoadImageNotifyRoutine - function register_module_notifier. And similarly they don't have a function to enumerate registered notifications - I don`t know why. Maybe they were bitten by Microsoft. Or maybe I want too much from people whose even "The Linux Kernel Module Programming Guide" contains an error in the code example. Anyway I decided to write my own (btw the last time I wrote drivers for Linux was something around 20 years ago)

How to run

git clone https://github.com/redplait/lkcd.git

cd lkcd

make

sudo insmod ./lkcd.ko

cd test

make

sudo ./dtest

Sample of output (from fresh Ubuntu):

codewars heisenbug

 I got following crash when tried to solve some trivial task:

UndefinedBehaviorSanitizer:DEADLYSIGNAL ==1==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000020 (pc 0x0000004271a4 bp 0x7ffcf51d9a28 sp 0x7ffcf51d9070 T1) ==1==The signal is caused by a READ memory access. ==1==Hint: address points to the zero page. ==1==WARNING: invalid path to external symbolizer! ==1==WARNING: Failed to use and restart external symbolizer! #0 0x4271a3 (/workspace/test+0x4271a3) #1 0x4276d0 (/workspace/test+0x4276d0) #2 0x4273f0 (/workspace/test+0x4273f0) #3 0x427eed (/workspace/test+0x427eed) #4 0x4282b9 (/workspace/test+0x4282b9) #5 0x42abe3 (/workspace/test+0x42abe3) #6 0x4295ce (/workspace/test+0x4295ce) #7 0x429129 (/workspace/test+0x429129) #8 0x428d1b (/workspace/test+0x428d1b) #9 0x43b625 (/workspace/test+0x43b625) #10 0x42810d (/workspace/test+0x42810d) #11 0x7fdeedfdabf6 (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6) #12 0x405339 (/workspace/test+0x405339)

ok, nothing special, just dereferencing zero ptr. But where? code looks like:

double eval(const std::shared_ptr<ASTNode> &tree) {

    if ( !tree )

      return 0.0;

    switch(tree->token.type) {

At least we have address of crash - lets dump first 0x40 bytes of eval function:

   unsigned char *c = (unsigned char *)&eval;

   for ( int i = 0; i < 0x40; i++ )

     printf("%2.2X ", *(c + i));

And then put them in 64bit disasm:

000000000000000d 55               push rbp

000000000000000e 4157             push r15

0000000000000010 4156             push r14

0000000000000012 53               push rbx

0000000000000013 4883ec18         sub rsp, 0x18

0000000000000017 488b37           mov rsi, [rdi]

000000000000001a 660f57c0         xorpd xmm0, xmm0

000000000000001e 4885f6           test rsi, rsi

0000000000000021 0f849d020000     jz dword 0x2c4

0000000000000027 8b4620           mov eax, [rsi+0x20] ; crash is here

WHAT? how rsi can be zero if it passed check test rsi, rsi? Is it buggy qemu, docker or some speculative read-ahead or what is it? I doubt if I wish continue use this service

ecdsa in driver

Lets assume that we have buggy and dangerous driver (which "rely on many unexported functions and select them via pattern scans which are regularly revalidated against windows insider builds", he-he). Sure we want restrict access to it, for example like ProcessHacker do

Unfortunately the latter uses CNG and cannot work on xp/w2k3. So I made fork of libecc to use this library with WDK7. Test driver and client also included

How to build user-mode part

I commited VS2017 project files for library, ec_utils and test client - they located in directory vs.

Next you must sign your client:

Generate your keys (constants BRAINPOOLP512R1, ECRDSA and SHA3_512 hardcoded in driver - sure you can use what you want):
ec_utils.exe gen_keys BRAINPOOLP512R1 ECRDSA mykeypair

and sign your client 
ec_utils.exe sign BRAINPOOLP512R1 ECRDSA SHA3_512 testclnt.exe mykeypair_private_key.bin testclnt.sig

now copy file mykeypair_public_key.h to directory drv

Also you need convert file testclnt.sig to 1.inc to driver source code - I am too lazy to read signatures from registry so they hardcoded in driver body

How to build driver

Launch right "Build Environment" from WDK7, Makefile for library located in directory src and Makefile for driver in directory drv. I hope you know what to do with them

Run

You will need admin privileges, at first install driver

testclnt.exe full_path2_ecdsadrv.sys

and just run

testclnt.exe

If you were careful enough with the signatures you can see something like:
IOCTL_TEST_IOCTL return 1

This means that driver checked EC DSA of your testclnt.exe and now agree to work with it. Sure you can have several trusted clients - just change ALLOWED_CLIENTS in vrfy.c and init each client with right signature

And finally when you have enough playing you can uninstall driver:

testclnt.exe -u

poorgcc: IDA Pro plugin to fix poor gcc code on arm64

Lets see what generates gcc for arm64 - for example gcc7.5 and linux kernel
Function do_sysinstr:

ADRP            X0, #__func__.48604@PAGE ; "arm64_show_signal"

ADD             X0, X0, #__func__.48604@PAGEOFF

ADRP            X3, #ctr_read_handler@PAGE

ADD             X0, X0, #0x218

ADD             X3, X3, #ctr_read_handler@PAGEOFF

Wtf happened here? Instead of loading x0 with address of sys64_hooks we have two consecutive loads and no value x0 used between. You can peek some random functions - this is very common pattern, I personally think this is bug in gcc arm64 codegen. Anyway, it does not allow to see right xrefs so I wrote simple plugin for IDA Pro to fix this

Plugin just try to find instructions "add add reg, reg, imm" without data xref and backtrack if this register was loaded somewhere above - sure code is not sample of elegance. You can add to plugins.cfg string like this

process_all_poor_gcc_functions    poorgcc64     0      1

to process all functions

Some results - after applying plugin to function do_sysinstr code looks like:

ADRP            X0, #__func__.48604@PAGE ; "arm64_show_signal"

ADD             X0, X0, #__func__.48604@PAGEOFF

ADRP            X3, #ctr_read_handler@PAGE

ADD             X0, X0, #0x218 ; FFFFFFC010C116C8

ADD             X3, X3, #ctr_read_handler@PAGEOFF

FFFFFFC010C116C8 is address of sys64_hooks and now it has right xref