crowdstrike

since everyone (including "professional C++ programmers") has already written about so I will also write it down (for memory)

This was not first time: proof

Official report is very vague but we can conclude that root cause was bug in config parser leading to dereference of bad (it's unclear if it contained zero or just was uninitialized) pointer, as I assumed

Well, this is not big deal - I made thousands bsod/kernel panics. There usually long path from developers machine to production, right?

Then I read second report - and this is pure madness. They DIDN'T TESTED IT AT ALL!!! Proof is very simple - lets assume we have bug with probability of bsod P. Then for N test machines probability to not catch this bug is (1-P) ^ N. for P = 0.5 (bsod happens or not, he-he) and 8 test machines this probability 0.4%. Given that in this incident probability of bsod was close to 1 - size of their test cluster is exactly 0

Instead they used some kind of spell-checkar Content Validator. I'm almost sure that they were bitten by adherents of totalitarian eBPF sect with their “20,000 lines of code” verifier (which exists solely to drink blood and ruin lives also unable to prevents cases like this)

Well, this is already serious problems but still not disaster - like you can deploy only for very small percent of users, employ fuse to prevent spreading of malicious update (for example by tracking heartbeats from updated machines) and so on. Right? ...

So right recipe for catastrophe from group of rebels:

• never test anything - static validator is enough
• deploy in Friday
• deploy to all users at the same time
   

moving string literals to discardable sections

As kotest shows we can achieve reduction in the size of non-discardable sections of LKMs (and thus reduce total size of memory occupied by kernel) with moving constants used in functions from .init.text to discardable sections like .init.rodata. As usually we can do it manually or invent some dirty hacks to automate this boring process. And the first thought that comes to mind is to employ

gcc

Lets assume that we have some string literal referred from set of functions F1 ... FN. We can safely move it to discardable section only if all this functions itself located in discardable sections. This leads to two consequences

1. in case of gcc plugin we must operate on RTL bcs some functions can be inlined (or even fully eliminated) - you just cannot do it in GIMPLE
   
2. we need some tracking of cross-references between functions and literals

Unfortunately gcc does not have such tracking - all string literals stored in string pool. To make live even worse sometimes string pool belongs to function. From this comment

Only a few targets need per-function constant pools. Most can use one per-file pool

we can conclude that actually nobody ever understand when per-function constant pools are used. Probably I could patch my gcc plugin to add tracking of string literals, finding candidates for eviction to discardable section and marking them with __section__ attribute. However debugging of RTL gcc-plugins is real nightmare

 

Patching obj files

The next logical step is try to move string literals directly in object files (for example with LIEF). kotest already able to find candidates so the rest is simply to move them, right? Personally I believe that this is possible but there are several problems with this approach:

1. currently LIEF (or binutils) cannot do it
2. we need to patch not only relocs but also DWARF debug info and symbol table - bcs after removing of some string from .rodata addresses of all entries below must be changed

In general patching of binary is not perfect idea, especially if there is good alternative

ebpf map as communication channel

Recently I've done small research to repurpose overvalued ebpf into something useful and even achieved some modest results. It seems that at least you can use ebpf maps in your old-school native drivers without writing single line of code for ebpf progs. You can ask me - c'mon, there are tons of ways to communicate with driver under linux, just to name few:

• ioctls
• read from driver/write to driver + possible employ polling
• file in kernfs
• futex in shared memory
  
• io_uring
  
• netlink like auditd or XFRM do

etc

Let's look at typical situation - your IT crew was bitten by violent adherent of the totalitarian sect of rust ebpf Witnesses and as a consequence you now have hundreds of ebpfs (and no single person who know how this pile of spaghetti code works)

Probably now is much better to integrate new drivers with ebpf, right? Considering that ebpf progs have very serious limitations (like you can't read content of linked lists like raw_notifier_head with right locking) it would be very nice to produce output of your EDR drivers directly to ebpf maps

So I made simple POC to show how you can do it. Source of driver & userland test program

Lets dive into gory details and see what other non-obvious advantages can be obtained from such heretical crossbreeding

dumping ebpf kind_ops

Very funny article about (im)possible future of ebpf. Given that right now 8 small BPF scripts with only 7 opcodes occupy 1Mb whole kernel on ebpf will require exabytes of RAM, he-he

Anyway there is another case of info hiding in linux - mentioned in article struct_ops has type btf_kind_operations and cannot be found in include (as usually). So I add today dumping of it in my lkcd. Sample of output:

frame sizes in dwarfdump

Add today dumping of stack frame sizes to my dwarfdump (well, where they are exists). Format of .debug_frame section obviously was invented by martian misantrophes so patch is huge and ugly

Sample of output for some random function from mips kernel:

// Addr 0x183D27C .text
// Frame Size 18
// FileName: drivers/char/random.c
// LocalVars:
//  LVar0, tag 6965A71
//    int ret
//  LVar1, tag 6965A8F
//    bool branch
ssize_t random_read_iter(struct kiocb* kiocb,struct iov_iter* iter);

and the same in JSON format: 

"110516780":{"type":"function","file":"drivers/char/random.c","type_id":"110427157","name":"random_read_iter","addr":"25416316","section":".text","frame_size":"24","params":[{"name":"kiocb","id":"110516807","type_id":"110466611"},{"name":"iter","id":"110516828","type_id":"110470510"}],"lvars":["110516849":{"type":"var","file":"drivers/char/random.c","owner":"110516780","type_id":"110426600","name":"ret"},
"110516879":{"type":"var","file":"drivers/char/random.c","owner":"110516780","type_id":"110427073","name":"branch"}]}

Unfortunately dwarfdump can't work with kernel modules bcs they are actually just object files and for this sad reason they have relocations even for debug sections. So to properly deal with this files I need to apply relocations first and this is arch-specific action (which I prefer to avoid)

Binutils has 2 solution of this problem:

1. objcopy calls bfd_simple_get_relocated_section_contents from libbfd.so and this means that tool should have dependency from it
2. readelf has it's own relocation code in apply_relocations and this is huge pile of code

And I really don’t like both of the above approaches

function stack size in GCC

Let's continue our wandering in endless dead end

GCC has struct stack_usage and even field su inside struct function. And this last is accessible as global var cfun. So it's should be easy to patch dwarf2out.cc yet one more time for example to extract stack size (like function output_stack_usage_1 do) and put it inside DW_AT_frame_base block, right?

NO

As you can see function allocate_stack_usage_info called only when

• flag_callgraph_info is set - corresponding to option -fcallgraph-info
• flag_stack_usage_info is set - corresponding to options -fstack-usage

In other cases field function->su is zero
There should probably be heartbreaking conclusion about quality of opensource in general and gcc and in particular...

stack frames size in DWARF

As you might suspect, the stack size in the kernel is quite meager so it's very important to know how much of stack occupy your driver. So I conducted inhumane experiments on my own driver lkcd to check if stack frame size can be extracted from DWARF debug info. Biggest function in my driver is lkcd_ioctl so lets explore it

mips

Prolog of lkcd_ioctl looks like:

addiu   sp,sp,-688

 

Lets try to find this number in output of objdump -g