slides from our talk at Black Hat EU 2021

link

and some

afterword

all presented attacks caused by misuse of Windows logging mechanism for ETW-based EDRs. And I see bad sign when the same thing happens with eBPF on Linux. So who knows - maybe my next paper will be called "blinding eBPF-based EDRs on Linux" :-)

linux kernel tracing

It`s hard to believe but linux kernel has almost exact copy of windows ETW - event tracing. It is just as difficult to make it work, it is poorly documented, very complex and fragile. And yes, as you can guess - it also can`t show who and which parts of it in use. So I wrote some code to dump registered funcs in tracepoints and to check file ops for files in /sys/kernel/tracing/events

Lets start with tracepoints. As you see this structure has strange looked list of functions in field funcs, and calling happens in functions like event_triggers_call. How we can find this tracepoints? Well,  they stored in trace_event_call->tp and array of pointers to trace_event_call located between symbols __start_ftrace_events & __stop_ftrace_events. Unfortunately all this treasures located in discardable section .init.data. But because they were all declared in the same manner we can find them by name - all symbols with prefix __tracepoint_ is what we need. So some examples (you can run lkmem -c -t vmlinux system.map to get this):

 __tracepoint_sys_enter at 0xffffffff8b82e340: enabled 0 cnt 0
  regfunc 0xffffffff8a192330 - kernel!syscall_regfunc
  unregfunc 0xffffffff8a1923f0 - kernel!syscall_unregfunc

Well, no clients right now - cnt 0

Next about /sys/kernel/tracing/events files (this is perverted inhuman interface to manage trace events). I just dumping file->f_path.dentry->d_inode->i_fop for each such file. Sample of output (you can achieve this with lkmem -s vmlinux system.map path_to_some_sys_kernel_tracing_file):

etw part 4½: MCGEN_TRACE_CONTEXT

let's continue to dissect ETW (parts 1,2, 3 & 4)

Now consider structures generated with mc.exe (Windows Message Compiler). It seems that this is very old technology - some .mc files in official Microsoft github repository have copyrights from 1992! Despite this they are still supported for example in MSBuild - see rule for MessageCompile

This generated with mc structure has name MCGEN_TRACE_CONTEXT and looks like:

typedef struct _MCGEN_TRACE_CONTEXT

{

    HANDLE                 RegistrationHandle;

    HANDLE                 Logger;

    ULONGLONG              MatchAnyKeyword;

    ULONGLONG              MatchAllKeyword;

    ULONG                  Flags;

    ULONG                  IsEnabled;

    UCHAR                  Level; 

    UCHAR                  Reserve;

    USHORT                 EnableBitsCount;

    PULONG                 EnableBitMask;

    const ULONGLONG*       EnableKeyWords;

    const UCHAR*           EnableLevel;

} MCGEN_TRACE_CONTEXT, *PMCGEN_TRACE_CONTEXT;

Looks very similar to _TlgProvider_t. Unfortunately they cannot be found with some simple signatures scan - you need to use some disasm magic. I wrote simple PoC to find them in arm64 windows kernel

Let`s see where you can encounter this ancient variant of ETW

etw part 4: _TlgProvider_t in kernel

let's continue to dissect ETW (parts 1,2 & 3)
Basically structure _TlgProvider_t in kernel almost the same as in user mode but field RegHandle points to ETW_REG_ENTRY. You can easily find them using simple search for known guids - I made PoC for arm64 (and for ndis.sys too)

kernel contains following tlg providers:

• Microsoft.Windows.TlgAggregateInternal, GUID 703FCC13-B66F-5868-DDD9-E2DB7F381FFB
• KernelExecutive, GUID 8944A53C-A561-4E53-A0C6-D565414745FC
• Microsoft.Windows.Kernel.BootEnvironment, GUID 23B76A75-CE4F-56EF-F903-C3A2D6AE3F6B
• MSTelCov, GUID 1DD9B8C9-E078-4075-B9DE-4E5125071A18
• KernelProcess, GUID 2839FF94-8F12-4E1B-82E3-AF7AF77A450F
• KernelGeneral, GUID 7614521C-4D0B-4341-BFC9-873082C0F1D3
• Microsoft-Windows-Kernel-Vm, GUID B7FBD4E0-FA8F-4C58-B0FB-3CC227B86ED6. Located in section ALMOSTRO
• Microsoft.Windows.Kernel.Security, GUID 09A69A38-2680-4BFA-AD01-792AD63A4FF2
• Microsoft.Windows.Security.Capabilities, GUID 27A8FDF4-9B77-575B-BE3B-E7163EF159BB
• Microsoft.Windows.Kernel.ProcessSubsystem, GUID C59673D8-B796-58DF-FBF8-A70BAD656DCA
• Microsoft.Windows.Kernel.Ttm, GUID 5E753E4D-2B0D-4451-B8F9-0F1253CA0B44. Located in section PAGEDATA
• Microsoft.Windows.Kernel.Power.PowerTransitions, GUID 050BF899-DA06-4852-A63A-81E6B9A1C74F
• Microsoft.Windows.Kernel.Power.DirectedDrips, GUID 0D2ED727-38A0-4B2B-9F7E-EC79B5EC4AA5. Located in section PAGEDATA
• Microsoft.Windows.Kernel.Power, GUID 63BCA7A1-77EC-4EA7-95D0-98D3F0C0EBF7
• Microsoft.Windows.Kernel.Power.DiagFxAccounting, GUID 57D04B7B-550A-49A2-ABCC-A7FA15598A30
• Microsoft.Windows.Kernel.ObjectManager, GUID F39412D1-C9FD-5E79-8A82-9C9CBD8CA809
• Microsoft-Windows-Kernel-Mm, GUID 7E9E8B9C-406C-5D73-E566-0F50EA3ADE3E
• Microsoft.Windows.Kernel.Kernel, GUID 061C37C3-1363-5C1B-B8ED-F3D8F74633CE
• Microsoft.Windows.Kernel.PnP, GUID 6C0EBBBB-C292-457D-9675-DFCC1C0D58B0
• Microsoft.Windows.Kernel.DeviceConfig, GUID C8BDE9FF-F31F-59DC-6C27-CA37C516ADA5
• Microsoft.Windows.Kernel.SysEnv, GUID A9FDF37B-D72D-4051-A3CD-D422103CE079
• Microsoft.Windows.Kernel.LiveDump, GUID A4D16FC5-D1CF-4D72-A055-25F3EB02A70E
• Microsoft.Windows.Kernel.Dump, GUID A51EE86B-8EA5-454C-9A7D-37B6655A535D
• IumTelemetryProvider, GUID 73A33AB2-1966-4999-8ADD-868C41415269
• Microsoft.Windows.Containers.RegistryVirtualization, GUID 252D9ECC-1C9F-4917-8760-F872A83BF018
• Microsoft.Windows.Kernel.FeatureConfigurationManager, GUID F7E83426-2B81-58F9-C5D4-F2DB6D0AD473
• Microsoft.Windows.Kernel.Registry, GUID E9EAF418-0C07-464C-AD14-A7F353349A00
• Microsoft.Windows.FileSystem.Cache, GUID 74093E1D-DBE3-4019-B97D-54EDCB02CFED
• Microsoft.Windows.Kernel.HAL, GUID 0F51C5A7-0E76-47A5-BEDE-7CF62C5822F6

So what bad guys can do knowing addresses of this _TlgProvider_t?

etw tracing handles in kernel

let's continue to dissect ETW (part 1 & 2). This time consider how tracing is implemented in the kernel itself. I made PoC to find all tracing handles in arm64 kernel and now give short explanation of what each of them is used for

EtwpEventTracingProvRegHandle

GUID B675EC37-BDB6-4648-BC92-F3FDC74D3CA2 (EventTracingProvGuid). Used in lots of internal etw related functions like EtwpTraceStackWalk, EtwpWriteUserEvent, EtwpFailLogging, NtTraceEvent, WmiTraceMessage, EtwWriteEx, EtwWrite etc

EtwKernelProvRegHandle

GUID A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D (KernelProvGuid). Used for kernel tracing in functions like SeLogAccessFailure, CmpReorganizeHive, SepSetTokenUserAndGroups, EtwTraceSystemTimeChange, EtwTraceTimeZoneInformationRefresh etc

EtwpPsProvRegHandle

GUID 22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716 (PsProvGuid). Used for tracing processes subsystem activity in functions like PsImpersonateContainerOfThread, PspRevertContainerImpersonation, PspSetJobIoRateControl, PspSetJobIoAttribution, EtwTraceFreezeThawProcess etc

EtwpNetProvRegHandle

GUID 7DD42A49-5329-4832-8DFD-43D979153A88 (NetProvGuid). Used only in function EtwpTraceNetwork which assigning as callback in WmiSetNetworkNotify

_TlgProvider_t

let's continue to dissect ETW and consider one of the many usermode tracing structures - _TlgProvider_t. It is even officially documented in platform sdk in header TraceLoggingProvider.h (sample of using):
struct _TlgProvider_t
{
    UINT32 LevelPlus1;
    UINT16 const UNALIGNED* ProviderMetadataPtr; // Points to the RemainingSize member of provider metadata.
    ULONGLONG KeywordAny;
    ULONGLONG KeywordAll;
    REGHANDLE RegHandle;
    TLG_PENABLECALLBACK EnableCallback;
    PVOID CallbackContext;
};

purpose of the fields is pretty obvious except RegHandle - it's not real HANDLE but some structure with address to ETW_REGISTRATION_ENTRY.

How we can find it? Field ProviderMetadataPtr is pointer to _TlgProviderMetadata_t:
struct _TlgProviderMetadata_t
{
    UINT8 Type; // = _TlgBlobProvider3
    GUID ProviderId;
#define _TLG_PROVIDER_METADATA_PREAMBLE 16 // = sizeof(ProviderId)
    UINT16 RemainingSize; // = sizeof(RemainingSize + ProviderName)
    /*
    char ProviderName[sizeof("providerName")]; // UTF-8 nul-terminated provider name
    for each additional chunk of metadata {
        UINT16 ChunkSize;
        UINT8 ChunkType;
        UINT8 ChunkData[ChunkSize - 3];
    }
    */
};
actually it points to _TlgProviderMetadata_t.RemainingSize. Algo is simple - if you know provider GUID you can locate _TlgProviderMetadata_t.ProviderId by signature (usually located in .text or .rdata sections) and then find in .data section _TlgProvider_t whose ProviderMetadataPtr points to _TlgProviderMetadata_t.RemainingSize. I made simple PoC for arm64

How we can abuse it? Let`s see how this structures used for example in combase.dll:

what`s wrong with Etw

Disclaimer: as I am aware that the given code examples can be dangerous for Etw-based EDR products - all code was made for least popular version of windows - for arm64

Let's assume that we have some application that wants to hide its activity from trace logs - not necessary evil or malicious, for example just to hide used algos or bit paranoid like crypto-wallet. Lets see how can it achieve this (I have no desire to consider trivial cases like removing records from eventlog)

Semiofficial ways

1. Sure all you readed about COMPlus_ETWEnabled but there is also promising COMPlus_ETWFlags 
2. You can switch off etw tracing for services.exe with registry key TracingDisabled in Software\Microsoft\Windows NT\CurrentVersion\Tracing\SCM\Regular
3. And the same for rpcrt4.dll with registry key ExtErrorInformation in HKLM\Software\Policies\Microsoft\Windows NT\Rpc

Actually there are virtually countless ways to do it. And many perhaps not documented bcs was written in Ms by some poor intern who was kicked out in the cold after another review 10+ years ago. I struggled with temptation to make clickbait caption like "99% of windows dlls can disable etw logs" but it`s close to the truth

Patching

ETW private loggers

as you know ordinary etw loggers can be checked in compmgmt.msc\performance\data collector sets\event trace sessions
But private etw sessions cannot be showed in compmgmt.msc
Actually all private sessions stored in ntdll!EtwpLoggerArray. This array has size of 0x40 items (see allocation in function EtwpGetNextAvailableLoggerId) and looks like:

EtwpLoggerArray: 000000000524D380
00000000  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000010  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000020  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000030  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000040  80 6D 2B 05-01 00 00 00|01 00 00 00-00 00 00 00  Ђm+.............
00000050  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000060  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000070  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000080  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
00000090  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................
000000A0  01 00 00 00-00 00 00 00|01 00 00 00-00 00 00 00  ................

kernel etw traces in windows 10

In windows 10 there is no good old EtwpGuidHashTable and all registered Etw stored in SILO. Let`s see how we can extract them

Check first function exported function EtwRegister:
  call    _PsGetCurrentServerSiloGlobals@0 ; PsGetCurrentServerSiloGlobals()
  push    [ebp+arg_C]
  mov     edx, [ebp+arg_0]
  push    dword ptr [ebp+4]
  mov     ecx, [eax+1F0h]               ; ESERVERSILO_GLOBALS.EtwSiloState
  push    [ebp+arg_8]
  push    [ebp+arg_4]
  push    3
  call    _EtwpRegisterProvider@28      ; EtwpRegisterProvider(x,x,x,x,x,x,x)

Function PsGetCurrentServerSiloGlobals returns processes SILO or default SILO PspHostSiloGlobals stored in kernel which has type ESERVERSILO_GLOBALS. Field EtwSiloState has type ETW_SILODRIVERSTATE:
struct _ETW_SILODRIVERSTATE {
  /*   0x0 */ /*|0x4|*/ struct _ESERVERSILO_GLOBALS* SiloGlobals;
  /*   0x8 */ /*|0x168|*/ struct _ETW_GUID_ENTRY EtwpSecurityProviderGuidEntry;
  /* 0x170 */ /*|0x100|*/ struct _EX_RUNDOWN_REF_CACHE_AWARE* EtwpLoggerRundown[64];
  /* 0x270 */ /*|0x100|*/ struct _WMI_LOGGER_CONTEXT* WmipLoggerContext[64];
  /* 0x370 */ /*|0x700|*/ struct _ETW_HASH_BUCKET EtwpGuidHashTable[64];
  /* 0xa70 */ /*|0x10|*/ unsigned short EtwpSecurityLoggers[8];
  /* 0xa80 */ /*|0x1|*/ unsigned char EtwpSecurityProviderEnableMask;
  /* 0xa84 */ /*|0x4|*/ long EtwpShutdownInProgress;
  /* 0xa88 */ /*|0x4|*/ unsigned long EtwpSecurityProviderPID;
  /* 0xa8c */ /*|0x10|*/ struct _ETW_PRIV_HANDLE_DEMUX_TABLE PrivHandleDemuxTable;
  /* 0xa9c */ /*|0x10|*/ struct _ETW_COUNTERS EtwpCounters;
  /* 0xab0 */ /*|0x8|*/ union _LARGE_INTEGER LogfileBytesWritten;
  /* 0xab8 */ /*|0x4|*/ struct _ETW_SILO_TRACING_BLOCK* ProcessorBlocks;
};