While poking around RT kernels (extracted from libcuda.so) I stumbled upon very remarkable sample - sm159.elf. First let's check what tools were used to compile it - dump section .note.nv.tkinfo (here it has index 5):
readelf -x 5 sm159.elf
As you can see it was compiled with regular ptxas from SDK 13.1 (and not with nvasm_internal). This file contains huge function _ZN6RefTMA12TMACommand_tC1ERKNS_19ptx_inst_parametersE - first branch happens at offset 0xc70 and first back-edge at 37c0. Let's take a look at this piece of code:
Hex dump of section '.note.nv.tkinfo':
0x00000000 0c000000 9c000000 d0070000 4e564944 ............NVID
0x00000010 49412043 6f727000 02000000 00000000 IA Corp.........
0x00000020 01000000 07000000 35000000 54000000 ........5...T...
0x00000030 00707478 61730043 75646120 636f6d70 .ptxas.Cuda comp
0x00000040 696c6174 696f6e20 746f6f6c 732c2072 ilation tools, r
0x00000050 656c6561 73652031 332e312c 20563133 elease 13.1, V13
0x00000060 2e312e30 00427569 6c642063 7564615f .1.0.Build cuda_
0x00000070 612e6375 64615f61 2e333630 34343438 a.cuda_a.3604448
0x00000080 365f3000 2d6d6178 72726567 636f756e 6_0.-maxrregcoun
0x00000090 74203332 202d6172 63682073 6d5f3132 t 32 -arch sm_12
0x000000a0 31202d64 6c636d20 6367202d 6d203634 1 -dlcm cg -m 64
0x000000b0 20000000 ... /*01b0*/ LDC.64 R28, c[0x0][0x358] ; move base ptr into R28-29What noteworthy things can we examine here?
/*01c0*/ R2UR UR4, R28.reuse ; assign UR4
/*01d0*/ R2UR UR5, R29.reuse ;
/*01e0*/ R2UR UR6, R28 ;
/*01f0*/ R2UR UR7, R29 ;
/*0200*/ R2UR UR8, R28.reuse ; assign the same value from R28-29
/*0210*/ R2UR UR9, R29.reuse ;
/*0220*/ R2UR UR10, R28.reuse ;
/*0230*/ R2UR UR11, R29.reuse ;
/*0240*/ ST.E desc[UR4][R16.64+0x18], R3 ; use UR4 for desc operand
/*0250*/ R2UR UR12, R28.reuse ;
/*0260*/ R2UR UR13, R29 ;
/*0270*/ ST.E desc[UR6][R16.64+0x38], R3 ;
/*0280*/ R2UR UR70, R28 ;
/*0290*/ R2UR UR71, R29.reuse ;
/*02a0*/ ST.E desc[UR8][R16.64+0x58], R3 ;
/*02b0*/ R2UR UR68, R28.reuse ;
/*02c0*/ R2UR UR69, R29.reuse ;
/*02d0*/ ST.E desc[UR10][R16.64+0x78], R3 ;
/*02e0*/ R2UR UR66, R28.reuse ;
/*02f0*/ R2UR UR67, R29 ;
/*0300*/ ST.E desc[UR12][R16.64+0x118], R5 ;
First: inconsistent marking registers for reusing
Second: value in registers R28-29 does not change, yet it is repeatedly assigned to many of different uniform registers via instruction R2UR. Since R28-29 haven’t been modified, why is ptxas burning a fresh uniform register pair on almost every single store? Maybe they are used somewhere below for different purposes? Lets run dg2.pl with registers tracking: ; UR4Here addresses with '<-' denote write operation, and we can verify that for example at address 0x6c0 happens exactly the same:
; 1C0 <-
; 240
; 540
; 5D0 GENERIC_ADDRESS
; 680
; 6C0 <-
; 750
; 8E0 <-
; B50
; BF0
/*06c0*/ R2UR UR4, R4.reuse ;
/*06d0*/ R2UR UR5, R5.reuse ;
and below we can see another pattern of filling numerous uniform registers - just with new pair R4-5
Also we can notice that read/store operations happen on memory descriptors. We can check that instructions like LD/ST/ATOM etc has very limited number of forms for desc operands - for example run my ina:
./ina -v sm120.so
width 128, 259 instructions
> ld
6 forms:
LD +d
1 forms:
1) n 1110 line 68860 @Pg LD E:e E:cop E:sp2 E:sz E:sem E:sco E:private E:Rd ,desc:[E:Ra_URb],[E:Ra E:input_reg_sz_64_dist Ra_offset] ,Pnz req_bit_set src_rel_sb dst_wr_sb E:usched_info E:batch_t E:pm_pred
i Ra_URb
MaskLen 8 Enum UniformRegister
As we can see base register for desc operand must always be uniform register
It seems that ptxas has bug in its register allocator and marks those uniform registers as dirty - and so allocating and assign new pair every time.
Furthermore, this creates a cascading effect - lets estimate how many uniform registers function used and how many holes exist in their allocation:
; URHoles max 71: UR0 UR1 UR2 UR3 UR36 UR37 UR38 UR39 UR40 UR41 UR42 UR43 UR44 UR45 UR46 UR47 UR48 UR49 UR50 UR51 UR52 UR53 UR54 UR55 UR56 UR57 UR58 UR59 UR60 UR61 UR62 UR63
; 95 regs, avg 11.875000 per block
; 76 uregs, avg 9.500000 per block
; 2 preds, avg 0.250000 per block
; 7 BD, avg 0.875000 per block
; 2 single block registers (0.021053 from total)
; 14 single block uregisters (0.184211 from total)
Even if we skip UR0-4 - anyway 28 uniform registers are never used, and this is 28 / 72 =39%
In my opinion, this is
Комментариев нет:
Отправить комментарий