vga/igfx: fix integer overflow in datam calculation

vga/igfx: fix integer overflow in datam calculation

[qwx]

vga/igfx: fix integer overflow in datam calculation

data[mn] and link[mn] are 24-bit values.
in the expression 'm = (n * ((freq * bpp)/8)) / (lsclk * lanes)',
uvlongs are used to prevent integer overflow, but since freq, bpp, lsclk and
lanes are all ints, the cast to uvlong does not happen until it's too late,
getting a wrong value.
instead, use u32int for m and n, and use casts where necessary.

example of bad calculation:
freq = 141400000
lsclk = 270000000
lanes = 2
bpp = 18
→ 0x7f3ee1ca6 (correct value: 0x4b69d0)

diff -r 0a3cf47fce65 sys/src/cmd/aux/vga/igfx.c
--- a/sys/src/cmd/aux/vga/igfx.c	Mon Jun 27 00:36:54 2016 +0200
+++ b/sys/src/cmd/aux/vga/igfx.c	Thu Jun 30 09:22:25 2016 +0300
@@ -692,16 +692,16 @@
 static void
 initdatalinkmn(Trans *t, int freq, int lsclk, int lanes, int tu, int bpp)
 {
-	uvlong m, n;
+	u32int m, n;
 
 	n = 0x800000;
-	m = (n * ((freq * bpp)/8)) / (lsclk * lanes);
+	m = (n * (((uvlong)freq * bpp)/8)) / ((uvlong)lsclk * lanes);
 
 	t->dm[0].v = (tu-1)<<25 | m;
 	t->dn[0].v = n;
 
 	n = 0x80000;
-	m = (n * freq) / lsclk;
+	m = ((uvlong)n * freq) / lsclk;
 
 	t->lm[0].v = m;
 	t->ln[0].v = n;

Re: [9front] vga/igfx: fix integer overflow in datam calculation

[cinap_lenrek]

what was i thinking :-)

many thanks!

--
cinap