Fwd: [9front] Porting RTL8111 (ether8169) driver to bcm64

Fwd: [9front] Porting RTL8111 (ether8169) driver to bcm64

[Mack Wallace]

realized I wrote the wrong expected address -  should be 0x00008678, not 0x0000c681 - but what I end up reading is still not what I expect. 

mackbe

> Begin forwarded message:
> 
> From: Mack Wallace <mackbw@mapinternet.com>
> Subject: Re: [9front] Porting RTL8111 (ether8169) driver to bcm64
> Date: December 7, 2021 at 3:41:43 PM EST
> To: 9front@9front.org
> 
> What was outlined was generally what I was doing. A notable difference was when pcienable occurred - I don’t know whether it is that important to execute that before or after the vmap.
> 
> Previously, the controller did not appear to be initializing, and so my question was whether or not the PCI setup was occurring at all. Whether 9front was set up to properly configure extra PCIe devices, or if more work was needed.  
> 
> I’ve moved around my code - tried a few different things, just trying to see if I can get the device to reset or read whether I have a link - and I am failing at those basic tasks… and there are a few symptoms that concern me, but I am not sure whether I should be concerned.
> 
> Symptom 1: the irq is 255. Maybe it is a non-issue - maybe expected the way interrupts are allocated, I don’t know.
> 
> Symptom 2: I get an odd MAC address: c6:3d:40:80:0:0. I say this is odd for a few reasons. The last two octets being zero are unusual, but also knowing that this board does not have a rom to load a MAC address from and that the linux driver, upon failing to read a MAC address from the unit, assigns its own. I would presume that without a ROM, that such an unassigned address would be all zeros… but that’s a complete guess. 
> 
> The data sheet seems to contradict itself regarding the MAC address field: "ID registers 0-5 are only permitted to read/write via 4-byte access. Read access can be byte, word, or double word access. …” I’ve attempted reading by bytes to check if the latter statement is true, but I get something completely different than when I read via 4-byte access. So maybe only 4-byte access is viable. However, if I read the first four bytes of the MAC address (by either mmio[0], or by using the drivers csr32r(ctlr, 0)), and then the last four bytes (by either mmio[2], or by using the drivers csr32r(ctlr, 2)), the latter value does not seem to contain the two middle bytes shifted - i.e. for the first read I get 0x8678a1c6, and the second read with a two byte offset I get 0x00020040 when I expect 0x0000c6a1. 
> 
> Symptom 3: The lower nibble of Config1 is read only and should always be 0xF. - But when I read it, it is 0.
> 
> There are some other items - but it seems that I am missing something in the vmap.
> 
> I’ll include my current code - debugging acrobatics and print statements and all. I know, ‘io’ is a misnomer as I am trying to look at the PCI memory. Although, looking at the data sheet. the MAC registers are available in both I/O and PCI Memory space. I’d try the I/O space, but mem[0].bar = 0x01 - which makes the address 0x0 - though this seems consistent with the linux driver, and then linux just says that it can not allocate the memory for the i/o space.
> 
> — thanks, 
> mackbw
> 
> 
> Current Code: 
> 
> static void
> rtl8169pci(void)
> {
> 	Pcidev *p;
> 	Ctlr *ctlr;
> 	int i, pcie;
> 	uvlong io;
> 	u32int *mmio, a, b;
> 	uint macv, r;
> 
> 	p = nil;
> 	print("Checking ether8169\n");
> 	while(p = pcimatch(p, 0, 0)){
> 		if(p->ccrb != 0x02 || p->ccru != 0)
> 			continue;
> 		pcie = 0;
> 		switch(i = ((p->did<<16)|p->vid)){
> 		default:
> 			continue;
> 		case Rtl8100e:			/* RTL810[01]E ? */
> 		case Rtl8168b:			/* RTL8168B */
> 			pcie = 1;
> 			break;
> 		case Rtl8169c:			/* RTL8169C */
> 		case Rtl8169sc:			/* RTL8169SC */
> 		case Rtl8169:			/* RTL8169 */
> 			break;
> 		case (0xC107<<16)|0x1259:	/* Corega CG-LAPCIGT */
> 			i = Rtl8169;
> 			break;
> 		}
> 
> 		print("Mem0 is %d bytes, and BAR: %llux\n", p->mem[0].size, p->mem[0].bar);
> 		print("Mem2 is %d bytes, and BAR: %llux\n", p->mem[2].size, p->mem[2].bar);
> 
> 		if(p->mem[2].bar & 1) //checks to see if this is "memory" or "io"
> 			continue;
> 
> 		pcienable(p);
> 
> 		io = p->mem[2].bar & ~0xF;
> 		mmio = vmap(io, p->mem[2].size);
> 
> 		if(mmio == nil){
> 			print("rtl8169: cannot map registers\n");
> 			continue;
> 		}
> 
> 		ctlr = malloc(sizeof(Ctlr));
> 		if(ctlr == nil){
> 			print("rtl8169: can't allocate memory\n");
> 			vunmap(mmio, p->mem[2].size);
> 			continue;
> 		}
> 		ctlr->port = (uint)mmio;
> 		ctlr->mmio = mmio;
> 		ctlr->pcidev = p;
> 		ctlr->pciv = i;
> 		ctlr->pcie = pcie;
> 		print("rtl8169: %#x %#x: port %ux size %d irq %d\n",
> 			p->vid, p->did, *mmio, p->mem[2].size, p->intl);
> 
> 		if(vetmacv(ctlr, &macv) == -1){
> 			print("rtl8169: %T: unknown mac %.4ux %.8ux\n", p->tbdf, p->did, macv);
> 			pcidisable(p);
> 			vunmap(mmio, p->mem[2].size);
> 			free(ctlr);
> 			continue;
> 		}
> 
> 
> /*		if(rtl8169reset(ctlr)){
> 			print("Failed to reset!\n");
> 			pcidisable(p);
> 			vunmap(mmio, p->mem[2].size);
> 			free(ctlr);
> 			continue;
> 		} */
> 
> 		print("bytes of MAC address %ux:%ux:%ux:%ux:%ux:%ux\n", mmio[0], mmio[1], mmio[2], mmio[3], mmio[4], mmio[5]);
> 		a = mmio[0];
> 		b = mmio[2];
> 		print("bytes of MAC address %ux:%ux\n", a, b);
> 
> 		print("Configuration byte: %ux\n", mmio[Config1]);
> 
> 	r = csr8r(ctlr, Phystatus);
> 
> 	/*
> 	 * Maybe the link changed - do we care very much?
> 	 * Could stall transmits if no link, maybe?
> 	 */
> 	print ("Phystatus: %ux\n", r);
> 	if (r & Linksts)
> 		print("pre: We have link!\n");
> 	else
> 		print("pre: We don't have a link!\n");
> 
> 		rtl8169halt(ctlr);
> 
> 		/*
> 		 * Extract the chip hardware version,
> 		 * needed to configure each properly.
> 		 */
> 
> 		ctlr->macv = macv;
> 		if(rtl8169ctlrhead != nil)
> 			rtl8169ctlrtail->next = ctlr;
> 		else
> 			rtl8169ctlrhead = ctlr;
> 		rtl8169ctlrtail = ctlr;
> 	}
> }
> 
> 
> 
> Suggested Code from cinap:
>> 
>> // find my device.
>> Pcidev *pcidev = pcimatch(....);
>> 
>> // bring device out of suspend
>> pcienable(pcidev);
>> 
>> // make sure it has the membar
>> if(pcidev->mem[2].bar&1){
>> 	print("this is not a membar\n");
>> 	return -1;
>> }
>> 
>> // make sure it has the expected size
>> if(pcidev->mem[2].size <= HOWMANBYTESAREMYREGISTERBLOCK){
>> 	print("the membar is too small, does not cover all the registers\n");
>> 	return -1;
>> }
>> 
>> // map it into kernel virtual memory
>> uint32 *regs = vmap(pcidev->mem[2].bar&~0xF, pcidev->mem[2].size);
>> 
>> if(regs == nil){
>> 	print("can't map it\");
>> 	return -1;
>> }
>> 
>> // then you can poke at the register
>> regs[CONTROL123] = 1;
>> 
>> thats it.
>> 
>> —
>> cinap
> 
> Code from my prior message:
>> 	print("Mem2 is %d bytes, and BAR: %llux\n", p->mem[2].size, p->mem[2].bar);
>> 		if(p->mem[0].size == 0 || (p->mem[0].bar & 1) == 0)
>> 			continue;
>> 
>> 		if(p->mem[2].bar & 1) //checks to see if this is "memory" or "io"
>> 			continue;
>> 		io = p->mem[2].bar & ~0x0f;
>> 		if(io == 0)
>> 			continue;
>> 		print("rtl8169: %#x %#x: port %llux size %d irq %d\n",
>> 			p->vid, p->did, io, p->mem[2].size, p->intl);
>> 		mmio = vmap(io, p->mem[2].size);
>> 		if(mmio == nil){
>> 			print("rtl8169: cannot map registers\n");
>> 			continue;
>> 		}
>> 		ctlr = malloc(sizeof(Ctlr));
>> 		if(ctlr == nil){
>> 			print("rtl8169: can't allocate memory\n");
>> 			vunmap(mmio, p->mem[2].size);
>> 			continue;
>> 		}
>> 		ctlr->port = io;
>> 		ctlr->mmio = mmio;
> 
> 
> 

Re: Fwd: [9front] Porting RTL8111 (ether8169) driver to bcm64

[cinap_lenrek]

> Symptom 1: the irq is 255. Maybe it is a non-issue - maybe expected the way interrupts are allocated, I don’t know.

that doesnt matter. this is just a scratch register in pci config space that has
no impact on anything.

on a pc, it was used by the BIOS to communicate what effective interrupt line
on the PIC a device would signal on. so the OS would not need to know the
irq routing details on the mainboard.

on the raspberry pi, none of this matters. as this is pcie, we expect the
device to support message signaled interrupts and the pcie controller has
dedicated msi control block for this.

just call intrenable() with the devices tbdf, the irq argument is ignored in
case of a pci type tbdf.

> What was outlined was generally what I was doing. A notable difference was when pcienable occurred - I don’t know whether it is that important to execute that before or after the vmap.

yes, that makes sense. pcienable() power saving wakeup, as well as enabling
address decode bits in the pci command register appropriate to the bars.

> There are some other items - but it seems that I am missing something in the vmap.

no. the vmap() is invisible to the device. it is a pure software thing,
just to get access to the physical address space from the cpu's perspective.
vmap() always creates the mapping with the appropriate non-cached an
device attributes asuitable for device registers.

the more difficult part is sequencing access to/from dram from the device (dma),
as the dram is not held coherent in regards to your register accesses on arm.

so opposed to the pc, dmaflush() cache maintenance operations are required
to make sure memory becomes visible to the device. but that is, just like
interrupts something that should come later, once you can read the mac
address properly.

in regards to the differnet results depending on the access size, yes.
makes sense. it is something that is observable by the device... the
pcie transactions have byte enable bits in transactions. make sure your
not doing unaligned accesses in regards to the access size.
i cannot help you there tho... check datasheet and in doubt you can
take a look at openbsd or linux drivers... maybe the register map is
wrong or out of date for the device in question.

what i can see tho is that your mmio pointer is of type u32int*, so you'r
not doing byte reads here at all, and mmio[1] is byte offset 0x4.

and again, i already told you theres no I/O space on the pi. the I/O bar
is useless for you, and you cant vmap() it... I/O space and Memory space
are completely different things. and I/O space does not exist on the pi.

--
cinap