[9front] Porting RTL8111 (ether8169) driver to bcm64

[9front] Porting RTL8111 (ether8169) driver to bcm64

[Mack Wallace]

Hi all,

I have a Raspberry Pi Compute Module 4 (CM4) with a DFRobot mini router board (https://www.dfrobot.com/product-2242.html) for which I would like to use the second NIC.

The second NIC is a RTL8111 connected to the Pi through its PCIe interface. 

I’ve looked at the code in front that should support the RTL8111, (ether8169.c,) in the pc directory. For better or worse this driver makes use of inb, outb, ins, outs, inl and outl which are architecture specific to x86. I wasn’t able to find equivalent calls in 9fromt. Trying to find the equivalent calls for arm (bcm64) architecture, I found equivalents in the linux kernel which (after a few wrappers) wrap inline assembly. Looking at the driver from Realtek, they use different calls depending on pre-compiler directives - including the readb/l/w and writeb/l/w calls that wrap the assembly in linux. So I would think this could be done to the ether8169.c driver. 

My questions:

Are there already similar calls in 9front and I’ve (stupidly) overlooked them? 

Should it be as simple as adding the platform dependent calls in the driver to get it to work on the Pi, or is there more work that needs to be done to the bcm64 PCIe interface?


Regards,

mackbw

Re: [9front] Porting RTL8111 (ether8169) driver to bcm64

[cinap_lenrek]

> Are there already similar calls in 9front and I’ve (stupidly) overlooked them? 

no. i/o space is not supported on arm. this has nothing todo
with 9front.

i'm pretty sure the rtl chip will provide a membar with memory
mapped registers in addition to the i/o port interface,
you do not need special calls for this.

just vmap() the and access the registers with loads and stores.

get the datasheet and programmers manual for the rtl chip.

> Should it be as simple as adding the platform dependent calls in the driver
> to get it to work on the Pi, or is there more work that needs to be done to
> the bcm64 PCIe interface?

for dma, you need to manually keep the caches coherent with
the device using dmaflush() calls (see in port/usbxhci.c as an example).

this is the biggest reason most drivers cannot direcly be used
on arm, tho shouldnt be too difficult to make it work.

--
cinap

Re: [9front] Porting RTL8111 (ether8169) driver to bcm64

[cinap_lenrek]

https://recomb-omsk.ru/published/SC/html/scripts/doc/RTL8111B_8168B_Registers_DataSheet_1.0.pdf

or just search for "rtl8111 registers datasheet" on the internet.

page 28 descibes that it has a 64-bit MEMBAR (bar2/bar3) for the
registers as well as a IOBAR (bar0).

--
cinap

Re: [9front] Porting RTL8111 (ether8169) driver to bcm64

[Mack Wallace]

So, I am trying to understand porting of the driver. I’ve made some progress in understanding things - but not enough to get anything functional. I know there is a bit of DMA work that needs to be done which I have not even begun to broach, right now I am just focusing on getting the adapter to be recognized. I did make changes to the ether8169 driver from the /pc lineage and got that compiled. After some discussion in a chat room (thanks Grobe0ba) I was directed to the Tegra2 version of the driver. That didn’t solve anything, but between that and looking at the dmesg in linux led me to believe I needed to be looking at BAR2 rather than stupidly looking at BAR0 (sorry cinap should have read your last message in better detail). — However, the only thing I’ve managed to fool into thinking I’m a competent programmer is the compiler - which isn’t too hard. So I am looking for a bit of inspiration here, and some basic knowledge. 

In the pc driver, I first started changing all instances of ‘port’ to ‘mmio’ as I would start to set things up as in the usbxhci.c and vmap an area for io. Of course I then ran across edev->port in rtl8169pnp and realized that this needs to be set. Of course between type mismatches and not being to figure out how port is defined in the Ether struct, that has served as an obstacle.

After doing some fudging I was able to get it to compile so I could boot the kernel and use some print statements in the code to get a better understanding of the order calls were being made and whether the RTL8111 was being recognized, which it is. 


bus dev type     vid  did  intl memory
0   0/0 06 04 00 14e4 2711   0  ioa:00000000-00001000 4096 mema:600000000-600100000 1048576 ->1
1   0/0 02 00 00 10ec 8168 255  0:00000001 256 2:600000004 4096 4:600004004 16384 
…
Checking ether8169
We passed the first test
PCIe is: 1
Mem0 is 256 bytes, and BAR: 1
rtl8169: 0x10ec 0x8168: port 0 size 256 irq 255
…
#l1: rtl8169: reset failed

Obviously, the first two lines would be from any 9front kernel, recognizing PCI devices.
The rest of the lines are from the driver itself - including some of my debugging prints. 

Being confused (not unusual) and wanting more information, the Linux dmesg…

[    0.099846] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    0.099869] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    0.099905] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    0.099943] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
[    0.134142] brcm-pcie fd500000.pcie: link up, 2.5 GT/s x1 (SSC)
[    0.134309] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    0.134327] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.134344] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    0.134389] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    0.134479] pci 0000:00:00.0: PME# supported from D0 D3hot
[    0.137849] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.137972] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000
[    0.138022] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.138059] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    0.138088] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    0.138217] pci 0000:01:00.0: supports D1 D2
[    0.138230] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.141500] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    0.141537] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[    0.141557] pci 0000:01:00.0: BAR 4: assigned [mem 0x600000000-0x600003fff 64bit]
[    0.141587] pci 0000:01:00.0: BAR 2: assigned [mem 0x600004000-0x600004fff 64bit]
[    0.141614] pci 0000:01:00.0: BAR 0: no space for [io  size 0x0100]
[    0.141629] pci 0000:01:00.0: BAR 0: failed to assign [io  size 0x0100]
[    0.141643] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.141659] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
[    0.141806] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    0.141928] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
[    0.142148] pcieport 0000:00:00.0: AER: enabled with IRQ 38
...
[    6.777869] r8169 0000:01:00.0: enabling device (0000 -> 0002)
[    6.792249] r8169 0000:01:00.0: can't read MAC address, setting random one
[    6.803140] libphy: r8169: probed
[    6.806902] r8169 0000:01:00.0 eth1: RTL8168h/8111h, d6:3a:e5:18:0b:68, XID 541, IRQ 39
[    6.814951] r8169 0000:01:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko]

I have been confused by whether I need to assign an address to the rtl8169 and if so, what address to use. Confused by the two PCI devices recognized as being in the same address space (0x600000000) though on different busses. Do I need to include devpnp.c?

After looking at ethergenet.c, and what the data sheet says on page 37 of the registers data sheet, I believe (correct me if I am wrong) that the IOBAR and MEMBAR need to have their addresses set. And looking at the BCM2711 ARM Peripherals datasheet (https://datasheets.raspberrypi.com/bcm2711/bcm2711-peripherals.pdf), it seems that the peripherals are usually mapped to either between 0x7C00 0000-0x8000 0000. Those addresses may also be represented as 0x4 7C00 0000 - 0x4 8000 0000 or 0x0FC00 0000 - 0x0 FF80 0000 depending on which view of the address map one is using? The VIRTIO seems to maps into that 0x7c00 0000 region through comments in mem.h:

mem.h:#define VIRTIO2		(0xFFFFFFFFBC000000ULL)	/* 0x7C000000	-		0xFC000000 */

and there are the other VIRTIO definitions…

mem.h:#define VIRTIO1		(0xFFFFFFFFBD000000ULL)	/* 0x7D000000	-		0xFD000000 */
mem.h:#define VIRTIO		(0xFFFFFFFFBE000000ULL) /* 0x7E000000	0x3F000000	0xFE000000 */

I see that ethergenet.c seems to assign an address through ctlr->regs…

static int
pnp(Ether *edev)
{
	static Ctlr ctlr[1];

	if(ctlr->regs != nil)
		return -1;

	ctlr->regs = (u32int*)(VIRTIO1 + 0x580000);
	ctlr->rx->regs = &ctlr->regs[RdmaOffset + nelem(ctlr->rd)*3 + 16*RingCfg];
	ctlr->tx->regs = &ctlr->regs[TdmaOffset + nelem(ctlr->td)*3 + 16*RingCfg];

	edev->port = (uintptr)ctlr->regs;

But I am confused by the pc version of the rtl8169 device - first calling rtl8169pnp which fairly soon after calls on rtl8169pci. I originally thought that I would set up my vmap under the pci routine. Looking at the PCIDev there, i am getting the correct information about the vendor and device id, but then there are some calls to vet (get) the mac address, but before a base address would be established under pnp? I suppose an address could be established in either pnp or pci, but I presume it makes more sense to do it in the pnp routine. I guess my question is how do I establish the address? I see in ethergenet addresses written to the ctlr->regs, but that driver doesn’t seem to have any mention of pci, so I am guessing that is not it. I’m guessing this has to be done through the Pcidev structure and some call to write to that address? pcicfgw maybe?  - Then what address do I use? VIRTIO1 + 0x680000 (I have no idea what the memory window is for ethergenet.c is, I just added the window of the other pci device, which of course is not located at VIRTIO1+0x580000, so 0x680000 is a bullshit number - so I need to figure out a free window above ethergenet? Is there a general rule of thumb regarding how and where such spaces should be allocated (e.g. every PCI device gets 16k or something like that?


Below is some of my initial (ignorant) coding in the rtl8169pci routine. I need to better understand the purpose for Ctlr->port, vs Etherdev->port, vs Ctlr->regs. I/O? DMA? Memory? 

At the bottom are the results of lspci. 

Hope that someone can help with some pieces of the puzzle.

Thanks,

mackbw


	print("Mem2 is %d bytes, and BAR: %llux\n", p->mem[2].size, p->mem[2].bar);
		if(p->mem[0].size == 0 || (p->mem[0].bar & 1) == 0)
			continue;

		if(p->mem[2].bar & 1) //checks to see if this is "memory" or "io"
			continue;
		io = p->mem[2].bar & ~0x0f;
		if(io == 0)
			continue;
		print("rtl8169: %#x %#x: port %llux size %d irq %d\n",
			p->vid, p->did, io, p->mem[2].size, p->intl);
		mmio = vmap(io, p->mem[2].size);
		if(mmio == nil){
			print("rtl8169: cannot map registers\n");
			continue;
		}
		ctlr = malloc(sizeof(Ctlr));
		if(ctlr == nil){
			print("rtl8169: can't allocate memory\n");
			vunmap(mmio, p->mem[2].size);
			continue;
		}
		ctlr->port = io;
		ctlr->mmio = mmio;

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
00: ec 10 68 81 06 04 10 00 15 00 00 02 10 00 00 00
10: 01 00 00 00 00 00 00 00 04 40 00 f8 00 00 00 00
20: 04 00 00 f8 00 00 00 00 00 00 00 00 ec 10 68 81
30: 00 00 00 00 40 00 00 00 00 00 00 00 26 01 00 00

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
        Flags: bus master, fast devsel, latency 0, IRQ 39
        I/O ports at <unassigned> [disabled]
        Memory at 600004000 (64-bit, non-prefetchable) [size=4K]
        Memory at 600000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked+
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [170] Latency Tolerance Reporting
        Capabilities: [178] L1 PM Substates
        Kernel driver in use: r8169
lspci: Unable to load libkmod resources: error -12

> On Nov 14, 2021, at 12:57 AM, cinap_lenrek@felloff.net wrote:
> 
> https://recomb-omsk.ru/published/SC/html/scripts/doc/RTL8111B_8168B_Registers_DataSheet_1.0.pdf
> 
> or just search for "rtl8111 registers datasheet" on the internet.
> 
> page 28 descibes that it has a 64-bit MEMBAR (bar2/bar3) for the
> registers as well as a IOBAR (bar0).
> 
> --
> cinap
> 

Re: [9front] Porting RTL8111 (ether8169) driver to bcm64

[cinap_lenrek]

no, your completely confused. this is not how pci works at all.
stop mucking with  VIRTIO addresses, these are for SoC internal
peripheral buses.

you'r absolutely off the rails with this...

look at ANY pci device driver, theres no mucking with anything.

you have to understand how pci works. all you said i already
TOLD you from information from the datasheet, you just do not
seem to understand what it means.

so in the kernel, a pci device is represented by a Pcidev struct
(port/pci.h).

there are Pcidev.mem[6], which is a array of 6 bar/size pairs.

the bar field is just the register contents of the devices
BAR registers when the device got enumerated. the platform
pci code is responsible for (re-) programming / checking
the bars before any drivers touches any pci device.

it is not your drivers business to muck with them.

on a pc, that job is even done by the bios/uefi. and the os
doesnt really need todo anything. (except work around
misconfigurations).

on bcm64, the pci code allocates specifically reserved
physical address space for the bar windows and the programs
all found bars.

from your drivers point of view, you only deal with a already
programmed Pcidev struct of the device.

next important thing, thats absolutely basic pci knowledge:

bars are TYPED. the low 7 bits of the bars tell you what
TYPE it is. if it is i/o space or memory. (IOBAR, MEMBAR).

if a bar has bit 0 set, it means it is I/O bar.

so that is why EVERY driver checks bit 0 first, to see
if it is of type memory (bit0 == clear). and then
the whole 4 low bits get masked off to get the PHYSICAL
address value of the bar. the reason is that there is
also a "prefetchable" and "64-bit" bit in there that
is not part of the physical address.

so all your driver really needs todo is:

// find my device.
Pcidev *pcidev = pcimatch(....);

// bring device out of suspend
pcienable(pcidev);

// make sure it has the membar
if(pcidev->mem[2].bar&1){
	print("this is not a membar\n");
	return -1;
}

// make sure it has the expected size
if(pcidev->mem[2].size <= HOWMANBYTESAREMYREGISTERBLOCK){
	print("the membar is too small, does not cover all the registers\n");
	return -1;
}

// map it into kernel virtual memory
uint32 *regs = vmap(pcidev->mem[2].bar&~0xF, pcidev->mem[2].size);

if(regs == nil){
	print("can't map it\");
	return -1;
}

// then you can poke at the register
regs[CONTROL123] = 1;

thats it.

--
cinap

Re: [9front] Porting RTL8111 (ether8169) driver to bcm64

[Mack Wallace]

What was outlined was generally what I was doing. A notable difference was when pcienable occurred - I don’t know whether it is that important to execute that before or after the vmap.

Previously, the controller did not appear to be initializing, and so my question was whether or not the PCI setup was occurring at all. Whether 9front was set up to properly configure extra PCIe devices, or if more work was needed.  

I’ve moved around my code - tried a few different things, just trying to see if I can get the device to reset or read whether I have a link - and I am failing at those basic tasks… and there are a few symptoms that concern me, but I am not sure whether I should be concerned.

Symptom 1: the irq is 255. Maybe it is a non-issue - maybe expected the way interrupts are allocated, I don’t know.

Symptom 2: I get an odd MAC address: c6:3d:40:80:0:0. I say this is odd for a few reasons. The last two octets being zero are unusual, but also knowing that this board does not have a rom to load a MAC address from and that the linux driver, upon failing to read a MAC address from the unit, assigns its own. I would presume that without a ROM, that such an unassigned address would be all zeros… but that’s a complete guess. 

The data sheet seems to contradict itself regarding the MAC address field: "ID registers 0-5 are only permitted to read/write via 4-byte access. Read access can be byte, word, or double word access. …” I’ve attempted reading by bytes to check if the latter statement is true, but I get something completely different than when I read via 4-byte access. So maybe only 4-byte access is viable. However, if I read the first four bytes of the MAC address (by either mmio[0], or by using the drivers csr32r(ctlr, 0)), and then the last four bytes (by either mmio[2], or by using the drivers csr32r(ctlr, 2)), the latter value does not seem to contain the two middle bytes shifted - i.e. for the first read I get 0x8678a1c6, and the second read with a two byte offset I get 0x00020040 when I expect 0x0000c6a1. 

Symptom 3: The lower nibble of Config1 is read only and should always be 0xF. - But when I read it, it is 0.

There are some other items - but it seems that I am missing something in the vmap.

I’ll include my current code - debugging acrobatics and print statements and all. I know, ‘io’ is a misnomer as I am trying to look at the PCI memory. Although, looking at the data sheet. the MAC registers are available in both I/O and PCI Memory space. I’d try the I/O space, but mem[0].bar = 0x01 - which makes the address 0x0 - though this seems consistent with the linux driver, and then linux just says that it can not allocate the memory for the i/o space.

— thanks, 
mackbw


Current Code: 

static void
rtl8169pci(void)
{
	Pcidev *p;
	Ctlr *ctlr;
	int i, pcie;
	uvlong io;
	u32int *mmio, a, b;
	uint macv, r;

	p = nil;
	print("Checking ether8169\n");
	while(p = pcimatch(p, 0, 0)){
		if(p->ccrb != 0x02 || p->ccru != 0)
			continue;
		pcie = 0;
		switch(i = ((p->did<<16)|p->vid)){
		default:
			continue;
		case Rtl8100e:			/* RTL810[01]E ? */
		case Rtl8168b:			/* RTL8168B */
			pcie = 1;
			break;
		case Rtl8169c:			/* RTL8169C */
		case Rtl8169sc:			/* RTL8169SC */
		case Rtl8169:			/* RTL8169 */
			break;
		case (0xC107<<16)|0x1259:	/* Corega CG-LAPCIGT */
			i = Rtl8169;
			break;
		}

		print("Mem0 is %d bytes, and BAR: %llux\n", p->mem[0].size, p->mem[0].bar);
		print("Mem2 is %d bytes, and BAR: %llux\n", p->mem[2].size, p->mem[2].bar);

		if(p->mem[2].bar & 1) //checks to see if this is "memory" or "io"
			continue;

		pcienable(p);

		io = p->mem[2].bar & ~0xF;
		mmio = vmap(io, p->mem[2].size);

		if(mmio == nil){
			print("rtl8169: cannot map registers\n");
			continue;
		}

		ctlr = malloc(sizeof(Ctlr));
		if(ctlr == nil){
			print("rtl8169: can't allocate memory\n");
			vunmap(mmio, p->mem[2].size);
			continue;
		}
		ctlr->port = (uint)mmio;
		ctlr->mmio = mmio;
		ctlr->pcidev = p;
		ctlr->pciv = i;
		ctlr->pcie = pcie;
		print("rtl8169: %#x %#x: port %ux size %d irq %d\n",
			p->vid, p->did, *mmio, p->mem[2].size, p->intl);

		if(vetmacv(ctlr, &macv) == -1){
			print("rtl8169: %T: unknown mac %.4ux %.8ux\n", p->tbdf, p->did, macv);
			pcidisable(p);
			vunmap(mmio, p->mem[2].size);
			free(ctlr);
			continue;
		}


/*		if(rtl8169reset(ctlr)){
			print("Failed to reset!\n");
			pcidisable(p);
			vunmap(mmio, p->mem[2].size);
			free(ctlr);
			continue;
		} */

		print("bytes of MAC address %ux:%ux:%ux:%ux:%ux:%ux\n", mmio[0], mmio[1], mmio[2], mmio[3], mmio[4], mmio[5]);
		a = mmio[0];
		b = mmio[2];
		print("bytes of MAC address %ux:%ux\n", a, b);

		print("Configuration byte: %ux\n", mmio[Config1]);

	r = csr8r(ctlr, Phystatus);

	/*
	 * Maybe the link changed - do we care very much?
	 * Could stall transmits if no link, maybe?
	 */
	print ("Phystatus: %ux\n", r);
	if (r & Linksts)
		print("pre: We have link!\n");
	else
		print("pre: We don't have a link!\n");

		rtl8169halt(ctlr);

		/*
		 * Extract the chip hardware version,
		 * needed to configure each properly.
		 */

		ctlr->macv = macv;
		if(rtl8169ctlrhead != nil)
			rtl8169ctlrtail->next = ctlr;
		else
			rtl8169ctlrhead = ctlr;
		rtl8169ctlrtail = ctlr;
	}
}



Suggested Code from cinap:
> 
> // find my device.
> Pcidev *pcidev = pcimatch(....);
> 
> // bring device out of suspend
> pcienable(pcidev);
> 
> // make sure it has the membar
> if(pcidev->mem[2].bar&1){
> 	print("this is not a membar\n");
> 	return -1;
> }
> 
> // make sure it has the expected size
> if(pcidev->mem[2].size <= HOWMANBYTESAREMYREGISTERBLOCK){
> 	print("the membar is too small, does not cover all the registers\n");
> 	return -1;
> }
> 
> // map it into kernel virtual memory
> uint32 *regs = vmap(pcidev->mem[2].bar&~0xF, pcidev->mem[2].size);
> 
> if(regs == nil){
> 	print("can't map it\");
> 	return -1;
> }
> 
> // then you can poke at the register
> regs[CONTROL123] = 1;
> 
> thats it.
> 
> —
> cinap

Code from my prior message:
> 	print("Mem2 is %d bytes, and BAR: %llux\n", p->mem[2].size, p->mem[2].bar);
> 		if(p->mem[0].size == 0 || (p->mem[0].bar & 1) == 0)
> 			continue;
> 
> 		if(p->mem[2].bar & 1) //checks to see if this is "memory" or "io"
> 			continue;
> 		io = p->mem[2].bar & ~0x0f;
> 		if(io == 0)
> 			continue;
> 		print("rtl8169: %#x %#x: port %llux size %d irq %d\n",
> 			p->vid, p->did, io, p->mem[2].size, p->intl);
> 		mmio = vmap(io, p->mem[2].size);
> 		if(mmio == nil){
> 			print("rtl8169: cannot map registers\n");
> 			continue;
> 		}
> 		ctlr = malloc(sizeof(Ctlr));
> 		if(ctlr == nil){
> 			print("rtl8169: can't allocate memory\n");
> 			vunmap(mmio, p->mem[2].size);
> 			continue;
> 		}
> 		ctlr->port = io;
> 		ctlr->mmio = mmio;