Re: [9front] Porting RTL8111 (ether8169) driver to bcm64

[Mack Wallace <mackbw@mapinternet.com>]

So, I am trying to understand porting of the driver. I’ve made some progress in understanding things - but not enough to get anything functional. I know there is a bit of DMA work that needs to be done which I have not even begun to broach, right now I am just focusing on getting the adapter to be recognized. I did make changes to the ether8169 driver from the /pc lineage and got that compiled. After some discussion in a chat room (thanks Grobe0ba) I was directed to the Tegra2 version of the driver. That didn’t solve anything, but between that and looking at the dmesg in linux led me to believe I needed to be looking at BAR2 rather than stupidly looking at BAR0 (sorry cinap should have read your last message in better detail). — However, the only thing I’ve managed to fool into thinking I’m a competent programmer is the compiler - which isn’t too hard. So I am looking for a bit of inspiration here, and some basic knowledge. 

In the pc driver, I first started changing all instances of ‘port’ to ‘mmio’ as I would start to set things up as in the usbxhci.c and vmap an area for io. Of course I then ran across edev->port in rtl8169pnp and realized that this needs to be set. Of course between type mismatches and not being to figure out how port is defined in the Ether struct, that has served as an obstacle.

After doing some fudging I was able to get it to compile so I could boot the kernel and use some print statements in the code to get a better understanding of the order calls were being made and whether the RTL8111 was being recognized, which it is. 


bus dev type     vid  did  intl memory
0   0/0 06 04 00 14e4 2711   0  ioa:00000000-00001000 4096 mema:600000000-600100000 1048576 ->1
1   0/0 02 00 00 10ec 8168 255  0:00000001 256 2:600000004 4096 4:600004004 16384 
…
Checking ether8169
We passed the first test
PCIe is: 1
Mem0 is 256 bytes, and BAR: 1
rtl8169: 0x10ec 0x8168: port 0 size 256 irq 255
…
#l1: rtl8169: reset failed

Obviously, the first two lines would be from any 9front kernel, recognizing PCI devices.
The rest of the lines are from the driver itself - including some of my debugging prints. 

Being confused (not unusual) and wanting more information, the Linux dmesg…

[    0.099846] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    0.099869] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    0.099905] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    0.099943] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
[    0.134142] brcm-pcie fd500000.pcie: link up, 2.5 GT/s x1 (SSC)
[    0.134309] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    0.134327] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.134344] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    0.134389] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    0.134479] pci 0000:00:00.0: PME# supported from D0 D3hot
[    0.137849] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.137972] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000
[    0.138022] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.138059] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    0.138088] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    0.138217] pci 0000:01:00.0: supports D1 D2
[    0.138230] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.141500] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    0.141537] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[    0.141557] pci 0000:01:00.0: BAR 4: assigned [mem 0x600000000-0x600003fff 64bit]
[    0.141587] pci 0000:01:00.0: BAR 2: assigned [mem 0x600004000-0x600004fff 64bit]
[    0.141614] pci 0000:01:00.0: BAR 0: no space for [io  size 0x0100]
[    0.141629] pci 0000:01:00.0: BAR 0: failed to assign [io  size 0x0100]
[    0.141643] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.141659] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
[    0.141806] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    0.141928] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
[    0.142148] pcieport 0000:00:00.0: AER: enabled with IRQ 38
...
[    6.777869] r8169 0000:01:00.0: enabling device (0000 -> 0002)
[    6.792249] r8169 0000:01:00.0: can't read MAC address, setting random one
[    6.803140] libphy: r8169: probed
[    6.806902] r8169 0000:01:00.0 eth1: RTL8168h/8111h, d6:3a:e5:18:0b:68, XID 541, IRQ 39
[    6.814951] r8169 0000:01:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko]

I have been confused by whether I need to assign an address to the rtl8169 and if so, what address to use. Confused by the two PCI devices recognized as being in the same address space (0x600000000) though on different busses. Do I need to include devpnp.c?

After looking at ethergenet.c, and what the data sheet says on page 37 of the registers data sheet, I believe (correct me if I am wrong) that the IOBAR and MEMBAR need to have their addresses set. And looking at the BCM2711 ARM Peripherals datasheet (https://datasheets.raspberrypi.com/bcm2711/bcm2711-peripherals.pdf), it seems that the peripherals are usually mapped to either between 0x7C00 0000-0x8000 0000. Those addresses may also be represented as 0x4 7C00 0000 - 0x4 8000 0000 or 0x0FC00 0000 - 0x0 FF80 0000 depending on which view of the address map one is using? The VIRTIO seems to maps into that 0x7c00 0000 region through comments in mem.h:

mem.h:#define VIRTIO2		(0xFFFFFFFFBC000000ULL)	/* 0x7C000000	-		0xFC000000 */

and there are the other VIRTIO definitions…

mem.h:#define VIRTIO1		(0xFFFFFFFFBD000000ULL)	/* 0x7D000000	-		0xFD000000 */
mem.h:#define VIRTIO		(0xFFFFFFFFBE000000ULL) /* 0x7E000000	0x3F000000	0xFE000000 */

I see that ethergenet.c seems to assign an address through ctlr->regs…

static int
pnp(Ether *edev)
{
	static Ctlr ctlr[1];

	if(ctlr->regs != nil)
		return -1;

	ctlr->regs = (u32int*)(VIRTIO1 + 0x580000);
	ctlr->rx->regs = &ctlr->regs[RdmaOffset + nelem(ctlr->rd)*3 + 16*RingCfg];
	ctlr->tx->regs = &ctlr->regs[TdmaOffset + nelem(ctlr->td)*3 + 16*RingCfg];

	edev->port = (uintptr)ctlr->regs;

But I am confused by the pc version of the rtl8169 device - first calling rtl8169pnp which fairly soon after calls on rtl8169pci. I originally thought that I would set up my vmap under the pci routine. Looking at the PCIDev there, i am getting the correct information about the vendor and device id, but then there are some calls to vet (get) the mac address, but before a base address would be established under pnp? I suppose an address could be established in either pnp or pci, but I presume it makes more sense to do it in the pnp routine. I guess my question is how do I establish the address? I see in ethergenet addresses written to the ctlr->regs, but that driver doesn’t seem to have any mention of pci, so I am guessing that is not it. I’m guessing this has to be done through the Pcidev structure and some call to write to that address? pcicfgw maybe?  - Then what address do I use? VIRTIO1 + 0x680000 (I have no idea what the memory window is for ethergenet.c is, I just added the window of the other pci device, which of course is not located at VIRTIO1+0x580000, so 0x680000 is a bullshit number - so I need to figure out a free window above ethergenet? Is there a general rule of thumb regarding how and where such spaces should be allocated (e.g. every PCI device gets 16k or something like that?


Below is some of my initial (ignorant) coding in the rtl8169pci routine. I need to better understand the purpose for Ctlr->port, vs Etherdev->port, vs Ctlr->regs. I/O? DMA? Memory? 

At the bottom are the results of lspci. 

Hope that someone can help with some pieces of the puzzle.

Thanks,

mackbw


	print("Mem2 is %d bytes, and BAR: %llux\n", p->mem[2].size, p->mem[2].bar);
		if(p->mem[0].size == 0 || (p->mem[0].bar & 1) == 0)
			continue;

		if(p->mem[2].bar & 1) //checks to see if this is "memory" or "io"
			continue;
		io = p->mem[2].bar & ~0x0f;
		if(io == 0)
			continue;
		print("rtl8169: %#x %#x: port %llux size %d irq %d\n",
			p->vid, p->did, io, p->mem[2].size, p->intl);
		mmio = vmap(io, p->mem[2].size);
		if(mmio == nil){
			print("rtl8169: cannot map registers\n");
			continue;
		}
		ctlr = malloc(sizeof(Ctlr));
		if(ctlr == nil){
			print("rtl8169: can't allocate memory\n");
			vunmap(mmio, p->mem[2].size);
			continue;
		}
		ctlr->port = io;
		ctlr->mmio = mmio;

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
00: ec 10 68 81 06 04 10 00 15 00 00 02 10 00 00 00
10: 01 00 00 00 00 00 00 00 04 40 00 f8 00 00 00 00
20: 04 00 00 f8 00 00 00 00 00 00 00 00 ec 10 68 81
30: 00 00 00 00 40 00 00 00 00 00 00 00 26 01 00 00

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
        Flags: bus master, fast devsel, latency 0, IRQ 39
        I/O ports at <unassigned> [disabled]
        Memory at 600004000 (64-bit, non-prefetchable) [size=4K]
        Memory at 600000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked+
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [170] Latency Tolerance Reporting
        Capabilities: [178] L1 PM Substates
        Kernel driver in use: r8169
lspci: Unable to load libkmod resources: error -12

> On Nov 14, 2021, at 12:57 AM, cinap_lenrek@felloff.net wrote:
> 
> https://recomb-omsk.ru/published/SC/html/scripts/doc/RTL8111B_8168B_Registers_DataSheet_1.0.pdf
> 
> or just search for "rtl8111 registers datasheet" on the internet.
> 
> page 28 descibes that it has a 64-bit MEMBAR (bar2/bar3) for the
> registers as well as a IOBAR (bar0).
> 
> --
> cinap
>