Raw ethernet packets

Raw ethernet packets

[Marcel Telka]

Hi,

I'm trying to send a raw ethernet packet from an userland application
but all my attempts so far were unsuccessful.  I basically tried two
approaches (see below) but maybe both are in wrong direction.  I'd
appreciate some help, suggestion or pointer to an working example or
application already doing that.


1) socket()/bind()

	int s = socket(AF_PACKET, SOCK_RAW, 0);

	struct sockaddr_ll llp = {};
	llp.sll_family = AF_PACKET;
	llp.sll_protocol = 0x0800;
	llp.sll_ifindex = 5;	/* this is from ifconfig -a */

	bind(s, (struct sockaddr *)&llp, sizeof (struct sockaddr_ll));

The bind() call above failed.


2) bpf

	/* Copy of real ARP packet from snoop capture */
	char buf[] =
	"\xff\xff\xff\xff\xff\xff"
	"\x00\x1c\x25\xa0\xb7\x2e"
	"\x08\x06\x00\x01"
	"\x08\x00\x06\x04\x00\x01"
	"\x00\x1c\x25\xa0\xb7\x2e"
	"\x0a\x00\x00\x0d"
	"\xff\xff\xff\xff\xff\xff"
	"\x0a\x00\x00\x37"
	;

	int fd = open("/dev/bpf", O_RDWR);

	struct ifreq ifr = { .ifr_name = {} };
	memcpy(&ifr.ifr_name, "e1000g0", 7);
	ioctl(fd, BIOCSETIF, (caddr_t)&ifr);

	int enable = 1;
	ioctl(fd, BIOCIMMEDIATE, (caddr_t)&enable);

	enable = 1;
	ioctl(fd, BIOCSHDRCMPLT, (caddr_t)&enable);

	write(fd, buf, sizeof buf - 1);


In this case all functions passed so it looks like everything works, but
I'm unable to snoop the packet on the network.


Thank you.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 10:43:03AM +0200, Marcel Telka wrote:
> 1) socket()/bind()
> 
>         int s = socket(AF_PACKET, SOCK_RAW, 0);
> 
>         struct sockaddr_ll llp = {};
>         llp.sll_family = AF_PACKET;
>         llp.sll_protocol = 0x0800;
>         llp.sll_ifindex = 5;    /* this is from ifconfig -a */
> 
>         bind(s, (struct sockaddr *)&llp, sizeof (struct sockaddr_ll));
> 
> The bind() call above failed.

... with ENOENT.  When I change the bind call to:

	bind(s, (struct sockaddr *)&llp, sizeof (struct sockaddr));

then it fails with EINVAL.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Joshua M. Clulow]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

On Fri, May 31, 2024, 01:43 Marcel Telka <marcel@telka.sk> wrote:

> I'm trying to send a raw ethernet packet from an userland application
> but all my attempts so far were unsuccessful.  I basically tried two
> approaches (see below) but maybe both are in wrong direction.  I'd
> appreciate some help, suggestion or pointer to an working example or
> application already doing that.
>

You probably want to look at the DLPI, and in particular libdlpi:

    https://illumos.org/man/4P/dlpi

There's a simple program in the gate that demonstrates sending an ethernet
frame using libdlpi interfaces:


https://src.illumos.org/source/xref/illumos-gate/usr/src/cmd/dlutil/dlsend.c?r=20768856

Cheers.

[-- Attachment #2: Type: text/html, Size: 1449 bytes --]

Re: [developer] Raw ethernet packets

[Denis Kozadaev]

witch% cat main.c 
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/sockio.h>
#include <sys/ioctl.h>

#include <net/if.h>

#include <stdio.h>
#include <unistd.h>
#include <string.h>

int
main(void)
{
        int                     sock;
        struct sockaddr_ll      sll;
        struct lifreq           ifr;
        char                    buf[8192], dev[] = "dpi0";
        ssize_t                 result;

        sock = socket(AF_PACKET, SOCK_RAW, 0);
        if (sock < 0) {
                perror("socket");
                return (1);
        }

        strcpy(ifr.lifr_name, dev);
        result = ioctl(sock, SIOCGLIFINDEX, &ifr);
        sll.sll_family = AF_PACKET;
        sll.sll_ifindex = ifr.lifr_index;
        sll.sll_protocol = 0;
        result = bind(sock, (struct sockaddr *)&sll, sizeof(sll));

        for (;;) {
                result = recvfrom(sock, buf, sizeof(buf), 0, NULL, NULL);
                fprintf(stderr, "received %d bytes\n", (int)result);
        }

        close(sock);

        return (0);
}
/****************************/
This is my old code I played long time ago.
I hope this will help you...

On Friday 31 May 2024 11:54:26 Marcel Telka wrote:
> On Fri, May 31, 2024 at 10:43:03AM +0200, Marcel Telka wrote:
> > 1) socket()/bind()
> >
> >         int s = socket(AF_PACKET, SOCK_RAW, 0);
> >
> >         struct sockaddr_ll llp = {};
> >         llp.sll_family = AF_PACKET;
> >         llp.sll_protocol = 0x0800;
> >         llp.sll_ifindex = 5;    /* this is from ifconfig -a */
> >
> >         bind(s, (struct sockaddr *)&llp, sizeof (struct sockaddr_ll));
> >
> > The bind() call above failed.
>
> ... with ENOENT.  When I change the bind call to:
>
>         bind(s, (struct sockaddr *)&llp, sizeof (struct sockaddr));
>
> then it fails with EINVAL.

Re: [developer] Raw ethernet packets

[Pramod Batni]

[-- Attachment #1: Type: text/plain, Size: 2555 bytes --]

Regarding 2.

    I think the reason snoop not showing the
    Ethernet packet is probably because /dev/bpf
   works at the same (or lower) level than
   the level at which snoop captures the packet.


   Not sure but worth checking (is the
   Ethernet packet on the network by some
    other host on the same network)



On Fri, 31 May 2024 at 14:14, Marcel Telka <marcel@telka.sk> wrote:

> Hi,
>
> I'm trying to send a raw ethernet packet from an userland application
> but all my attempts so far were unsuccessful.  I basically tried two
> approaches (see below) but maybe both are in wrong direction.  I'd
> appreciate some help, suggestion or pointer to an working example or
> application already doing that.
>
>
> 1) socket()/bind()
>
>         int s = socket(AF_PACKET, SOCK_RAW, 0);
>
>         struct sockaddr_ll llp = {};
>         llp.sll_family = AF_PACKET;
>         llp.sll_protocol = 0x0800;
>         llp.sll_ifindex = 5;    /* this is from ifconfig -a */
>
>         bind(s, (struct sockaddr *)&llp, sizeof (struct sockaddr_ll));
>
> The bind() call above failed.
>
>
> 2) bpf
>
>         /* Copy of real ARP packet from snoop capture */
>         char buf[] =
>         "\xff\xff\xff\xff\xff\xff"
>         "\x00\x1c\x25\xa0\xb7\x2e"
>         "\x08\x06\x00\x01"
>         "\x08\x00\x06\x04\x00\x01"
>         "\x00\x1c\x25\xa0\xb7\x2e"
>         "\x0a\x00\x00\x0d"
>         "\xff\xff\xff\xff\xff\xff"
>         "\x0a\x00\x00\x37"
>         ;
>
>         int fd = open("/dev/bpf", O_RDWR);
>
>         struct ifreq ifr = { .ifr_name = {} };
>         memcpy(&ifr.ifr_name, "e1000g0", 7);
>         ioctl(fd, BIOCSETIF, (caddr_t)&ifr);
>
>         int enable = 1;
>         ioctl(fd, BIOCIMMEDIATE, (caddr_t)&enable);
>
>         enable = 1;
>         ioctl(fd, BIOCSHDRCMPLT, (caddr_t)&enable);
>
>         write(fd, buf, sizeof buf - 1);
>
>
> In this case all functions passed so it looks like everything works, but
> I'm unable to snoop the packet on the network.
>
>
> Thank you.
>
> --
> +-------------------------------------------+
> | Marcel Telka   e-mail:   marcel@telka.sk  |
> |                homepage: http://telka.sk/ |
> +-------------------------------------------+
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink:
> https://illumos.topicbox.com/groups/developer/T9ea91ef91413959c-M0f5ac7181f7a9e8e1120f867
> Delivery options:
> https://illumos.topicbox.com/groups/developer/subscription
>

[-- Attachment #2: Type: text/html, Size: 3921 bytes --]

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 03:13:35PM +0530, Pramod Batni wrote:
> Regarding 2.
> 
>     I think the reason snoop not showing the
>     Ethernet packet is probably because /dev/bpf
>    works at the same (or lower) level than
>    the level at which snoop captures the packet.
> 
> 
>    Not sure but worth checking (is the
>    Ethernet packet on the network by some
>     other host on the same network)

Yes, I snooped this on a different host than the sending one.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 02:03:07AM -0700, Joshua M. Clulow via illumos-developer wrote:
> On Fri, May 31, 2024, 01:43 Marcel Telka <marcel@telka.sk> wrote:
> 
> > I'm trying to send a raw ethernet packet from an userland application
> > but all my attempts so far were unsuccessful.  I basically tried two
> > approaches (see below) but maybe both are in wrong direction.  I'd
> > appreciate some help, suggestion or pointer to an working example or
> > application already doing that.
> >
> 
> You probably want to look at the DLPI, and in particular libdlpi:
> 
>     https://illumos.org/man/4P/dlpi
> 
> There's a simple program in the gate that demonstrates sending an ethernet
> frame using libdlpi interfaces:
> 
> 
> https://src.illumos.org/source/xref/illumos-gate/usr/src/cmd/dlutil/dlsend.c?r=20768856

I'm not sure I'll be able/allowed to use libdlpi for this, but
definitely this is valuable suggestion.

Thank you.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 12:13:37PM +0300, Denis Kozadaev wrote:
> witch% cat main.c 

[snip]

> This is my old code I played long time ago.
> I hope this will help you...

Great!  The packet receive seems to works here,  I'm going to try
sending soon.


Thank you.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 11:51:42AM +0200, Marcel Telka wrote:
> On Fri, May 31, 2024 at 12:13:37PM +0300, Denis Kozadaev wrote:
> > witch% cat main.c 
> 
> [snip]
> 
> > This is my old code I played long time ago.
> > I hope this will help you...
> 
> Great!  The packet receive seems to works here,  I'm going to try
> sending soon.

Lesson learned: the sll_ifindex is not the same as index reported by
ifconfig -a :-).

Anyway, the sending still does not work.  All below pass (the send
returns 60), but no packet on the network (snooped on both sending host
and another host on the same LAN):


	char buf[] =
	"\xff\xff\xff\xff\xff\xff"
	"\x00\x1c\x25\xa0\xb7\x2e"
	"\x08\x06\x00\x01"
	"\x08\x00\x06\x04\x00\x01"
	"\x00\x1c\x25\xa0\xb7\x2e"
	"\x0a\x00\x00\x0d"
	"\xff\xff\xff\xff\xff\xff"
	"\x0a\x00\x00\x3d"
	"\x00\x00\x00\x00\x00\x00"
	"\x00\x00\x00\x00\x00\x00"
	"\x00\x00\x00\x00\x00\x00"
	;

	s = socket(AF_PACKET, SOCK_RAW, 0);

	struct lifreq lifr;
	strcpy(lifr.lifr_name, "e1000g0");
	ioctl(s, SIOCGLIFINDEX, &lifr);

	struct sockaddr_ll llp = {};
	llp.sll_family = AF_PACKET;
	llp.sll_protocol = 0;
	llp.sll_ifindex = lifr.lifr_index;

	bind(s, (struct sockaddr *)&llp, sizeof llp);

	send(s, buf, sizeof buf - 1, 0);

So this is basically the same outcome as with my bpf approach :-(.

I suspect there might be some packet filtering involved in kernel?  This
is mostly the default OpenIndiana installation (up-to-date) on real HW
(not a VM).


To check that there is really no packet on the network I started snoop
on another machine (a snoop machine).  When I try `ping 10.0.0.61` (on
machine 10.0.0.13) I see following (on the snoop machine):

ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 24 arrived at 12:54:11.78570
ETHER:  Packet size = 60 bytes
ETHER:  Destination = ff:ff:ff:ff:ff:ff, (broadcast)
ETHER:  Source      = 0:1c:25:a0:b7:2e, 
ETHER:  Ethertype = 0806 (ARP)
ETHER:  
ARP:  ----- ARP/RARP Frame -----
ARP:  
ARP:  Hardware type = 1 (Ethernet (10Mb))
ARP:  Protocol type = 0800 (IP)
ARP:  Length of hardware address = 6 bytes
ARP:  Length of protocol address = 4 bytes
ARP:  Opcode 1 (ARP Request)
ARP:  Sender's hardware address = 0:1c:25:a0:b7:2e
ARP:  Sender's protocol address = 10.0.0.13, 10.0.0.13
ARP:  Target hardware address = ?
ARP:  Target protocol address = 10.0.0.61, 10.0.0.61
ARP:  


           0: ffff ffff ffff 001c 25a0 b72e 0806 0001    ........%.......
          16: 0800 0604 0001 001c 25a0 b72e 0a00 000d    ........%.......
          32: ffff ffff ffff 0a00 003d 0000 0000 0000    .........=......
          48: 0000 0000 0000 0000 0000 0000              ............

Please note the packet data (should be same as the buf above).
The IP address 10.0.0.61 does not exist on the network.

With this setup when I try the code above on machine 10.0.0.13 I see
nothing in the snoop output on the snoop machine.

Any suggestion where to look next?


Thank you.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Denis Kozadaev]

The send() function can be used only when the socket is in a connected state.
(c) manual page at https://illumos.org/man/3SOCKET/send
Use sendto(), Marcel.

On Friday 31 May 2024 14:08:45 Marcel Telka wrote:
> On Fri, May 31, 2024 at 11:51:42AM +0200, Marcel Telka wrote:
[snip]

>         send(s, buf, sizeof buf - 1, 0);
>
> So this is basically the same outcome as with my bpf approach :-(.

>
> I suspect there might be some packet filtering involved in kernel?  This
> is mostly the default OpenIndiana installation (up-to-date) on real HW
> (not a VM).
>
>
> To check that there is really no packet on the network I started snoop
> on another machine (a snoop machine).  When I try `ping 10.0.0.61` (on
> machine 10.0.0.13) I see following (on the snoop machine):
>
> ETHER:  ----- Ether Header -----
> ETHER:
> ETHER:  Packet 24 arrived at 12:54:11.78570
> ETHER:  Packet size = 60 bytes
> ETHER:  Destination = ff:ff:ff:ff:ff:ff, (broadcast)
> ETHER:  Source      = 0:1c:25:a0:b7:2e,
> ETHER:  Ethertype = 0806 (ARP)
> ETHER:
> ARP:  ----- ARP/RARP Frame -----
> ARP:
> ARP:  Hardware type = 1 (Ethernet (10Mb))
> ARP:  Protocol type = 0800 (IP)
> ARP:  Length of hardware address = 6 bytes
> ARP:  Length of protocol address = 4 bytes
> ARP:  Opcode 1 (ARP Request)
> ARP:  Sender's hardware address = 0:1c:25:a0:b7:2e
> ARP:  Sender's protocol address = 10.0.0.13, 10.0.0.13
> ARP:  Target hardware address = ?
> ARP:  Target protocol address = 10.0.0.61, 10.0.0.61
> ARP:
>
>
>            0: ffff ffff ffff 001c 25a0 b72e 0806 0001    ........%.......
>           16: 0800 0604 0001 001c 25a0 b72e 0a00 000d    ........%.......
>           32: ffff ffff ffff 0a00 003d 0000 0000 0000    .........=......
>           48: 0000 0000 0000 0000 0000 0000              ............
>
> Please note the packet data (should be same as the buf above).
> The IP address 10.0.0.61 does not exist on the network.
>
> With this setup when I try the code above on machine 10.0.0.13 I see
> nothing in the snoop output on the snoop machine.
>
> Any suggestion where to look next?
>
>
> Thank you.

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 02:23:04PM +0300, Denis Kozadaev wrote:
> The send() function can be used only when the socket is in a connected state.
> (c) manual page at https://illumos.org/man/3SOCKET/send
> Use sendto(), Marcel.

	sendto(s, buf, sizeof buf - 1, 0, NULL, 0);

gives the same result: succeeds, but no packet on the network.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Denis Kozadaev]

On Friday 31 May 2024 15:07:13 Marcel Telka wrote:
> On Fri, May 31, 2024 at 02:23:04PM +0300, Denis Kozadaev wrote:
> > The send() function can be used only when the socket is in a connected
> > state. (c) manual page at https://illumos.org/man/3SOCKET/send
> > Use sendto(), Marcel.
>
>         sendto(s, buf, sizeof buf - 1, 0, NULL, 0);
>
> gives the same result: succeeds, but no packet on the network.
it is because
send(sock, buf, len, flags)
could be equivalent to
sendto(sock, buf, len, flags, NULL, 0)
it depend of the implementation
You should specify a real address for sendto()
and yes, the packet and the address have duplicate of this data.
Try it.

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 03:18:23PM +0300, Denis Kozadaev wrote:
> On Friday 31 May 2024 15:07:13 Marcel Telka wrote:
> > On Fri, May 31, 2024 at 02:23:04PM +0300, Denis Kozadaev wrote:
> > > The send() function can be used only when the socket is in a connected
> > > state. (c) manual page at https://illumos.org/man/3SOCKET/send
> > > Use sendto(), Marcel.
> >
> >         sendto(s, buf, sizeof buf - 1, 0, NULL, 0);
> >
> > gives the same result: succeeds, but no packet on the network.
> it is because
> send(sock, buf, len, flags)
> could be equivalent to
> sendto(sock, buf, len, flags, NULL, 0)
> it depend of the implementation
> You should specify a real address for sendto()
> and yes, the packet and the address have duplicate of this data.

I tried to add this:


	llp.sll_pkttype = PACKET_BROADCAST;
	llp.sll_hatype = ARPHRD_ETHER;
	llp.sll_halen = 6;
	llp.sll_addr[0] = 0xff;
	llp.sll_addr[1] = 0xff;
	llp.sll_addr[2] = 0xff;
	llp.sll_addr[3] = 0xff;
	llp.sll_addr[4] = 0xff;
	llp.sll_addr[5] = 0xff;

	sendto(s, buf, sizeof buf - 1, 0, (struct sockaddr *)&llp, sizeof llp);

And the result is stil the same: the function succeeded, no packet on
the network.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 11:50:07AM +0200, Marcel Telka wrote:
> On Fri, May 31, 2024 at 02:03:07AM -0700, Joshua M. Clulow via illumos-developer wrote:
> > On Fri, May 31, 2024, 01:43 Marcel Telka <marcel@telka.sk> wrote:
> > 
> > > I'm trying to send a raw ethernet packet from an userland application
> > > but all my attempts so far were unsuccessful.  I basically tried two
> > > approaches (see below) but maybe both are in wrong direction.  I'd
> > > appreciate some help, suggestion or pointer to an working example or
> > > application already doing that.
> > >
> > 
> > You probably want to look at the DLPI, and in particular libdlpi:
> > 
> >     https://illumos.org/man/4P/dlpi
> > 
> > There's a simple program in the gate that demonstrates sending an ethernet
> > frame using libdlpi interfaces:
> > 
> > 
> > https://src.illumos.org/source/xref/illumos-gate/usr/src/cmd/dlutil/dlsend.c?r=20768856
> 
> I'm not sure I'll be able/allowed to use libdlpi for this, but
> definitely this is valuable suggestion.

FYI, the following works:

/usr/lib/dl/dlsend e1000g0 ff:ff:ff:ff:ff:ff

Snooped on another machine:

15:42:09.75924            ? -> (broadcast)  drops: 0 ETHER Type=DEED (Unknown), size=310 bytes

So apparently there is some way how to do that.  I just need to find how
to make it working without libdlpi/libdladm.


Thank you.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Denis Kozadaev]

[-- Attachment #1: Type: text/plain, Size: 3777 bytes --]

Ok, I tryed to reproduce your task.
Speaking shortly it works, see the attached file.
A test system is a VM undef FreeBSD bhyve, the global zone has an interface:
vioif0: flags=1000943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,IPv4> mtu 1500 
index 2
        inet 192.168.1.3 netmask ffffff00 broadcast 192.168.1.255
Over this nic I use VNIC and test it in a non-global zone:
LINK         OVER       SPEED    MACADDRESS        MACADDRTYPE VID  ZONE
dev0         vioif0     10000    2:8:20:2d:76:5d   fixed       0    dev01

Now the non-global zone:
dev01% ifconfig dev0
dev0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.1.10 netmask ffffff00 broadcast 192.168.1.255

dev01% gcc -o tst main.c
dev01% pfexec id
uid=0(root) gid=0(root) группы=0(root),1000(denis)
dev01% pfexec ./tst
result = 60; errno = 0

On the host the NIC is attached to a bridge via VMNET:
asus# ifconfig bridge0
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 
0 mtu 1500
	options=0
	ether 58:9c:fc:10:c6:27
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vmnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 4 priority 128 path cost 2000000
	groups: bridge
	nd6 options=9<PERFORMNUD,IFDISABLED>
asus# ifconfig vmnet0
vmnet0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> 
metric 0 mtu 1500
	options=80000<LINKSTATE>
	ether 58:9c:fc:10:0d:5d
	hwaddr 58:9c:fc:00:58:3f
	groups: vmnet
	media: Ethernet 1000baseT <full-duplex>
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	Opened by PID 4683
And the output of tcpdump on the bridge0:
asus# tcpdump -i bridge0 -n not port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), snapshot length 262144 
bytes
17:05:02.479797 ARP, Request who-has 10.0.0.61 (ff:ff:ff:ff:ff:ff) tell 
10.0.0.13, length 46
^C
1 packet captured
17 packets received by filter
0 packets dropped by kernel

So, 10.0.0.61 and 10.0.0.13 are from your packet,
therefore it works (in my test env)

On Friday 31 May 2024 16:33:05 Marcel Telka wrote:
> On Fri, May 31, 2024 at 03:18:23PM +0300, Denis Kozadaev wrote:
> > On Friday 31 May 2024 15:07:13 Marcel Telka wrote:
> > > On Fri, May 31, 2024 at 02:23:04PM +0300, Denis Kozadaev wrote:
> > > > The send() function can be used only when the socket is in a
> > > > connected state. (c) manual page at
> > > > https://illumos.org/man/3SOCKET/send Use sendto(), Marcel.
> > >
> > >         sendto(s, buf, sizeof buf - 1, 0, NULL, 0);
> > >
> > > gives the same result: succeeds, but no packet on the network.
> >
> > it is because
> > send(sock, buf, len, flags)
> > could be equivalent to
> > sendto(sock, buf, len, flags, NULL, 0)
> > it depend of the implementation
> > You should specify a real address for sendto()
> > and yes, the packet and the address have duplicate of this data.
>
> I tried to add this:
>
>
>         llp.sll_pkttype = PACKET_BROADCAST;
>         llp.sll_hatype = ARPHRD_ETHER;
>         llp.sll_halen = 6;
>         llp.sll_addr[0] = 0xff;
>         llp.sll_addr[1] = 0xff;
>         llp.sll_addr[2] = 0xff;
>         llp.sll_addr[3] = 0xff;
>         llp.sll_addr[4] = 0xff;
>         llp.sll_addr[5] = 0xff;
>
>         sendto(s, buf, sizeof buf - 1, 0, (struct sockaddr *)&llp, sizeof
> llp);
>
> And the result is stil the same: the function succeeded, no packet on
> the network.


[-- Attachment #2: main.c --]
[-- Type: text/x-csrc, Size: 1933 bytes --]

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/sockio.h>
#include <sys/ioctl.h>

#include <net/if.h>

#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>

int
main(void)
{
        int                     sock;
        struct sockaddr_ll      sll, llp;
        struct lifreq           ifr;
        char                    buf[8192], dev[] = "dev0";
        ssize_t                 result;
        char pkt[] =
        "\xff\xff\xff\xff\xff\xff"
        "\x00\x1c\x25\xa0\xb7\x2e"
        "\x08\x06\x00\x01"
        "\x08\x00\x06\x04\x00\x01"
        "\x00\x1c\x25\xa0\xb7\x2e"
        "\x0a\x00\x00\x0d"
        "\xff\xff\xff\xff\xff\xff"
        "\x0a\x00\x00\x3d"
        "\x00\x00\x00\x00\x00\x00"
        "\x00\x00\x00\x00\x00\x00"
        "\x00\x00\x00\x00\x00\x00"
        ;
        sock = socket(AF_PACKET, SOCK_RAW, 0);
        if (sock < 0) {
                perror("socket");
                return (1);
        }

        strcpy(ifr.lifr_name, dev);
        result = ioctl(sock, SIOCGLIFINDEX, &ifr);
        sll.sll_family = AF_PACKET;
        sll.sll_ifindex = ifr.lifr_index;
        sll.sll_protocol = 0;
        result = bind(sock, (struct sockaddr *)&sll, sizeof(sll));

/*        llp.sll_pkttype = PACKET_BROADCAST;
        llp.sll_hatype = ARPHRD_ETHER;
        llp.sll_halen = 6;
        llp.sll_addr[0] = 0xff;
        llp.sll_addr[1] = 0xff;
        llp.sll_addr[2] = 0xff;
        llp.sll_addr[3] = 0xff;
        llp.sll_addr[4] = 0xff;
        llp.sll_addr[5] = 0xff; */

	result = send(sock, pkt, sizeof(pkt) - 1, 0);
	fprintf(stderr, "result = %d; errno = %d\n", result, errno);
/*        for (;;) {
                result = recvfrom(sock, buf, sizeof(buf), 0, NULL, NULL);
                fprintf(stderr, "received %d bytes\n", (int)result);
        } */

        close(sock);

        return (0);
}

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Fri, May 31, 2024 at 05:15:39PM +0300, Denis Kozadaev wrote:
> Ok, I tryed to reproduce your task.
> Speaking shortly it works, see the attached file.

Thank you for the test!

Was this tested on illumos?  Your `gcc -o tst main.c` suggests it is not
a vanilla illumos machine because to link successfully I had to add
-lsocket.

There is definitely something strange because it does not work here.  I
tried your main.c on a different machine (a virtualbox guest with
up-to-date OpenIndiana) with replaced "dev0" by "e1000g0" and I see the
same results:

result = 60; errno = 0

and no packet on the LAN.


Thanks.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Denis Kozadaev]

On Friday 31 May 2024 18:04:22 Marcel Telka wrote:
> Thank you for the test!
>
> Was this tested on illumos?  Your `gcc -o tst main.c` suggests it is not
> a vanilla illumos machine because to link successfully I had to add
> -lsocket.
it was tested under dilos (probably illumos-like system)
the network stack is the same.


> There is definitely something strange because it does not work here.  I
> tried your main.c on a different machine (a virtualbox guest with
> up-to-date OpenIndiana) with replaced "dev0" by "e1000g0" and I see the
> same results:
>
> result = 60; errno = 0
>
> and no packet on the LAN.
What would I do?
connect an empty hardware NIC directly to another hardware NIC
and snoop traffic on the test host (on the test NIC)
you should see the same packet on the outgoing and remote NICs.
it could be a kernel or a hardware filter if you don't see packets on 
the remote NIC.


> Thanks.
you are welcome

Re: [developer] Raw ethernet packets

[Marcel Telka]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

On Fri, May 31, 2024 at 10:43:03AM +0200, Marcel Telka wrote:
> I'm trying to send a raw ethernet packet from an userland application
> but all my attempts so far were unsuccessful.  I basically tried two

Attached is the testing program ethersend.c that uses both bpf and
socket approaches to send the raw ethernet frame.  In both cases the
frame is silently dropped in mac_tx() because fe_tx_srs is NULL:

https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/mac/mac_client.c?r=119d61cc&mo=100885&fi=3519#3576

There is attached the ethersend.d dtrace script that shows the issue:

# gcc -Wall -lsocket -o ethersend ethersend.c
# ./ethersend.d -c './ethersend e1000g0' 2>&1 | grep mac_tx
  1              -> mac_tx                    
  1               | mac_tx:entry                              0
  1              <- mac_tx                                    0
  1            -> mac_tx                      
  1             | mac_tx:entry                                0
  1            <- mac_tx                                      0
#

Since something similar reportedly works on dilos (I assume it is a fork
of old illumos-gate) then it looks like the issue is in new illumos.

I'll try to do some archaeology...

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

[-- Attachment #2: ethersend.c --]
[-- Type: text/plain, Size: 2727 bytes --]

#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <net/if.h>
#include <net/bpf.h>
#include <sys/socket.h>
#include <sys/sockio.h>

char packet[] =
	"\xff\xff\xff\xff\xff\xff"
	"\x00\x1c\x25\xa0\xb7\x2e"
	"\x08\x06\x00\x01"
	"\x08\x00\x06\x04\x00\x01"
	"\x00\x1c\x25\xa0\xb7\x2e"
	"\x0a\x00\x00\x0d"
	"\xff\xff\xff\xff\xff\xff"
	"\x0a\x00\x00\x3d"
	"\x00\x00\x00\x00\x00\x00"
	"\x00\x00\x00\x00\x00\x00"
	"\x00\x00\x00\x00\x00\x00"
;

int
send_bpf(const char *iface, char *data, size_t len)
{
	int fd = open("/dev/bpf", O_RDWR);
	if (fd < 0) {
		perror("open");
		return -1;
	}

	struct ifreq ifr;

	(void) memset(&ifr, 0, sizeof ifr);
	(void) memcpy(&ifr.ifr_name, iface, strlen(iface));
	if (ioctl(fd, BIOCSETIF, &ifr) < 0) {
		perror("BIOCSETIF");
		return -2;
	}

	int enable;
       
	enable = 1;
	if (ioctl(fd, BIOCIMMEDIATE, &enable) < 0) {
		perror("BIOCIMMEDIATE");
		return -3;
	}

	enable = 1;
	if (ioctl(fd, BIOCSHDRCMPLT, &enable) < 0) {
		perror("BIOCSHDRCMPLT");
		return -4;
	}

	size_t ret = write(fd, data, len);
	if (ret >= 0)
		printf("send_bpf sent %ld bytes\n", ret);
	if (ret != len) {
		fprintf(stderr, "write returned %ld, expected %ld\n", ret, len);
		return -5;
	}

	if (close(fd) < 0) {
		perror("close");
		return -6;
	}

	return 0;
}

int
send_socket(const char *iface, char *data, size_t len)
{
	int s = socket(AF_PACKET, SOCK_RAW, 0);
	if (s < 0) {
		perror("socket");
		return -1;
	}

	struct ifreq ifr;

	(void) memset(&ifr, 0, sizeof ifr);
	(void) memcpy(&ifr.ifr_name, iface, strlen(iface));
	if (ioctl(s, SIOCGIFINDEX, &ifr) < 0) {
		perror("SIOCGIFINDEX");
		return -2;
	}

	struct sockaddr_ll lls;
	(void) memset(&lls, 0, sizeof lls);
	lls.sll_family = AF_PACKET;
	lls.sll_protocol = 0;
	lls.sll_ifindex = ifr.ifr_index;

	if (bind(s, (struct sockaddr *)&lls, sizeof lls) < 0) {
		perror("bind");
		return -3;
	}

	size_t ret = send(s, data, len, 0);
	if (ret >= 0)
		printf("send_socket sent %ld bytes\n", ret);
	if (ret != len) {
		fprintf(stderr, "send returned %ld, expected %ld\n", ret, len);
		return -4;
	}

	if (close(s) < 0) {
		perror("close");
		return -5;
	}

	return 0;
}

int main(int argc, char *argv[])
{
	if (argc != 2) {
		fprintf(stderr, "Missing interface\n");
		return 1;
	}

	int ret;

	ret = send_bpf(argv[1], packet, sizeof packet - 1);
	if (ret != 0)
		fprintf(stderr, "send_bpf: %d\n", ret);

	ret = send_socket(argv[1], packet, sizeof packet - 1);
	if (ret != 0)
		fprintf(stderr, "send_socket: %d\n", ret);

	return 0;
}

[-- Attachment #3: ethersend.d --]
[-- Type: text/plain, Size: 430 bytes --]

#!/usr/sbin/dtrace -s

#pragma D option flowindent

fbt::send:entry
{
	self->t = 1;
}

fbt::write:entry
/execname == "ethersend" && arg0 > 2/
{
	self->t = 1;
}

fbt:::entry
/self->t/
{
}

fbt:::return
/self->t/
{
	trace(arg1);
}

mac_tx:entry
/self->t/
{
	trace(((mac_client_impl_t *)arg0)->mci_flent->fe_tx_srs);
}

fbt::send:return
{
	self->t = 0;
}

fbt::write:return
{
	self->t = 0;
}

Re: [developer] Raw ethernet packets

[Marcel Telka]

On Sun, Jun 02, 2024 at 11:05:40AM +0200, Marcel Telka wrote:
> On Fri, May 31, 2024 at 10:43:03AM +0200, Marcel Telka wrote:
> > I'm trying to send a raw ethernet packet from an userland application
> > but all my attempts so far were unsuccessful.  I basically tried two
> 
> Attached is the testing program ethersend.c that uses both bpf and
> socket approaches to send the raw ethernet frame.  In both cases the
> frame is silently dropped in mac_tx() because fe_tx_srs is NULL:
> 
> https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/mac/mac_client.c?r=119d61cc&mo=100885&fi=3519#3576
> 
> There is attached the ethersend.d dtrace script that shows the issue:
> 
> # gcc -Wall -lsocket -o ethersend ethersend.c
> # ./ethersend.d -c './ethersend e1000g0' 2>&1 | grep mac_tx
>   1              -> mac_tx                    
>   1               | mac_tx:entry                              0
>   1              <- mac_tx                                    0
>   1            -> mac_tx                      
>   1             | mac_tx:entry                                0
>   1            <- mac_tx                                      0
> 
> Since something similar reportedly works on dilos (I assume it is a fork
> of old illumos-gate) then it looks like the issue is in new illumos.
> 
> I'll try to do some archaeology...

The archeology revealed that the ethersend works on Solaris, but does
not on work on even very old illumos:

Solaris with the net0 network device:

Solaris 11.1 FCS	10/2012		WORKS!
Solaris 11.4 FCS	08/2018		WORKS!

Below is illumos.  All tests with e1000g0 device:

oi_147			10/2010		DOES NOT WORK
oi_151a8		08/2013		DOES NOT WORK
hipster_20131023	10/2013		DOES NOT WORK
dilos 2.0.2.48		11/2019		DOES NOT WORK

I also tested up-to-date illumos (OpenIndiana) with vioif0 interface
(qemu/kvm guest) and it does not work.

Solaris apparently changed something since we parted and even there is
the same hardware underneath (82540EM, VirtualBox) their networking
stack looks differently (net0 vs. e1000g0).  The ethersend simply works
there.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel@telka.sk  |
|                homepage: http://telka.sk/ |
+-------------------------------------------+

Re: [developer] Raw ethernet packets

[Alan Coopersmith]

On 6/6/24 05:54, Marcel Telka wrote:
> Solaris apparently changed something since we parted and even there is
> the same hardware underneath (82540EM, VirtualBox) their networking
> stack looks differently (net0 vs. e1000g0). 

You should have the underlying technology from the Clearview project,
the change in Solaris 11 was to enable the "vanity naming" by default
for all physical links:

      This project provides default physical network device
      datalink names based on a common prefix - "net" by default -
      and an instance number based on the physical location of the
      device within the system.  Instead of having physical datalinks
      with names tied to the underlying hardware - e.g. e1000g0,1 or
      bge0,1 - the more generic names net0, net1 etc are used.  The
      ordering of instances net0-N is based on the relative physical location
      of the network devices in the system.    This work will also leverage
      libtopo and DDI properties to provide slot location information in
      dladm.
	

	-alan-