I came to realize that the reason for my problems was the fact I had been moving cables around in my Netgear GS748T v5 switch and even though it seemed like the VLANs configs were correct, somehow my old PVID (Advanced-Port PVID Configuration) settings were messing things up. The scenario I have is 4 ESX hosts, one Synology array, plus one Internet link. I have four VLANS, 1=Default/home network, 10=iSCSI, 20=Internet, 30=VSAN traffic. I just upgraded my hosts to the Intel NUC's (because I want to be like William Lam), These Intel NUC's can only use the 1 onboard NIC with vSphere 6.0 U2 right now, hopefully someone will integrate a USB nic driver soon. So back to my challenge, the ESX hosts can ride on the default network and use VLAN tagging for access to the other 3 networks. My internet connection is a dumb device that can't use VLAN tagging, so I needed to find a way of integrating it. Normally that would just be an untagged port, but that doesn't work on these Netgear Switches. In order to get that to work I had to setup PVID, I used port g1 for Internet and g48 for iSCSI, and g39-42 for the ESXi hosts. The key here is that in the PVID settings, the port must be a Member of the VLAN, but not Tagged.
That seems to be working well. From the VLAN membership tab, I left my default VLAN (1) everywhere but the two untagged ports I will need my storage and internet connected to. For the other 3 VLANs I mostly emptied it out and set it up like this:
If you have a similar setup and you get stuck, I hope this helps you!
