ESXI Home Lab Questions

[notta]

I am setting up an ESXI home lab and I'm having a brain fart here. I have my ESXI (6.5) installed on an Dell R710. I want my lab network to be separated from my home network which is the standard 192.168.1. I'm going to make my lab network something like 192.168.150. How do I get internet access on my lab systems as my router is 192.168.1.1 and I cannot see that network when I'm on 192.168.150. I have a Linksys 1900 router and a Dell Power Connect switch. I know I have to setup a separate network on the ESXI and my R710 has plenty of Ethernet ports available. Any guidance would be appreciated.

[Jason S. Global Moderator]

is the Dell switch managed or dumb? where are the subnets set, exactly? perhaps you can change the mask from /24 to /16...

[notta]

Thanks for the response. It's somewhat dumb. It's a Dell Power Connect 2724. Been an absolutely great switch.

 

Well it seems I need another router to put the test lab behind. I then have a cable from the WAN port on the second router to switch 1. Since the second router only has 4 ports I guess this is also going to require me to get a second switch for behind the second router. Then my question becomes how do I access the test lab network from my main machine on the second floor? I guess I need a second NIC in my main machine to go to the second router or second switch? Could I also VLAN a single port on switch 1 to the second router? That would eliminate the need for a new run from the second floor? Or nm, that won't route right? I know this is basic network stuff but you think you know this stuff in theory but when you actually go to implement it, that's an entirely different story.

Edited July 17, 2017 by notta

[+BudMan MVC]

14 hours ago, notta said:

Linksys 1900 router

With such a crappy soho router you most likely can not run downstream networks, and unless you have some 3rd party firmware can not do it.  Does it support actual routing?  Can it do vlans and firewall rules between the vlans?

 

Is this esxi box going to be on all the time?  If so you could actually run a real router on your esxi, say pfsense or smoothwall or ipcop, etc.  I would highly suggest pfsense.  You could then just use pfsense as your internet router and route/firewall between any local networks you want to run..

 

Then you could use your linksys as just AP for wifi, or better yet get a real AP that also supports vlans.  If your wanting to setup a lab and start to isolate stuff you most likely will want a smart switch that can do vlans as well.  You can partly do isolation with just natting soho routers - but it would be a ###### setup.. and very limited into what you cold do and play with.

 

If you want I can throw together a simple drawing of how I have my home network setup, I have like 8 different network segments running both ipv4 and ipv6 with pfsense running as vm on my esxi host (6.5)..

[notta]

Thanks Bud. I actually already have your diagram up as we speak I've had it a while. Could you recommend some good hardware? I don't mind spending a little money. I don't like to do anything half ass I just purchased a VMUG license and a MSDN Pro license to get my all the software I need for the test lab. When I say test lab, I mean test/functional.

 

I already have an SFF Optiplex 790 with a 4-port NIC already installed with Pfsense installed, but  I have not had a chance to work on it. The rules are going to take time to setup so I will do that at a later date.

 

Sorry forgot to answer your question. The lab is mainly going to be on when I'm at home. When I go to work I will shut down VCenter and ESXI host. No need to waste power. If I get some type of Minecraft Server or MOHAA Server running it may stay up full time. I also have another R710 that I would like to add for clustering so I can fool with HA, but as I said power is a concern of mine and it's not a priority.

Edited July 17, 2017 by notta

[+BudMan MVC]

whatever drawing you might have is quite old at this point - been some major changes

 

So if your running pfsense on hardware already.. You would really just need a decent vlan capable switch, and then possible some real AP if you want to do more than just 1 network of wifi.

 

As to switch, I am a huge fan of the cisco sg300 line, or you could go with the unifi switches they seem to be very capable from a feature set point of view and not all that costly.  Comes down to the port density you need/want, etc.

 

As to AP, unifi all the way!!  They just rock for the price point.. They even just enabled up to 8 ssids per band as well..

[notta]

If it's not too much work I would love to see a new diagram With subnets would be great.

 

If you're saying I can achieve this with just VLAN's and no need for a second router then my current PowerConnect should work since it supports VLAN's. I will replace the Linksys with the PFSense box and get a Ubiquti AP after I get the PFSense box setup. For now I would like to get the test lab up and running.

 

I'm still having a difficult time visualizing the finish product. So I assume I have a single VLAN cable going from the main house switch going to a second switch in front of the test network? Everything behind the second switch will be 192.168.150.x. In the test lab I want multiple functionalities such as DNS, DHCP, and AD. Also multiple workstations joined to that domain. What is the gateway for all the devices behind the second switch? My only interaction with my main network will be from my main PC to the ESXI host to work on the test lab. Other than that no connectivity is needed between my main network.

 

After I get some more knowledge on this I would like to segregate my network camera's (currently on their own POE switch) and separate some other wireless devices from my main network.

 

Sorry I'm being so thick, but I'm a little confused by this. 

 

 

[+BudMan MVC]

do you need more than 24 ports total.. If not you only need the 1 vlan capable switch to do it all.  The whole point of vlan switch is to be able to break it up into little layer networks that are isolated from each other.  only reason you would need more than 1 switch would be more ports, or location of devices and ports needed in that location.  If your going to go physical with your pfsense (how many physical nics will it have?) could be done with just 1 but then your sharing a lot of bandwidth on just the one nic..  I would suggest atleast 2 so you have wan and then lan side, more the better if you want to have more networks so your not putting all networks having to share the same physical bandwidth of a nic.  Intervlan traffic on the same physical nic is a hairpin and your bandwidth is cut in half between those 2 vlans.

 

I think I have a drawing laying around with esxi and typical vlan setup... Let me see if can find it or redraw it.

 

How many nics do you have in your esxi host.. 2 is better if you can, more even better if you want to have lots of networks and you have the switch ports to be able to do it with vs having to vlan everything on limited number of nics.

 

BRB with a drawing..

 

edit:  Ok here is real quick (very ugly) drawing but I got some real work to do so did this is a couple of min.

 

In this setup your pfsense box has 2 nic, and your esxi box has 2.  One you use for management of the esxi box (vmkern) other you connect to another vswitch that would be on vlan 300 in this drawing.  Vmkern is on vlan 100.  So the different colors on your switch show which port are in which vlans and which ports are "trunk" or carry tagged vlans.  In are example 100,200 and 300.

 

So in your pfsense box you would have your 1 wan nic that would be connected to your modem or router.  This network would either be public or could be natted by your internet router..  Public is best so your not double natting.

 

Then on your other nic in pfsense you would create 3 vlans.  100, 200 and 300.  On your switch you would put whatever ports you want on the different vlans.  You would put 3 different networks on these lets call them 192.168.100/24 and 192.168.200/24 and 192.168.300/24

 

Does this help you visualize it?

 

So any device connected to a that is green would be on 192.168.200, on red would be 192.168.100 and purple would be 192.168.300

 

All of the gateways of these networks would be the IP address of pfsense for those vlans.. Lets make them 192.168.100.1, 200.1 and 300.1  You would create firewall rules as you see fit to allow or block whatever traffic you want between your different vlans.  So if device on vlan 200 wanted to talk to something on vlan 300 it would talk to its gateway 192.168.200.1 (pfsense) which would route and allow (firewall) the traffic to the 192.168.300 vlan.

 

This can be expanded with more nics or more switches as needed.  Does this help?

 

To do a setup like this you need to configure these vlans on your switch and on pfsense.  In this setup you would not have to do any thing special in esxi for the 2 vlans.  Now if you want more vlans for different vms and you don't have any more physical nics on your esxi then you would have to trunk a port to a nic in esxi (tagged vlans) and then on the vswitch in esxi you would create port groups with the different vlan IDs on them.

 

[notta]

WTH?? I didn't get any notification that you responded.

 

Dude, this is beautifully explained and has helped clear up things tremendously. I think the problem is that I don't, as of yet, fully understand how VLAN's work but have been reading a lot about them the past couple days. I wish I had some more time, I would draw you up a diagram of my network so you can see everything I have. Maybe this weekend. By the way I would still love to see an updated diagram of your network when you get some time.

 

To answer a few questions, the Optiplex 790 that I'll be using for my PFSense box has a total of 5 NIC's. Quad port Intel NIC and the on board NIC. Would you recommend 1 for Wan, 1 for House LAN, and 1 for Lab Network? That still leaves me 2 ports which 1 would have to be used for wireless. I have been reading and come to find that my AC1900 is crap and needs to be replaced. I could flash it with OpenWRT but is that over kill for an AP?

 

While I'm on wireless, I have not been real happy with it. I use my cell phone to do speed tests and the max I can get is 38 mbps. I'm watching people on youtube with these new mesh networks get some sick wireless speeds. Now my router is 2 floors down so I have been thinking I would move it up a floor or even get one of the mesh networks like the Orbi to get some better house coverage. If I test 2.4 upstairs the best I can get is 38. If I go right next to the router I get about the same. If I switch to 5GHZ I get 90 at the router, but get 8 upstairs so that's not an option.

 

Next, my R710 ESXI host has 4 NIC's plus the DRAC. Right now I have the following VM's: 2 domain controllers, SQL Server, and a Nessus Vulnerability Scanner (more on this in a second) with more to be added when I get going. I also plan to add that second R710 after I progress. Now, the Vulnerability scanner is a must for me. I want to consolidate my systems, which is partly why I'm setting up the lab, so I want to use the Nessus server to scan my network. It seems putting that system in the lab network is not an ideal setup. I would have to open that one VM to the entire network. It seems it would be better to setup another machine on the home network so it can access the important devices that I'm worried about. I just didn't want to have another machine for this when I could just use a VM. How would you handle this?

 

I also have 2 Kodi boxes that I would like to isolate because, well frankly I just don't trust them. They will need to access a QNAP NAS on my home network as that is where my media is stored. For this I guess I could put each on their own VLAN or group them in a single VLAN and only have access to the QNAP and separating them from everything else? I also have a a camera system from a Chinese company, and as you may have expected, I would like to isolate them as well because I don't trust them either Scratch that. I just purchased a NVR that puts the cameras on their own network, but the NVR I would like to lock down to my main machine so I can see them streams, but have access to nothing else.

 

Regarding the PFSense, I have not setup the PFSense box mainly because of my fear of creating an incorrect rule and opening up my network to something by accident. With the purchased router I have it in my mind that I'm secure as I can be because that's what these people do. I have been dreading dealing with everything breaking because everything is locked down, but that is also why I want PFSense because of the security. I plan to work on the PFSense box this weekend and just add it to my switch and configure it with a laptop that way it doesn't take down my entire network.

 

I am going to have to read your post several more times to get a grasp of it. I really appreciate the detailed post BudMan.

 

Thanks.

[notta]

Oh yea one more thing, multiple times yesterday I was on the brink of ordering that SG300. Then I kept fighting with myself that I'm crazy because I have a perfectly good switch in place. On top of that I found a brand new TP-Link 24 port managed switch on my shelf that I forgot I purchased a while back I even signed up for the Cisco course last night to get the packet tracing software to fool with some network designs. I wish the day had more than 24 hours. One of the things that you mentioned that I would like to have is VPN. That would be nice, but then again I'm too paranoid to open it up. I won't even open up my router for my camera's. It would be so nice to view my camera's remotely, but the risk is not worth the reward IMO.

[+BudMan MVC]

What risk with a vpn??  For anyone to access they have to have a cert signed by the CA you create.. Sorry but its secure - To a point this is how enterprises let their users into their networks, etc..   You don't have to worry about billy the script kiddy hacker accessing your network.

 

As to creating the wrong rule?  Its pretty impossible to do!  There are no WAN rules out of the box.. so unless you create port forwards inbound there would be no risk of anything from the outside accessing your stuff.

[notta]

Well, your statement that there are no WAN rules out of the box convinced me to upgrade. I disconnected my existing router last night and put PFSense in its place. I've only assigned the NIC's at this point, but it went very smoothly. I'm excited to have the weekend to work on this. I have been looking at the Unify AP's and they look very sweet and I see myself buying one this weekend after a little more research. I guess I could turn off DHCP and use the replaced Linksys router as an access point. We'll see.

 

I was thinking of putting my entire wireless network on it's own subnet. I don't see any reason to access any of the wireless devices from the main network. Is there any difference between connecting the AP to the 3rd port on the Pfsense box or connecting the AP to the main switch?

[+BudMan MVC]

If you want to put the wifi on their own network - sure have at it.  If you have a spare nic open on pfsense, then yeah you can leverage it as another network to connect your wifi too if you so desire.

 

Even if you do put your wifi on a different network, you can still allow wifi devices to access stuff on your wired network - just a simple firewall rule.  For example on my eap-tls wifi (my devices) I allow access to my plex server so I can stream movies and music to my ipad/phone/laptops..  Devices on my psk or guest wifi can not.. But the 3 different rokus all on their own wifi segment can access plex, etc.

 

Once you start segmenting your network you control what can or can not access what.

[notta]

Bud, what would you recommend? As I said I have 2 ports left on the quad NIC in the PFSense box. Would you use the interfaces on the PFSense box or setup VLAN's on the switch for the 2 remaining segments (Wireless and Test lab)? Using the switch takes the load off the PFSense box, but I lose the ability to use Firewall rules. I want to learn this stuff so I want to do it right. Thanks man.

[notta]

Duh, you can disregard my last post.

[notta]

I have a few more questions. I setup the interfaces on the PFSense box. Right now WAN and LAN have the default PFSense firewall rules. WAN is blocking all incoming and letting out all outgoing. LAN is allowing all traffic from LAN.NET to any (*). The OPT1 (LAB) and OPT2 (WIRELESS) have no rules assigned. What I don't understand is that my main PC that is on the LAN subnet can ping everything on all the subnets even though LAB and WIRELESS have no rules yet defined. Even though LAN is set to ANY I thought the LAB and WIRELESS would block any traffic because no rules have yet to be defined. Could someone explain why this is allowed?

 

So I purchased the Ubiquiti AP and setup a WIRELESS interface on PFSense. Now here I have to admit I was lost. Because you have to setup the AP with the Unify software I setup a laptop (just here for setup only. only wireless devices on this interface afterwards) and just set the IP to the 95 subnet. I connected to the switch thinking it was going to route though the LAN interface and then over to the WIRLESS interface. I have no idea how I was going to get there to be perfectly honest. Of course no matter what I did I could not ping the 192.168.95.1 and the Unify software could not see the AP. To complicate matters I'm sure the AP is probably defaulted to 192.168.1.1 but the WIRLESS interface is on 192.168.95.0. Also, should the AP be connected by the WAN port or the switch port on the AP? I could use some help on getting this configured.

 

Thanks.

Edited July 27, 2017 by notta

[+BudMan MVC]

46 minutes ago, notta said:

WAN is blocking all incoming and letting out all outgoing.

I think your misunderstanding how rules work in pfsense.. It does not do outbound rules on an interface.  Rules are evaluated as traffic enters and enters towards pfsense.  Top down, first rule to trigger wins - no other rules are evaluated.  So letting traffic OUT has nothing to do with rules on the WAN interface.. If you want LAN devices to get out, those rules would be on the LAN interface - as this is where the traffic enters pfsense.

 

Now you can get fancy with some outbound rules on an interface, but those are done on the floating tab.  And they by default are not "quick" rules so all rules are looked at in the floating tab, etc.

 

46 minutes ago, notta said:

should the AP be connected by the WAN port or the switch port on the AP? I could use some help on getting this configured.

Huh???  There is no wan/lan port on the AP.. There is a main port and a "secondary" port

Out of the box the AP would be dhcp.. So it would get an IP on the network its connected too.

 

What rules did you put on this wifi network?  Did you enable dhcp?  Where are you running the controller software?  It should be on this wifi network as well so you can do simple L2 discovery in the controller of the AP.  You can if you want later move to L3 management so your controller and AP run on different networks.  But I would not suggest that until your more familiar with it all.

46 minutes ago, notta said:

The OPT1 (LAB) and OPT2 (WIRELESS) have no rules assigned

Well dhcp will work once you enable it on an interface, those rules are hidden and enabled on an interface once you enable dhcp server on a interface.  But no device will be able to do anything if there are no rules setup..  While your first starting I would suggest any any rules just like the default lan rule until you get your feet wet!

 

So you setup vlans on your switch??  You should really connect your AP to the switch, then to the pfsense interface - this should be its own vlan in the switch.  Then put the box your running the controller on port on the switch so its in the same vlan.. Now your controller software will be able to see the AP!!  And you can set it up.

 

So more like this - so box running controller is on same L2 as your AP.

 

[notta]

Quote

So more like this - so box running controller is on same L2 as your AP.

I see what you're saying. I'm so worried about try to learn segmenting that I never thought to wire it that way. I will do that tonight when I get home. I have the the Unify software installed on the laptop. If both LAN and WIRELESS are now attached to the switch and both have DHCP enable on that interface what will be the result of the AP trying to acquire an IP? Are you saying setup the VLAN first between the ports and then allow the access point get the address from the WIRELESS interface?

 

I am taking this one step at a time. I will do the VLAN's after I get to a good point. I fooled with them the other day. I'm at work, but it seemed I had to define a VLANID (i.e 95). Then I went to the membership page and it shows a diagram of the switch ports. My options are Empty, Untagged and Tagged for each of the port. I didn't have any success getting them to communicate, but I will try again.