Building a ESXi box...

[riahc3]

Hello,

I problably wont get this now because I plan to buy this "part per month" to get costs down but Id like to build a ESXi box.

Now most will recommend me the HP microserver (this or previous gen). What pros and cons are there to this?

This ESXi box will run:

Windows Server 2012 R2 (my domain controller)

pfSense (my router)

The WS2012R2 will have StableBit DrivePool to setup some storage pools and divide 4 disks of 3TB into my personal space and two other family members so when they log on they will have their own personal space and will also load profiles, settings, etc from it. So in a sense, it will also act as a NAS to stream to them and to media device thru my home.

Id like to hear some opinions :)

Thank you

[The_Observer]

I just moved from DrivePool to FlexRaid. Had DrivePool for years, but losing half my spaces wasn't cool. Reddit has a great VMware subreddit i recommend. I don't know how FlexRaid will work in VM. Good luck on this.

[Klownicle]

If you plan on using ESXi 5.5 beware the latest limits on legacy/consumer hardware no longer in the compatibility list.  You can get around this by using 5.1 U1 first then upgrading to 5.5 as the drivers from 5.1 U1 are kept during the upgrade.  Also, any 5.5 ESXi VM10 cannot be managed with the VmSphere Client, it must use the Web Client which isn't free.  So if you create anything to use the 5.5 features you must have a vCenter server which has the web client to fully manage.  The VmSphere Client will only be able to power on or use the console for VM's that are version 10.

 

A lot of legacy/consumer Sata controllers and Nic's have been removed.

[riahc3]

Hello,

  On 19/01/2014 at 23:58, Klownicle said:

If you plan on using ESXi 5.5 beware the latest limits on legacy/consumer hardware no longer in the compatibility list.  You can get around this by using 5.1 U1 first then upgrading to 5.5 as the drivers from 5.1 U1 are kept during the upgrade.  Also, any 5.5 ESXi VM10 cannot be managed with the VmSphere Client, it must use the Web Client which isn't free.  So if you create anything to use the 5.5 features you must have a vCenter server which has the web client to fully manage.  The VmSphere Client will only be able to power on or use the console for VM's that are version 10.

 

A lot of legacy/consumer Sata controllers and Nic's have been removed.

Yup, Ive seen there are a lot of limits now put in place by ESXi 5.5

I plan on consulting the VMWare Compatability list before hand and see what hardware is fully compatible.

I read about 5.5 ESXi VM10 not being able to be manages without a paid client but can I manage it with VMWare Workstation 10? Or do I have to use the the client?

Also Im a bit lost on the "vCenter server" bit; What exactly is this? Isnt the ESXi the server and thats it? Excuse me for my ignorance.

[Chicane-UK Veteran]

vCenter server is what you use if you need to centrally manage more than one ESXi server, and is basically required if you want to start using any of the more advanced VMware features such as clustering, vMotion, Storage DRS, etc etc. All this is underpinned by vCenter Server. 

 

If however you only want to run a single ESXi server, with a couple of VM's on it, you won't need vCenter :)

 

I have a number of friends who run ESXi on HP MicroServers and they're all pretty happy with it. As has been said though, VMware and some of the vendors have been turds with ESXi 5.5 and really limited the amount of hardware it can run on.. I'm bitter because a number of not even 3 year old HP servers at work are no longer supported for 5.5 and it's really scuppered our plans as we were expecting to go right from ESXi 5.0 to 5.5! So definitely check that HCL - I'd imagine the Gen8 MicroServer will be supported though so that would be an ideal platform to build on for a small home ESXi box. 

[riahc3]

Hello,

  On 20/01/2014 at 08:19, Chicane-UK said:

vCenter server is what you use if you need to centrally manage more than one ESXi server, and is basically required if you want to start using any of the more advanced VMware features such as clustering, vMotion, Storage DRS, etc etc. All this is underpinned by vCenter Server. 

 

If however you only want to run a single ESXi server, with a couple of VM's on it, you won't need vCenter :)

Then I dont think I need it. Im not gonna do anything except deal with one ESXi server.

 

  On 20/01/2014 at 08:19, Chicane-UK said:

I have a number of friends who run ESXi on HP MicroServers and they're all pretty happy with it. As has been said though, VMware and some of the vendors have been turds with ESXi 5.5 and really limited the amount of hardware it can run on.. I'm bitter because a number of not even 3 year old HP servers at work are no longer supported for 5.5 and it's really scuppered our plans as we were expecting to go right from ESXi 5.0 to 5.5! So definitely check that HCL - I'd imagine the Gen8 MicroServer will be supported though so that would be an ideal platform to build on for a small home ESXi box.

Yes, Ive read all the hate towards 5.5; Its even gotten me to think about staying on 5.1 even if things like USB 3.0 passthrough isnt avaliable (AFAIK, VM guests have to be at version 10 to support this, right?)

Also, for the HP MicroServers, arent they ECC RAM? Forgive my ignorance...

I would love to build a ESXi box but hey if the MicroServer Gen8 gives me all I need that I couldnt get from a DIY (and then some), why complicate myself with builds? I have no idea so thats why Im asking opinions :)

[riahc3]

Hello,

For the record I see HP ProLiant MicroServer Gen8 G1610T (2GB RAM, Intel CeleronG1610T) for 438,36?

This excludes G.Skill Ripjaws X DDR3 1333 PC3-10666 8GB at 71.95? each stick (143,90 ?) (why gaming RAM? Its actually cheaper than some Kingston valueram....)

Im saying this because I was tempted to max it out but I just saw that WS2012R2 only asks for 512MB RAM (long time I havent read that). I might be able to get away with less but I did read BudMan that ESXi sucks a lot of RAM up (although he was able to get it down to 3GB). Excuse again my ignorance, but, besides processor, when streaming a 1080 video to a 802.11n device, I need a large ammount of RAM right?

[+BudMan MVC]

NOT esxi -- vserver appliance.. NOT esxi itself.. Only if you plan on playing with the vcenter server appliance does it like to use up a bit of ram.. Which you can tweak down yes. If you plan on going with 5 or 5.1 you wouldn't never need the vcenter anything. If 5.5 and you upgrade your vms to 10 is the only time you would need vcenter or workstation to CHANGE their settings, ie add a nic or something.. If you create the machine how you like it - and then upgrade it to 10 you won't need vcenter unless you need to "change" something on it. You can use the client to access console, etc.

As to the HCL -- yeah mine was a upgrade.. I had heard 5.5 stripped some equipment, but its working fine on my N40L via an upgrade and my added nics, etc.

You only need large ram if you plan on running more than a couple of vms. I currently run 4 of them 24/7/365 -- if your only running 2 where pfsense only needs 512/1G unless you plan on playing with snort and squid, etcl

If all you plan on doing with it is pfsense and another 512 box.. I highly doubt 2012 performs well with only 512 ;)

Taking them to 8GB is fairly cheap, taking them to 16 cost a bit more. You do not need to run ecc, I sure am not in my n40l.. Maybe that changed with the gen8? You would only really need that if machine doesn't run unless it is.. Or this was critical box, not a home setup.

[+John Teacake MVC]

Yeah my HP Microserver works just fine running 5.5 I havent updated the VM's though. It seems to all work OK for what I need. 

[riahc3]

Hello,

  On 20/01/2014 at 14:55, BudMan said:

If 5.5 and you upgrade your vms to 10 is the only time you would need vcenter or workstation to CHANGE their settings, ie add a nic or something..

I had that doubt; Do I need vcenter OR could all modifications (such as add a nic, another HDD, passthrough, etc) be done with VMWare Workstation 10? Im used to Workstation 10 so I can get by with that :)

 

  On 20/01/2014 at 14:55, BudMan said:

Only if you plan on playing with the vcenter server appliance does it like to use up a bit of ram

Im missing something because the vCenter Server I install on a client machine (regular laptop running regular Windows for example) so why would that take up RAM on a barebone ESXi server???

 

  On 20/01/2014 at 14:55, BudMan said:

You can use the client to access console, etc.

Accessing the console is basically accessing a unix shell directly on the ESXi server right?

 

  On 20/01/2014 at 14:55, BudMan said:

As to the HCL -- yeah mine was a upgrade.. I had heard 5.5 stripped some equipment, but its working fine on my N40L via an upgrade and my added nics, etc.

Ive read sources that 5.5 has dropped a lot of old hardware support.......old meaning 2-3 years. Yeah, in technology that's pretty old but imagine if a regular OS did it...

 

  On 20/01/2014 at 14:55, BudMan said:

You only need large ram if you plan on running more than a couple of vms. I currently run 4 of them 24/7/365 -- if your only running 2 where pfsense only needs 512/1G unless you plan on playing with snort and squid, etcl

I only plan 2 right now 24/7/365 (cant think of another interesting VM to run always)

I don't think Ill be interested in major packages (always had a thing for Asterisk though).

 

  On 20/01/2014 at 14:55, BudMan said:

If all you plan on doing with it is pfsense and another 512 box.. I highly doubt 2012 performs well with only 512 ;)

I know :p I was gonna put 4GB on it as that's what my Windows 7 box has.

 

  On 20/01/2014 at 14:55, BudMan said:

Taking them to 8GB is fairly cheap, taking them to 16 cost a bit more. You do not need to run ecc, I sure am not in my n40l.. Maybe that changed with the gen8? You would only really need that if machine doesn't run unless it is.. Or this was critical box, not a home setup.

From Neowin's review, they look like standard non ECC RAM sticks....which I imagine ANY RAM can be used that is non HP.

Worst that could happen is that I see 8GB isn't enough and just get another stick I guess....

Also I thought it was pretty important but since no one has mentioned it, maybe its something stupid and pointless: the processor of the microserver doenst support VT-d ....

[riahc3]

Hello,

Well this caught me off guard; Installing VMWare vCenter, I can't install one of the main prerequisites which is vCenter Single Sign-On because it seems it requires a 64-bit operating system.

[+BudMan MVC]

You would only need workstation if you upgraded the vm hardware to 10, if 8 or 9 or 7, etc. then the client works just fine. No its not console based. See the screenshot showing the pfsense vm with only 512. That is the Vclient. The vcenter appliance.. Looks for 8 gig on install, you can change it.. And then tweak it to actually run.. for example I have mine with 3GB currently..

Here is guide on lowering its ram requirements

http://www.virtuallyghetto.com/2013/08/quick-tip-minimum-amount-of-memory-to.html

here is link to how to install the appliance

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007619

As to 64 bit..

What were you going to install vcenter on - thought you had

Windows Server 2012 R2 (my domain controller)

2012 does not come in 32bit versions..

As to VT-d -- what hardware do you want your VMs to directly connect too? In a home lab why would you need this.. So what if your vm sees virtual nic vs direct access to the physical nic. You can give your vm direct access to disk with a raw map.. It doesn't need direct access to the controller the disks are on.

I have connected wifi nics and camera to vms all via just usb passthru.. Its a home/lab setup not sure why you would need direct I/O access to specific hardware.

That being said I thought one of the features the gen8 models brought was VT-d?

[riahc3]

Hello,

  On 20/01/2014 at 21:26, BudMan said:

You would only need workstation if you upgraded the vm hardware to 10, if 8 or 9 or 7, etc. then the client works just fine. No its not console based. See the screenshot showing the pfsense vm with only 512. That is the Vclient. The vcenter appliance.. Looks for 8 gig on install, you can change it.. And then tweak it to actually run.. for example I have mine with 3GB currently..

I understood console as a actual terminal console....sorry.

  On 20/01/2014 at 21:26, BudMan said:

Here is guide on lowering its ram requirements

http://www.virtuallyghetto.com/2013/08/quick-tip-minimum-amount-of-memory-to.html

here is link to how to install the appliance

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007619

Ill refer to these later. Its obvious I am VERY lost sadly :(

  On 20/01/2014 at 21:26, BudMan said:

As to 64 bit..

What were you going to install vcenter on - thought you had

Windows Server 2012 R2 (my domain controller)

I was installing the VMWare vSphere portion of this entire thing:

It fails as soon as I try vCenter Single Sign-On, saying I need 64-bit. I believe like you mentioned that this is only necessary to MODIFY VM 10s I create, right?

  On 20/01/2014 at 21:26, BudMan said:

As to VT-d -- what hardware do you want your VMs to directly connect too? In a home lab why would you need this.. So what if your vm sees virtual nic vs direct access to the physical nic. You can give your vm direct access to disk with a raw map.. It doesn't need direct access to the controller the disks are on.

Well, now that you mention NICs, wouldn't Backtrack need direct access to a 802.11g USB adapter? Im doubting that right now so sorry.....

  On 20/01/2014 at 21:26, BudMan said:

I have connected wifi nics and camera to vms all via just usb passthru.. Its a home/lab setup not sure why you would need direct I/O access to specific hardware.

Does performance suffer a lot?

  On 20/01/2014 at 21:26, BudMan said:

That being said I thought one of the features the gen8 models brought was VT-d?

Its a processor limitation, not a chipset one :)

[riahc3]

Hello,

I have another question:

Like I mentioned, I have 4 3TB disks. Obviously I need to install WS2012R2 and pfSense somewhere.

How exactly does this work out? Would a 5th 320GB disk be OK to install this?

[+Fahim S. MVC]

The Gen8 MicroServer doesn't have a place you can put a 5th disc unless you use a 2.5" disc (spinning or SSD) in the optical bay.

The Gen7 MicroServer has room for an extra 2 3.5" discs or up to 8 2.5" discs (you need a controller card for them to plug into).

[+BudMan MVC]

Use one of the disks as your datastore. Or sure install a 5th drive in the optical bay area, it comes with a 250 doesn't it?

As to your scratched out backtrack? I have run backtrack on mine and just used usb wifi to it.. Works fine. Performance suffer to a USB device?? No video still works, It still talk at wifi N speeds.. ;)

As to that installer - again why are you installing vcenter? Install ESXI on somethig, boot it from a usb if you want, install the vclient on your workstation. YOU HAVE NO USE/NEED OF VCENTER!!! It is NOT FREE!!

Once you have played with esxi and got some vms running, if you want to play with vcenter then -- go for it.

[riahc3]

Hello,

  On 21/01/2014 at 12:12, BudMan said:

Use one of the disks as your datastore. Or sure install a 5th drive in the optical bay area, it comes with a 250 doesn't it?

From HP directly, it brings a 1TB drive. I guess I would have to buy a 2.5" drive...

Because storing the OS(s) on one of the disks that stores all my data? I dont know...dont feel comfy doing that :)

  On 21/01/2014 at 12:12, BudMan said:

As to your scratched out backtrack? I have run backtrack on mine and just used usb wifi to it.. Works fine. Performance suffer to a USB device?? No video still works, It still talk at wifi N speeds.. ;)

I have no idea what I was thinking there; I posted it, then scratched it out because I remember running BT and using a wifi adapter perfectly on my E6400...

  On 21/01/2014 at 12:12, BudMan said:

As to that installer - again why are you installing vcenter? Install ESXI on somethig, boot it from a usb if you want, install the vclient on your workstation. YOU HAVE NO USE/NEED OF VCENTER!!! It is NOT FREE!!

I have ESXi installed in a VM on Workstation on a 32bit Linux remote host. (mouthful) so thats why I wanted to install vCenter :) to play with it and test it out.

  On 21/01/2014 at 12:12, BudMan said:

Once you have played with esxi and got some vms running, if you want to play with vcenter then -- go for it.

On that remote host, Im gonna try to install Windows 64 b its and see if I can play with it...

[+BudMan MVC]

"Because storing the OS(s) on one of the disks that stores all my data? I dont know...dont feel comfy doing that"

Ok then don't -- You remember that your losing a disk to parity in your current raid setup.. So you really only need 3 disks to keep your 9TB of storage.. That if I recall correctly your not anywhere near filling up in the first place..

So what I would do is install install esxi and datastore to the 1TB disk. Use 3 of your disks in the other bays, raw map them to whatever vm/os your going to be running drivepool on there you go. Keep the other 3TB disk as spare, add it via the esata/usb port if you want.. But not sure of the need of it if your not anywhere close to 9TB of space anyway. Sell it to buy other toys, beer, etc... that you would have more use for than a disk that has nothing on it ;)

[riahc3]

Hello,

  On 21/01/2014 at 12:44, riahc3 said:

I have ESXi installed in a VM on Workstation on a 32bit Linux remote host. (mouthful) so thats why I wanted to install vCenter :) to play with it and test it out.

On that remote host, Im gonna try to install Windows 64 b its and see if I can play with it...

OK, I have a VM with ESXi.....made another VM with Windows 7 64bit and when installing vCenter on it, it gives a error that it cannot continue (Single Sign-On) but it doesnt specify anything else. The VM has 4GB...

[riahc3]

Hello,

  On 21/01/2014 at 12:54, BudMan said:

Ok then don't -- You remember that your losing a disk to parity in your current raid setup.. So you really only need 3 disks to keep your 9TB of storage.. That if I recall correctly your not anywhere near filling up in the first place..

...and now it starts to sink in...

  On 21/01/2014 at 12:54, BudMan said:

So what I would do is install install esxi and datastore to the 1TB disk.

Sorry, lost you: what exactly is the datastore? Is that a DB that keeps track of storage?

  On 21/01/2014 at 12:54, BudMan said:

Use 3 of your disks in the other bays, raw map them to whatever vm/os your going to be running drivepool on there you go.

Those 3 disks I would just connect to the W2012R2 which would run DrivePool and from WS2012R2 I would share, make maps to each domain user, etc.

  On 21/01/2014 at 12:54, BudMan said:

Keep the other 3TB disk as spare, add it via the esata/usb port if you want.. But not sure of the need of it if your not anywhere close to 9TB of space anyway. Sell it to buy other toys, beer, etc... that you would have more use for than a disk that has nothing on it ;)

I could keep it as a spare if S.M.A.R.T detects a drive is failing or something....

Thank for you for the idea; I dont know how I didnt see something so obvious in front of my nose...

[+Fahim S. MVC]

  On 21/01/2014 at 15:23, riahc3 said:

Sorry, lost you: what exactly is the datastore? Is that a DB that keeps track of storage?

 

http://pubs.vmware.com/vsphere-4-esxi-installable-vcenter/index.jsp?topic=/com.vmware.vsphere.esxi_server_config.doc_41/esx_server_config/introduction_to_storage/c_datastores.html

[+BudMan MVC]

  On 21/01/2014 at 15:06, riahc3 said:

Hello,

made another VM with Windows 7 64bit and when installing vCenter on it, it gives a error that it cannot continue (Single Sign-On) but it doesnt specify anything else. The VM has 4GB...

Yeah I don't believe windows 7 is a supported OS for vcenter..

http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software&testConfig=17

I see XP for the OLD versions, and then only server OSes.. So if your using 5.x your going to need a server OS from the below list, or use the vcenter appliance I linked too

[riahc3]

Hello,

  On 21/01/2014 at 15:33, Fahim S. said:

http://pubs.vmware.com/vsphere-4-esxi-installable-vcenter/index.jsp?topic=/com.vmware.vsphere.esxi_server_config.doc_41/esx_server_config/introduction_to_storage/c_datastores.html

Thank you for the link and sorry to sound like a idiot but it still isnt quite clear what it is to me....just seems like a file which represents a file system on a (which) HDD.

 

 

  On 21/01/2014 at 16:12, BudMan said:

Yeah I don't believe windows 7 is a supported OS for vcenter..

I see XP for the OLD versions, and then only server OSes.. So if your using 5.x your going to need a server OS from the below list, or use the vcenter appliance I linked too

OK, that blows...The vcenter appliances you linked seems to me like a standard VM.

As for the http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007619 Im gonna try that tommorow and see if it works.

Thanks for all the patience as I imagine I am annoying some members with all the questions and doubts.

[+Fahim S. MVC]

  On 21/01/2014 at 17:18, riahc3 said:

Hello,

Thank you for the link and sorry to sound like a idiot but it still isnt quite clear what it is to me....just seems like a file which represents a file system on a (which) HDD.

 

 

OK, that blows...The vcenter appliances you linked seems to me like a standard VM.

As for the http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007619 Im gonna try that tommorow and see if it works.

Thanks for all the patience as I imagine I am annoying some members with all the questions and doubts.

 

The datastore is the filestore where you store your VMs, and their associated Virtual Disks in a typical ESXi installation - think of each virtual disk as being the same of the VHD concept by Microsoft.  ESXi can have as many file stores as it needs, but as far as I understand you can only put one on each physical disk.

 

Now there is something called raw device mapping (RDM) too, which essentially presents a redirect from a virtual disk to a physical disc so the file system isn't stored as a file but uses the disk directly.  The physical disk appears as a virtual disk and is sized acccordingly but it is really just a redirect.  Probably worth checking out the following link as to how to set one of these through some commands on ESXi:

http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/

 

You need to understand this before you build a file server based on an ESXi VM - it's really very important.

[+BudMan MVC]

Standard VM? Its a pre-configured VM, to work as a vcenter. Yes it runs on SUSE, but vmware has customized it to be a vcenter VM.. You get this via a console to the vcenter vm. No not console like ssh session, when I say console in this context, I mean like you were standing in front of the real machine and looking at the monitor connected to it ;)

If you go to that url it gives you, you get a gui to admin the appliance, services that are running, SSO setup, etc. etc..

You can then hit the web client, to admin your vmware esxi host(s) and the vms on each host, etc. etc.

But again in a HOME setup you have little need of vcenter anything.. You can just install esxi, and then admin it from the fat vclient that you can run on windows 7

example

Only problem is - see my newstorage box, I had upgraded it to version 10, and now can not click the edit settings in the fat vclient.. Now you edit with vmware workstation if you have it, or you can run 60 day trial on vcenter appliance or installed on supported OS.

But there really is little reason to upgrade your vms to 10, I had done it on that vm before I realized I would no longer be able to use the vclient, and have to use web client that only runs on vcenter (which is NOT free).

I will most likely edit this back down to <than version 9, or will at somepoint just redo it and not upgrade it, etc etc.

But in HOME setup you have really no use of vcenter.. And if you just install 5 or 5.1 you will be able to just use the fat vclient to edit vm settings like add hdd, nic, usb devices, etc.

As to your datastore. No the datastore is not just a file - It can be quite a few things.

--

An ESXi system uses datastores to store all files associated with its virtual machines. After you create datastores, you can manage them by performing a number of tasks.

A datastore is a logical storage unit that can use disk space on one physical device, one disk partition, or span several physical devices. The datastore can exist on different types of physical devices, including SCSI, iSCSI, Fibre Channel SAN, or NFS.

--

So you see it on my esxi host here

You can host your datastore(s) wherever you want, does not even have to be locally connected disk to your esxi host. It could be isci target served from something else, it could be a nfs share of another server in your network. Its the place where you store whatever you want, yes your vms, but also any sort of files you might need to mount to vms, like ISOs, etc.