HomeLab thread? Whitebox, PowerEdge, ESXi, Hyper-V...

Bad thing about going with non Xeons for VMs is the current 32GB ram limit. For what you want to do it's prolly fine though. Not sure how much storage you need but linked clones on SSD drives are so much faster than platter drives when mucking around in them.

CPU is probably ok, hard drives are awful though. Not sure why you are asking about vlans and CPU wattage?

HardForum has a much more active tech community with a whole subforum dedicated to virtualization:Virtualized Computing - [H]ard|Forum

I assume you are just storing all the VMs on the 5900 RPM bigfoot drives? I seem to think you're not going to get the performance you want out of two 5900 RPM spindles if you're going to be running more than a couple VMs, especially if you're planning to do have something like a domain controller, maybe a domain certificate authority, a file server, etc.

While of course not the only one, IOPS is a serious concern with VM infrastructure. I would buy more cheaper, smaller drives and put a nice big RAID in there to bump it up, assuming you don't want to float the cash for SSD or flash memory units.

Also, does that mobo have an SD slot? IIRC you can just get ESXi on a bootable SD.

SSDs/flash drives are fantastic for VMs, the practical issue is usually the cost of space. If you don't care about durability, you can ignore some of that.

It really depends what you want to do. If you are just leaving 3 2012R2 instances up doing nothing for posterity... then sure who cares if you can fit it all in RAM. When you start adding a domain controller or other databases into the mix, you can start getting some pretty high drive hits.

You could easily create the VMs on SSDs then attach VHDs on the bigfoot mag drives for simple storage for media (I'm assuming the reason for the NAS is that you want this whole enterprise to be fun(c)tional as well as utilitarian?) but the second you start putting applications or other data on that you are going to see a huge drag.

What I would do, personally, is find a Microcenter or Frys and order a bunch of refurb SSDs for $40-$60 and make a big raid array of them. That or grab a bunch of cheap spindles for $20-$40 to grab more space. You'll run into issues with case size and power supply going the HDD route though and the bus on a non-enterprise MOBO likely means you'll probably want a separate raid controller card. Both issues are fixed if you can afford to drop big enough SSDs into the equation.

The shitty thing about a separate NAS is as soon as you step outside the MOBO you are basically fucked without at least several duplexed 1Gb ports, or ideally a 10Gb card or iSCSI... and that just throws cost through the roof for something like this.

To really optimize your home system you'll want to connect to your PC via an Infiniband FDR connection @56Gb/s and then Fiber Channel out the back end to the storage (Stick with 8Gb/s for now, 16 is too expensive). To really saturate that IB connection though you'll want 8 FC connections to your disk enclosure.

Hi,
What you are describing above I built for ?1k two to three years ago with an SSD drive included (some small differences and a nice graphis card extra). I think if you looking around you might do better on price.

Definitely Palum is on the ball, with a similar system to what you describe I get performance on par with Palum's analysis. I also find disk I/O to be my main bottleneck. I'd like to go to 32GB RAM as an easy upgrade but another SSD would probably be best. Currently I use the SSD for swap and boot from SD card.
One thing I have found handy is a second nic (making sure it is on the HCL is the trick) for the management traffic and then using the inbuilt nic for all the VMs. It comes in handy when you do something silly on a VM and DOS yourself.

I have spoken with a few people about their setups and I feel that only those learning about storage speak highly of running some sort of NAS/SAN within the same host.
In terms of using an external cheap home NAS, I'd recommend against it. Fighting with performance issues on my first lab (host+NAS) would have turned me off vitualisation if I hadn't seen it running well elsewhere.

While it seems over the top Noodle's response is how you can end up feeling if you under spec the disks. :/

Obtenor_sl said:

Wow, very informative; They won't be doing a lot of resource intensive tasks at all; I do vulnerability research in my spare time and I usually try exploits and things like that, so I need some ESXi where I can spin up VMs of different OSes to attack. I use VMWorkstation right now but it takes on my main desktop space/resources/etc.

Having said that I wanted to keep it under 900$. I've seen some NAS solutions and then the 1GB cards and such, so freaking expensive. I guess if I go small form factor I'll have to decide what kind of card do I want in my only PCIe slot (nic? raid controller?). I can probably bump up the HD speed (7200 or 10k) and maybe get a raid controller card.

Click to expand...

Here's the thing, HDD RPM does not directly correlate to faster seek time and IOPS so that can be deceiving - sometimes companies pad that with average seek time due to huge caches and who the hell knows what else in their tests... Spindles are honestly only still used because they are cheap and plentiful and by time you get to the storage and data durability you want, you often times have so many spinning drives that you hit the IOPS you need incidentally.

For cost savings- just go get some refurb SSDs, get a nice MOBO with a good built in raid controller. About a year ago I picked up 3 stupidly fast refurb Corsair 120GB drives on sale for $60/ea at Frys. I run one drive with my OS on it, 2 of them in RAID 0 for 1GBps read/write and stupid IOPS for apps/games and then I have my 2TB mag drive for all my files. Something similar would probably work fine for you here and not bottleneck your system.

To give you an example, a 5900RPM consumer grade HDD will probably run like 50-70 IOPS. My Corsair Force GTs run 80K a piece... yea.

Those 5900 RPM drives will kill you first of all... My suggestion is to go 2012 R2 Hyper-V since you get all the bells and whistles that you'll get with ESX without having to pony up the extra cash. You'll also get the native iSCSI support and hyper-v replica, which will allow you to cheaply replicate your VM's to a totally different box without having to worry about shared storage or FC. In fact, you can even use the native file system on a remote server to technically hold your VM's if you want to. For a home setup where you don't have the luxury of an FC based SAN, Hyper-V is the better choice imho. Build your server with Hyper-V (seriously, why not just buy a 3 year old proliant off Ebay for peanuts?) along with an iSCSI based SSD NAS and you're golden.

EDIT: Oh yeah, if you base it on a 2012 R2 server, you can also turn dedupe on for your drives that hold the VM's. This will save you anywhere from 20-50% drive space depending on your data trends. At a minimum, you'll get at least 20% simply by virtue of deduplication of the OS itself.

ESXi is free but vCenter is not. You can control the host directly without having a vcenter server however.

Frenzied Wombat_sl said:

Those 5900 RPM drives will kill you first of all... My suggestion is to go 2012 R2 Hyper-V since you get all the bells and whistles that you'll get with ESX without having to pony up the extra cash. You'll also get the native iSCSI support and hyper-v replica, which will allow you to cheaply replicate your VM's to a totally different box without having to worry about shared storage or FC. In fact, you can even use the native file system on a remote server to technically hold your VM's if you want to. For a home setup where you don't have the luxury of an FC based SAN, Hyper-V is the better choice imho. Build your server with Hyper-V (seriously, why not just buy a 3 year old proliant off Ebay for peanuts?) along with an iSCSI based SSD NAS and you're golden.

EDIT: Oh yeah, if you base it on a 2012 R2 server, you can also turn dedupe on for your drives that hold the VM's. This will save you anywhere from 20-50% drive space depending on your data trends. At a minimum, you'll get at least 20% simply by virtue of deduplication of the OS itself.

Click to expand...

I assume he didn't want to go HyperV because you can't do anything but Windows VMs? The used gen 6/7 Proliant is a good idea though.

Palum_sl said:

I assume he didn't want to go HyperV because you can't do anything but Windows VMs? The used gen 6/7 Proliant is a good idea though.

Click to expand...

With 2012 R2 Red Hat, CentOS, Debian, and Oracle Linux are fully supported as guest VM's.

Frenzied Wombat_sl said:

With 2012 R2 Red Hat, CentOS, Debian, and Oracle Linux arej fully supported as guest VM's.

Click to expand...

I must have missed the memo on that one. Interesting, looks like Microsoft is playing for keeps with HyperV now.

I had a HP microserver gen8 with celeron 2.4, 8 gb ram, 2 x 2tb raid 0, 2 x 512gb raid 0, 60gb ssd, 16 gb stick. Was running esxi 5.5 with pfsense for 1gbps routing and debian with transmission & flexget & plex for my media needs.

Sold the server (kept the storage) and bought anhttp://www.advantech.com/products/1-...41953FF70.aspxthat just got here 2 days ago. Planning to buy an i5-2520m (vt-x, vt-d, vpro) + 16 gb ram for it and I already have an IBM M1015 in IT for passthrough.

Gonna miss the the ECC ram but this new server is gonna cost me less than a decent cpu for the Microserver.


edit : i got the motherboard from some guy on ebay for 50$ but he has an even better offer now :
Advantech Aimb 272 Main Board with i7 2710QE CPU and 16GB RAM Mini ITX | eBay

My mistake then about the RAM. I remembered that for awhile Intel's desktop LGA 1155, 1150, and 2011 chipsets were a max of 32GB - maybe something has changed?