I've got a few questions and would like some input and suggestions from anyone with experience.

I would like to build a home lab  where I can play with different server level 'stuff'.  Stuff like a NAS, personal cloud, web servers, game servers, a place to spin up VMs for whatever I feel like looking into, security camera network, I would also like to build my own router with PFSense, VLANs etc.  I've started messing with VM's on my main machine and it's peaked my interest in trying a bunch of different stuff.  I would prefer to not have all that running on my main machine.  I see a bunch of videos that are like 'Build a NAS on a Raspberry Pi' and I feel like that's more of a novelty than something people actually use day to day...but I could be wrong.  

My questions as of right now:

Should I look into used server grade hardware right off the bat?  I don't really expect/need 99.99999% uptime.  This is more for fun/learning than for highly valuable operations/data storage.  

Can I run all of these various 'tasks' on one physical box(depending on hardware obviously) or should I be looking into a specific appliance for the NAS itself and another one for the security cameras?  I feel like I could run those things on the same hardware in separate VM's...I feel like the Router should be it's own hardware though.  Not 100% sure on that either...

I have a few workstation level computers with old but quality hardware in them, I know I could just hook those up, but I've never really done any work on server grade hardware so I'm not sure what I would be missing.  



Please feel free to drop any knowledge or lessons learned from setting up your own home labs.  Background, I'm a c#/ASP.NET/JS/React/etc developer that's interested in getting into more networking/system admin/security stuff.

All that server-grade hardware sucks down a ton of power, and the more you use it to run cameras, IoT, etc. the more you're going to want to leave it on all the time.  

I used to have 2U and 4U boxes racked and now I have an 8-bay Synology, a Dell Precision Tower and a few OptiPlex Micros. It's plenty for what I need.

Pfsense CE is it's own OS you can't run it as a program under another OS. You need a dedicated box for that.

If you want to build your own NAS and run something like TrueNAS that again is it's own OS and you need a dedicated box. I would personally just go with Synology and call it a day.

As for other stuff as was said if you plan something like a rack mount server they do chew through power. They also are loud and put out heat so keep that mind, too.

As for Pfsense you can build your own box or buy something like a Dell SFF for ~$100-150 off Ebay put a two port ethernet card in it. I looked at the Netgate units but the 4200 that can handle gig fiber is $549. I ended up going with a Unifi Cloud Gateway Max which can do up to 1.5gbps with security features turned on. Unifi is whole ecosystem of products that all work together like Apple devices do. You can do basically most of what Pfsense does like VLAN's.

My home lab used to occupy a full size rack. I finally got sick of the heat and noise and my lab is now in a 12 unit rack and is fanless with the exception of a Lenovo ThinkCentre mini PC for VM use. Anything heavier than that and I will spin up a machine temporarily on DigitalOcean.

Having said that, if you really want a big server, take a look at the TechMikeNY listings on eBay. Something like: https://www.ebay.com/itm/326223088201

G9 DL360 with 28 cores and 256 GB of RAM for $530 (you will have to supply your own HDs for around $20 each). Runs ESXi 7 nicely. Probably ProxMox and Nutanix also.

Power consumption and heat would definitely be an issue.  I have a space I could theoretically put it with an exhaust fan which would keep the heat out of the rest of the house...but I'm not convinced it's worth it for the cost.  The hardware used is cheap, but paying the power bill is forever...

I would rather not go with a prebuilt box like a Synology for this use case.  Only because this is more about learning and I feel I will learn more from setting up a NAS from scratch vs buying a prebuilt solution.  But yes, from my research it looks like although TrueNAS can run in a virtualized environment, it can be a bit of a bear to configure and manage.  So I will probably dedicate one of my desktops to that. For the price of a Synology box I could have all the drives I would need and use the hardware I have.  

For the router I planned on buying something dedicated.  I like the idea of a SFF box.  I've been looking around and there are some passively cooled options that are very low power and noise...

That leaves me another entire box for everything else.  I'll toss a hypervisor on it like Proxmox and just set up a bunch of VM's for whatever I feel like doing.  


Appreciate the help.

I'm actually hoping that a rack mounted server isn't necessary.  I'm just not sure what I'm missing.  Obviously massive amounts of RAM, hot swap storage, huge power supplies, and fans that sound like jet engines...

I do think it would be 'cool' to have a rack in the back room...and it would be a nice place to put all my equipment, but I'm leaning towards just using consumer level hardware to host my home lab.  Unless someone  brings up some major points towards using server grade hardware.

I can't say about Pfsense CE, but you can run TrueNAS on a hypervisor that lets you pass disks or HBAs wholesale to a guest OS.  A coworker does this with Proxmox.  It does require a bunch of knowledge and work though.

I've gotten away from virtualizing key infrastructure (network/storage). Especially if you're in the learning stages of a new hypervisor like Prox that has a steep learning curve, you need certain things to be guaranteed always available. Before I switched to a FortiGate, I was using PFsense on a Dell SFF with a dual 10gb NIC, worked like a champ.

I understand your point there, as I run Redhat Core as a VM on Truenas to run Pihole.  Makes it annoying if something goes wrong.  But I did not have good luck with Piholes running on Rpi hardware being very resilient or long lasting and thus put up with it as a VM so I don't have to keep buying Pis.
 Mainly was just countering that it can in fact be done.

ETA:

You just listed most of the major points about using enterprise hardware.  Heat, Size, Noise, Power draw.  

Personally I'm also ok with some "prosumer" Unifi or similar network hardware, my NAS is just Truenas on my old I7 desktop in a case that holds a lot of hard drives.  I'd like to get the network gear in a mini wall mount rack just so it can be off a desk or floor but that requires I run a lot of ethernet and 2 circuits in my unfinished basement.  Not a priority for me right now.