Project: Network Rebuild

[Stavros]

Part 1 - Site Survey


So it has been a while since I have done any sort of networking upgrades. Last upgrade I had upgraded from a Buffalo WZR-HP-G300NH to an ASUS RT-AC66U. I had flashed RMerlin firmware and it had been a pretty reliable piece of hardware. Recently (in the last year or so) I've gotten a Synology NAS and built a Proxmox VM server. However to enable port bonding on the ASUS I would have had to do some under the hood work that I wasn't comfortable doing.

In the mean time I got married and moved into the wife's house in which she had an atrocious 5 Mbps connection and was paying $40 a month! Well, Comcast is now offering Gigabit for $80 a month and of course I'm upgrading. So I'm adding a whole bunch of shit to Wifi and I want to have as many things using a hard line as possible. A few problems arise. First, the Coax connection is on the other side of the house as the office. Lindsey tells me it starts on the same side of the house the office is on but is run in the attic to the carport then outside down the carport and into a hole in the wall. I talked to a friend at work and at the end of this month or start of October we're going to relocate the Coax to the office.

Current Home Network


I bought a wireless AC adapter a day after upgrading and the next day talked to the friend who is going to be helping me with the coax and he said he had a ton of free wifi cards so I took it back. I kind of wish I'd not, but I am saving myself about $50 bucks which right after coming back from a honeymoon I am glad to be saving money. It's a shitty Netgear WNA1100. It is a Wireless N dongle, but it doesn't have the dual band usage and beamforming capabilities. It's basically a Wireless G dongle in all but name. Shitty but I won't have to deal with it for long.

Speed test from Fast.com on laptop hardwired into router


Speedtest from Speedtest.net on laptop hardwired into router


Speed test from Fast.com on desktop over wifi


Speed test from Speedtest.net on desktop over wifi


Comcast was not my first preference. I really wanted the local cellphone provider, Cspire, to be my ISP as they provide symmetrical gigabit. Alas it was not to be. If you recall last decade WorldCom going bankrupt, well when that happened Cspire (then known as Cellular South) bought fiber for pennies on the dollar and has been laying dark fiber up and down Mississippi for the last 20 years. Speaking of WorldCom, the WorldCom building is where I used to work. It has since been bought by Hertz Investment Group (and then sold to Duckworth Realty). It houses the Mississippi Department of Revenue, UMMC (University of Mississippi Medical Center) billing department, some Verizon department and probably a couple others I'm forgetting.

[Stavros]

Part 2 - Planning and pfSense Build

I have been planning this network upgrade for the better part of a year. So my new network had to have multiple capabilities:

• VLANs
• IDS/IPS
• Firewall
• VPN
• More reliable wifi

I originally looked at going completely Ubiquiti. It would be expensive but capable. But as I looked more into the Ubiquiti Gateway the problems with it, especially IDS/IPS at Gigabit speed, unfortunately ruled it out. So I began looking for an alternative. The alternatives, that don't require ridiculously expensive licenses was pfSense. I looked at their hardware offerings from the company that makes pfSense (NetGate) and trying to fit my goals I would have had to go with either an SG-5100 ($799) or XG-7100 ($899) I ended up coming across a post by JDM_WAAAT at ServerBuilders.net for a Mini-ITX pfSense box. I also found a user that made a pfSense box with a 1U rackmountable case This intrigued me as I had bought a 6U Network Switch Rack from someone on Reddit. I also cross referenced with Reddit's /r/HomeServer. I knew that for just routing and Firewall virtually any CPU would do, but trying to do IDS/IPS and VPN and try to do that at speeds that won't kneecap the bandwidth and was told that I would have to step up to a quadcore Xeon.

So I ended up designing the following pfSense box:

• Intel Xeon E3-1260L 45W TDP LGA155 - $36 Ebay
• Silverstone Tek Super Slim Profile CPU Cooler - 21.99 Amazon
• Intel DQ77KB LGA 1155 Mini-ITX Motherboard - $45 Ebay
• 8GB 2x4GB DDR3 PC3-10600 1333MHz 204PIN SODIMM - $19.99 Ebay
• TCSUNBOW MSATA Mini PCIe 120GB - $25.99 Amazon
• Genuine E1G44HTBLK INTEL I340-T4 QUAD PORT NIC - $44.99 Ebay
• T POWER UL Listed (19v 65w-90w) Ac Adapter - $19.99 Amazon
• iStarUSA D Value D-118V2-ITX 1U Rackmount Mini-ITX Server Chassis - $51.99 Ebay
• Noctua NF-A4x10 PWM (2x) - $27.90 Amazon
• Electop 4 Pack Case Fan Y Splitter, PWM Cable 4 Pin 1 to 2 Converter (4 pack) - $8.99 Amazon

Grand Total $302.80

So I've come in under the price of NetGate with double the processing power. Downside is the Intel Atom C3558 only uses 16W max whereas Xeon is 45W max.

That covers the Router/Firewall portionof the network. So for Managed Switches I had more options. Ubiquiti, Mikrotik, Meraki, Cisco, Dell, HP, etc. The choices endless (and expensive). I quickly narrowed it down to Ubiquiti and Mikrotik. Mikrotik was a little more hands-on and professional grade than Ubiquiti's UniFi lineup. However it doesn't have the nice unified interface. Mikrotik had more available through CLI, but Ubiquiti had better compatibility with Ubiquiti APs.

Ultimately I ended up going with Ubiquiti. Current parts as follows:

• pfSense Build - $308
• Ubiquiti US-16-150W - $280
• UAP-NanoHD - 158.5 (x2)
• Tripp-Lite 6U Network Switch Rack - $80
• UniFi Cloud Key Gen 2 Plus - $200

Roughly $1200, though the friend helping me relocate the coax is getting me the Cloud Key Gen 2 Plus as a wedding gift.

As of now I have all parts except the case which should be arriving tomorrow. I'll document the build in the next update.

[bad_brain]

nice posts and even nicer build stav! also good to see you are going 19" hardware, that's where the fun is...

I briefly experimented with pfsense communitsy edition (been a while), but got distracted with Solaris then....but yeah, when you're used to Linux I found working with BSD distros always a little awkward. awkward but still interesting.

about your internet connection / choice of provider:
bandwidth is overrated... seriously, I have 400Mb/down and 100Mb/up, pretty much the fastest you can get here at the moment, but everything over 100Mb/down is pretty pointless. ok, a torrent here and a steam download there where it's nice, but in the professional world you have to deal with having 100Mb/s hardware somewhere in the path in 99% of the cases still.
what I find more important nowadays is:
- any ports blocked? ISPs often block port 80 for example.
- IPv4 address available? my ISP is doing 100% IPv6, which is a severe pain in the rear to access from the outside.

what are you plans with the UniFi Cloud Key?

[Stavros]

A couple setbacks and a minor deal:

• CPU Fan died. Spun a couple times and quit. Played around with BIOS settings but it never spun again. So I ended up buying this monstrosity. Arrives Thursday.
• 4 Port NIC doesn't fit even if I take the bracket off. If I could slightly move it to the right it would fit without the bracket. Thought I might be able to scrounge a full height bracket from work. Ended up buying a two pack of full height brackets. I guess that's what I get for trying to piece this together over ebay.
• Found a deal on Reddit's /r/homelabsales for a Cloud Key Gen 2 Plus. Ubiquiti gear rarely goes on sale and if it does not much better if at all than the price I found at $170. Should be in on Friday.

I know bandwidth is over rated, but I would like the ability to host servers. I have a proxmox VM server I built like 5 or 6 years ago that has been doing nothing. It's a tower case, so nothing rackmountable yet. Not that I might not try.

I still have a seedbox from when I was on AT&T. Payed a year for a symmetric Gig with 1TB storeage. I'd consider doing it here if I had a symmetric line. I do like the plausible deniability of having it hosted elsewhere though.

As far as ports blocked I haven't run into any. It's comcast though so who knows. I almost went with a business line, but a couple things stopped me. I didn't know how long we're going to be in this house. It's built in the 60s and the AC is older than we are and has been struggling to keep it 78 on hot days. We're looking into replacing it and the ductwork. Jackson isn't a great city (an understatement). All the suburbs outside Jackson are ok, so we're looking at moving in a few years. Since I'd have to get a 2 year agreement and finding out that to cancel I'd have to pay the remainder of the contract I just decided to get residential. Also not sure if we're going to be moving to another suburb or out to the country. Out in the country my options become extremely limited. Likely to either Comcast or AT&T or satellite internet. I'd like to live in the country and own land so I could set up my own gun range, but that might have to be when I'm retired. To date I know of no one using IPv6. I mean I could check my modem to see if it's issued an IPv6 address, but I don't think it is.

For the Cloud Key: Lindsey has a Google Home Security system (which I don't care for). She was considering getting the cameras to go with the Google home. Me, being the nerd that I am, would prefer something better like Hikvision or even Ubiquiti cameras. I want something that wont' call home. I mean I guess I could set up a VLAN and disallow contact to internet and achieve the same thing, but I've kind of been trying to move away from Google. At any rate I'd like to leave it as an option.

Also you need a Cloud Key (or set up a VM to host the Unifi controller) to run the single pane of glass that can control the switches, gateway and APs (if you run a full ubiquiti stack). Since I want to run switches and APs and try to have two APs and have AP handoff (which you really need the controller for) then it makes sense to have the cloud key on a POE switch that will run less Watts than my VM server. And I want to keep the VM server for something else. Not sure what yet.

On an unrelated note, I have dreams of building a house and designing a server room. Brother-in-law is an architect and he said he'd design a house for us. I'm envisioning at least one 42U rack. I also want to build a Datahoarder NAS on a Supermicro 24 bay 4U server chasis. Totally overkill and totally not necessary but a man can dream. If WD comes out with 15 and 20TB drives next year like they say I might be able to snag 10TB Reds for cheap. The Synology has 2 4TB Reds and 2 white labels from shucked WD Easystores. I like the reds better they don't randomly click. Soft clicking and nothing that would bother me, but might bother the wife.

[Stavros]

Good news and bad news.

Good news: The old CPU fan was the problem and replaced with the new expensive Thermaltake spinning heatsink. The Bracket for the Intel card fits perfectly.

Bad News: Spot the problem.


I should have bought a PCI-E 4x ribbon to begin with. Oh well. Yet more waiting before I can finalize build and plug up and play with. Will be in Monday. Yay Amazon Prime.

[bad_brain]

IPv6 bullshit in all its glory...

I have found no way yet to make my home server available to the public, at least not without commercial 3rd party service.

comcast is blocking port 25, at least I had a few customers not being able to send mails through my MX (so they use 587 instead).....buuut, now that I mention it, it also could have been bellsouth (which I hate with a passion).

there is no such thing as "overkill" when it comes to servers....
I remember when I got my hands on the first piece of 19 inch hardware, it was a used Dell Poweredge, weight was about a ton or so, I barely could carry it alone.
turned out it was impossible to control the fans (2 banks with 6 or 8 of them) with Linux, so they always ran on 100%....phone calls were pretty much impossible...


and lol about the last pic....I assume the case simply wasn't made for such a board config... but luckily the space is big enough for a ribbon, a little less and it would have become complicated.

you made me drool a little about your plans with the house....please tell me there will be a 2nd basement floor with a diesel generator...

[Stavros]

I have run into yet another problem. I didn't do due diligence with regard to CPU cooler. The Thermaltake Engine 17 is rated for a 35W TDP CPU . My CPU in the build (Xeon E3-1260L) is a 45W CPU. It could keep it about 70C but the TCASE for the E3-1260L is 58.6°C. So, after doing a little more research I have a few choices (unfortunately few). The Thermaltake Engine 27, Cryorig C7 or Silver Stone NT07-115X. I already got the Silver Stone and it stopped working. After going back to the case manufacturer site and seeing that the height is 44mm well that rules out the Cryorig. I think I'm going with the SilverStone, I said it was the same model as the last one but it wasn't. The Thermaltake has an extremely annoying high pitch whine at high RPM. At least I got free Kohls return. Will run that and the silicon wedding rings I got back over lunch. Funny enough the Engine 27 is like $5 cheaper so I'll end up saving myself enough to go to Wendy's for lunch.

The case is made for mini-ITX boards which the DQ77KB is. I just don't think the PCI card was made for 1U cases and was probably meant or 2U or larger cases. Either that or I didn't get the right riser card for the case. Oh well, the ribbon works like a charm.

I have no idea about house plans. Just dreams at the moment. The server room will probably just be a room/closet with an extra AC attached to it and proper cooling vents. That is if we do indeed build on her family land (which is another story). No basements in Mississippi Yazoo clay is a bitch (you might know it as Montmorillonite). It's all across mid Mississippi. Lots of houses with foundation problems because of that.

On a side note I'll show you a couple examples of the insanity that is Jackson, Mississippi and why we won't be sticking around and probably moving to the suburbs (which is a bit of a misnomer as the suburbs are more like auxiliary towns).

Exhibit A: " onclick="window.open(this.href);return false;

Exhibit B: https://kingfish1935.blogspot.com/2019/ ... ckson.html" onclick="window.open(this.href);return false;

Exhibit C: https://kingfish1935.blogspot.com/2019/ ... f-day.html" onclick="window.open(this.href);return false;

[bad_brain]

eww, thermaltake fans are awful, I had one too and it also made that annoying noise you described.
you might want to look into aftermarket fans, I can highly recommend Noctua ones...they are a little ugly but fantastic when it comes to performance and noise level. they offer whole coolers for servers too, but the smallest one needs 3U.
https://noctua.at/en/products/fan" onclick="window.open(this.href);return false;


I seriously just spent time to check if it's the Jackson in the Johnny Cash song... but obviously it's another one in Tennessee.
I moved too a while ago, the city where I lived went downhill fast, much better in the small town where I live now....I live like round the corner of the market place shown on the pic:
https://en.wikipedia.org/wiki/Besigheim" onclick="window.open(this.href);return false;

[Stavros]

I have come full circle and I am still pissed off. I ended up doing what I should have done in the first place and just replaced the CPU fan. I got a Silverstone NT07-115X. This one works, but I've run into another problem. i'm not sure if it's a bad combination of CPU and Case (likely), but when the top of the 1U case is off it runs at an acceptable 45-47C. When I put the top on it spins up to 58-60C. The problem being the Xeon E3-1260 has a thermal max of 58.6C.

I did put some Arctic Silver 5 between the CPU and fan but it's not cutting it. And even then the old piece of shit AC unit can't keep the house cooler than 77F without starting to freeze up on really hot days (for reference it is supposed to be a high of 94F/34C today and 97F/36C tomorrow). Maybe we'll get relief after this week. Also that AC unit is going to have to get replaced as Freon is getting phased out next year. I'm hoping we'll get the unit replaced and not have to replace the duct work and we'll be able to enjoy keeping the house cooler. Will definitely help with electronics. House is usually kept at 82F/27.7C when not at home. I've been learning to sleep in warmer than i like.

So more troubleshooting. This build is going to be the death of me. I'm now wondering if just buying the pfSense hardware from Netgate wasn't the better choice. Well I guess I just have to salvage what I can and either get a lower power CPU or a 2U case.

[Stavros]

I have fixed my temperature problem. Cant believe I didn't think of it. Drilled a bunch of holes above the CPU cooler and taped off the back above the IO shield, holes around the NIC card and front port where the extra HDD bay in an attempt to get slight negative pressure so most air comes through above the CPU cooler.

Next up will be relocation of Coax. This won't happen until either the 19th or 26th. Going to shoot for the 19th though. Gotta get up in the attic and take a look at what I'm dealing with first. I'll add pictures of the case. It's pretty ugly right now. Gotta take a wire wheel or something and knock all the burrs off the case.

[bad_brain]

good you solved it! working with hardware, especially if it's not really designed for the purpose (like your mainboard in a 19" environment), can be a pain in the rear.
at least all the industrial servers I have seen work with passive CPU cooling, means there are only heatsinks mounted and a whole battery of fans press air through the case, like here:

[Stavros]

Huzzah Phase 2 is finally really complete!

The last thing I had to do was clean up the burrs left by my drilling. I realize I should have drilled into boards instead of using two two by fours and using them as makeshift sawhorses. Well lesson learned. I used a 1/2" chamfer bit to try and countersink the holes and knock the burrs off. It did somewhat, but I had too many left. So I got a dremel and grindstone and knocked them down. It's not perfect but I won't cut myself like I did a couple days ago. Thankfully only papercut deep.





Still looks a little rough but I think that's mostly ground off paint. It feels pretty smooth.


Speaking of passive cooling. Me and my friend were talking about servers and passive cooling and he showed me the internals of a 1U server they have lying around at work for testing and it looked a lot like what you showed. Of course with the datacenter fans that spin at much higher RPMs than the noctuas I have which are relatively low RPM fans.

If I had to do this over again I would probably pick the Akasa Galileo TU 1U case (fanless passive) and pair it with this one of the supported CPUs.

For part 3 I'm going to have a "Future Network Map". If work isn't busy tomorrow I might get it done.

[bad_brain]

sawing a square hole and mounting a standard fan guard with 4 screws would have also been an option...but I like your "it worked on the MIR space station too" approach...

but yeah, unlike professional server rooms/farms you of course have to deal with noise reduction, so you have t go unusual ways sometimes.....so: well done!

[Stavros]

Part 3 - Wiring and Initial Setup


So me and a friend I met back in Junior College and now work with at the Bank wired in the Ubiquiti UniFi NanoHD APs. That was on Saturday. I crawled around in the attic. My knees and abs still hurt. Since the roof is low pitched I had to basically lay across 4 beams to get in position (and not put a foot through the drywall ceiling) to drill through the stud and run the Coax and Cat5e to the APs. One thing I didn't think about was how close to the Breaker box I had planned to put the AP, so walking around in the attic was nerve wracking. Especially since the house is 60 years old. The drywall was a bitch to cut through. Hope that wasn't asbestos. We'll find out in about 30 years.

As far as power efficiency goes I'm running at about 115 Watts according to the Kill-A-Watt. That's running

• Synology DS918+ NAS
• pfSense Router
• Ubiquiti US-25-250W UniFi switch (clocking in at 14.5W)
• Ubiquiti UniFi Cloud Key Gen 2 Plus
• APC BN450M UPS

Here's photos of everything racked.


Living room AP


Hallway AP


Kill-A-Watt


And the wall connections.


All total it took us about 3 or 4 hours.

Next part will be on configuring. I'm going to have to play with it for a while. No idea when next update will be. Maybe I'll get the whole day after work and play with the network on Wednesday. This week I'll be going to the gym Tuesday and Thursday. Not sure if I'll go back on Friday or Saturday.

I should also mention I was able to get the total price out of pocket for this upgrade down to about $850. I asked my Mom for a NanoHD for my birthday. They went ahead and got me two one as a birthday present and one as an early christamas present.

OH last thing. I was designing the new network and I came up with this.


The problem is I'm not sure if I should combine the Trusted LAN with the Trusted WLAN. I don't particularly care that they talk to each other, but I'm also not sure if media should have it's own VLAN instead or if I should put the Synology NAS on the Server LAN (my origninal idea.

[bad_brain]

good work!

personally I generally label WLAN as not trustworthy, at least not as trustworthy as a wired network...we all know how relatively easy it is to crack even the latest wireless encryption methods.