Rebuilding the Network

Back in May of this year I revealed that I was in the process of planning out a network rebuild as a consequence of our home network aging out (the newest network infrastructure components were at least 10 years old).

You can read that post by hitting this link for the a Home Network Revival Storypost.

While we are not each of us likely to choose to admit to matters like hero-worship easily, I have to admit that it is the Zen-like awareness of the character Chris Stevens - and specifically the sub-character he is when being "Chris in the Morning" - a DJ for radio staiton KBHR - which he announces as Kay-Bear...  Proof that figuring out what you want and then making it happen by recycling as much as possible is just plain cool man...

Surveying the Reality of the Network Situation

When you begin a new journey it usually helps when you start at the beginning - and in our case, for the re-imagining of our network - the beginning starts where the wires attach to the house.

Since we previously ran our consulting business from offices in our basement that we had finished off with walls, doors, paneling, and a finished ceiling with proper lighting, the place where the wires come into the house was the most logical place to deploy our network racks.

So in that spot there was (past tense) a Relay Rack (a two-post 19” rack used exclusively for network hardware), besides which was two matching 7-foot-tall standard box-style 19” equipment racks of the sort that are commonly known as “Server Racks.”

The pair of Server Racks we chose back in the day were the then-current model of Tripp-Lite 42U rack enclosures. We bought two of them, with side-panels, but no doors (front or back), no wheels, and no handles. Just the standard leveler bolting scheme that allowed us to drive in concrete receivers so that we could bolt the racks to the floor.

For the Relay Rack we chose a Tripp-Lite SR-2Post-45U Open Frame Rack, with a solid steel 4-way-mount base plate that allowed it to be properly secured to the concrete floor so that it could safely contain up to 800lb of hardware (for which it is rated).

Even when it looks like it is still good, old network hardware is just that - old.

Age and Treachery Require Flexibility!

As we contemplated an entire re-design of our network in order to bring it, kicking and screaming, into the 21^st Century, we had to make some tough decisions. The main reason for many of them had more to do with the fact that between 1995 (when the original network was planned out) and 2015, I have lost the use of my legs and require a wheelchair to get around.

As a result of this in 2002 we pretty much moved my office to the ground floor (because the basement office was no longer accessible to me), with the room on the ground floor that was designated as my office being specially adapted.

Because I converted my old and no longer manageable career of NetSec Engineering into Writing, and in particular as more than half of my writing was on the video games beat, that room was given the well known and patented Ikea treatment in order to make it as friendly to my needs as possible.

What that translates to is simple enough. Against the front wall of the room we installed a custom-designed and modular solution based on the "Besta" line of shelving and storage units from Ikea. 

Basically that included a primary Besta Unit with a large open TV shelf off-center to the right, with a bank of shelves along its left side. We then added a pair of large shelf banks, one of which was laid down on its side to function as a base for the main unit (to which it was bolted) that basically provides five identical box-shaped storage openings that match the five similar openings that serve as the base of the main unit.

By doing that the main unit is raised in height of the unit so that when we put the second bank or column of storage shelves on the left-side of the assembled main unit, their heights matched.  What that resulted in was a main unit combination that could easily accept our 52” flat screen TV and plenty of storage. 

The base of the TV space has a row of two very wide but low shelves that are meant to accept entertainment kit like DVD players, game consoles, and other modular kit, while the eight box-shaped openings plus the added five additional shelves from the second column laid on its side provide full customization space.

The total number of storage openings is 18 and they are sized so that you can use modular storage objects from Ikea - such as their conversion kits with backs and doors to turn them into independent storage space, as well as drawer inserts and filing container inserts just to name a few.

We went with a selection of different inserts for the office so that I had plenty of file space, as well as proper shelves for books and game cases.

Right so you now know what the central focus of the office consists of. Positioned directly in front of the E-Center is a large flat bed-like construction upon which I can sit, lay, or recline, while I work, with my notebook PC and game controllers easily accessible.

To the left of that work space is a small desk on which the A/V rendering server is placed, with its keyboard and mouse accessible to use from the work space. Finally along the left-hand wall is a proper desk for my wife's use.

It was into this fully functional and well-designed work space that we inserted - to the right of the E-Center and thus occupying the right-front-corner of the room - one of the 45U Server Racks, into which would be installed all of the hardware that for various reasons I may need direct access to.

The New MBR Rack Space and LAG

We decided to call this the MBR Rack, so the E-Center became the MBR Center.  Into the MBR Rack was initially installed a pair of shelves, and some spacer vent plates. The latter being perforated plates that provide front-coverage to serve as spacers for the various hardware while allowing ventilation so that air-flow was ensured.

A 4U spacer plate was installed into the top of the MBR Rack, and directly under that we installed the first of a matched pair of Netgear GS724TV2 Managed Smart Gigabit Ethernet Switches that would serve as the backbone of our new network (bearing in mind that the original network was left in place and fully functional as we created the new one!).

So the pair of GS724TV2 Switches were installed - on in the MBR Rack, one in the NOC Relay Rack - and these were fully and properly configured. That meant adding the proper configuration settings, and then making two custom CAT6 Ethernet cables that connected Ports 23 and 24 on the MBR Switch to Ports 23 and 24 on the NOC Switch.

These two switches were then configured to use the pair of Cat 6 cables as a single 2GB backbone that connects the upstairs and downstairs switches - as otherwise a single Gigabit cable would be all that connected them.

All fully-managed and most partially-managed Gigabit Ethernet Switches today offer up a backbone protocol of this sort, either using dedicated Network LAG configurations (LAG = Link Aggregation) or, as was the case for our Switches, a dedicated aggregated Trunk connection of two cables and two ports for a total of 2GB backbone speed.

The reason that we chose the Netgear GS724TV2 switches was a combination of price (they can be had on eBay dirt cheap) and the decision on Netgear's part to opt for using Trunking in place of proper modern LAG.

Trunking is preferable largely because in theory - and if needed - we could use up to 8 Ports (for a total of 8GB) for our backbone aggregation if we chose to, whereas most of the modern implementations of LAG protocol limit it to just two Ports - or 2GB.

Because we work with A/V files on a regular basis, and because file storage is planned to be centralized on our network, we need a minimum of a 2GB backbone and possibly a minimum of 4GB - so the ability to expand that as needed? Yeah, that was key.

With the two switches and the backbone connection in place, we tested the data transfer speeds and have come to the conclusion that, as long as no more than three users are accessing A/V media or working with A/V files across the network simultaneously at any given time, the 2GB backbone should be sufficient.

But just in case we made a third Cat 6 cable and attached Port 22 to Port 22 after disabling Port 22 on both routers (to ensure that there is no accidental looping) so a third 1GB connection exists that can instantly be added to the existing 2GB backbone to increase it to 3GB if we ever need to. You know, having the raw cable there really was no good reason not to do that.

Proper Termination

The network downstairs was originally a free-flow cable configuration but, once that got too messy, we ended up adding a CAT5E Patch Panel to it ten years ago, and then custom cabling to Keystone boxes installed at the baseboards for a neater cable management scheme.

So all of the systems that connect to the NOC end of the backbone do so via properly terminated permanent CAT 5E cable runs that utilize a patch panel and patch cables for their end-point connections.

Since we were re-imagining our network anyway, and moving the Server Rack and a Switch upstairs into the office anyway, it seemed like a good idea to duplicate that standard upstairs. So we purchased an industry standard CAT 6 Patch Panel and properly terminated all of the cable runs that connect to it using the standard box-and-keystone configuration, so that, at strategic points in the walls, there are nice clean Ethernet jacks in place of spaghetti cable runs.

At the MBR Rack installed just below the Ethernet Switch is a 48-Port CAT6 Patch Panel that is almost identical to the 48-Port CAT5E Patch Panel in the NOC Relay Rack. So how is that for symmetry?

Power Management

The NOC Relay Rack was originally planned and deployed with a standard, grounded, 8-Port PDU that connected to a 2400KVA UPS installed in the base of the Server Rack next to it, which allowed for the various bits of kit in the Relay Rack to have their power independently controlled, using the toggle switches on the PDU.

Of course at the time I was AB and thus able to walk over to the rack and flick a switch - but once I was crippled, and with the hardware now completely inaccessible to me, if I needed a device attached to the rack down there reset, that meant someone else had to go downstairs and flick the switch off, then on again.

Clearly that was not an ideal circumstance - and with so much of the hardware failing as it aged out, I was having to send switch-flickers downstairs several times a day, EVERY day!

So when the Server Rack was moved upstairs, we detached the mechanical PDU from the Relay Rack and brought it upstairs with the Server Rack, installing it at what would ordinarily be waist-high level in the rack. All of the network kit was then powered via that mechanical PDU.

Downstairs in the Relay Rack things were different. Changed. Better!

Via eBay we purchased an APC model AP7900 Network-Attached 8-Port PDU complete with the same style of web-based admin access that the switches use, so that I could basically recycle the power to any connected device remotely.

Yep, that meant that even if I needed something attached to the NOC Relay Rack reset, I would not have to dispatch a human to do it - I could do it myself! And how cool is that?

The AP7900 is a pretty well-known high-quality piece of equipment. Purchased brand new from APC you are looking at $500 a crack - but used on eBay? Less than $90! So that is totally win-win and within the budget.

In fact I am seriously considering buying a second AP7900 for the MBR Rack and I would if that was not so decadent and lazy a decision :)

Servers and Such

The Server Rack downstairs is part of the solution for our re-imagined network, because it is the home to a new server we purchased and over the course of 4 months (as we could afford it) upgraded until it was ready to be deployed.

I am speaking of the Dell PowerEdge 2950 II server we call “Monster” that functions as the VM Host for our network.

Virtual Servers are a new concept for us here - and while it just might maybe be overkill when you stop to consider that the network we are building is being built to provide Internet Access to the residents of our farmhouse, and work resources for a journalist, it probably IS overkill.

But when we did the math, when you combine the average cost of ownership and the electric bill for running four generic PC-based servers which each have PSUs ranging from 450W to 900W (the file server uses a 900W PSU due to the number of drives in it) it actually makes total sense!

The 2950 II has a pair of redundant 720W PSUs in it, and collectively they end up using less than half of the juice that the four original servers that they replace use.

So our new (it is actually a used and refurbished) PowerEdge 2950 II which we purchased from a server rehab outfit in New York City called TechMikeNY who we originally found on eBay (but who has their own store-site online as well) we basically got very very lucky.

The process of finding reliable, reputable, honest, and knowledgeable suppliers for specialized computer hardware in this day and age is often something of a crap-shoot. And considering that we not only needed a reliable and savvy refurbisher but also a supplier who would be willing to answer all of our questions and provide solid and sound advice pretty much whenever we needed it... Well, that is almost an impossible task these days!

To make this absolutely clear, we have no reservations whatsoever in recommending TechMikeNY as a supplier whether via eBay OR via their website sales center. The blokes working there know their hardware inside and out, can (and more importantly WILL) answer any and every question that you ask - even the really stupid ones - and provide a high-quality refurbishing services for servers that they back up 100%.

The MIT Flea - held third Sunday of the month, April thru October, and the place where we obtain the used and refurbished hardware as well as new batteries for the UPS's

Adventures in Server Acquisition

We felt so comfortable - and happy - with the services that they provided when we originally bought the server as an eBay transaction (which is a good idea for you to use to since using eBay adds a layer of insurance to a purchase like that thanks to the eBay and PayPal policy of going to bat for the buyer whenever issues crop up for such transactions) that we had no problems or concerns with hopping from eBay to their web-storefront to do the rest of the purchasing.

At their well-made website we bought the rest of the kit we needed -- which in this case consisted of the following (with source noted):

MONSTER (Our VM Host Server Build)

PowerEdge 2950 II Initial Server Purchase @ eBay -- Total Cost: $250.74

(Buy-It-Now Price = $215.99 plus Standard Shipping = $34.75)

eBay Order and Purchase Number 221295119685 check it out for yourself!

This was basically a starter-package for a Commercial-Grade Enterprise-Level Virtual Server Host System that included the following bits and bobs:

• Model: PowerEdge 2950 III
• Processors: 2 x Intel 2.33GHz E5345 Quad Core
• Memory: 8 x 2GB PC2-5300, FB
• Hard Drives: 2 x 1TB 7.2K SATA 3.5"
• Drive Bays: 6 x hot-swap 3.5"
• Power Supplies: 2 x Dell 750 Watt (redundant PSUs)
• Number of Power Cords: x2
• RAID Controller: PERC 5i w/battery backup installed
• Network Interface: 2 x Broadcom Gigabit Ethernet
• Video Card: ATI Technologies Inc ES1000
• Optical Drive: DVDRW-CDROM+
• Remote Access Controller: DRAC5
• OS: Windows Server 2008 R2 Evaluation Edition
• Warranty: Standard 30 Day Included

Yes, all of that and only $250?! Not only is that a heck of a deal, it is an awesome server! The ting is it is just what we went looking for - a starter kit. We knew going in that we would need to buy additional bits and bobs before we could actually deploy this as our server solution.

So happy were we with the initial purchase that when we found out that TechMikeNY was in the process of building its own web-based storefront, we waited for them to complete it and roll it out rather than doing the rest of the purchases via eBay (because why? Because we could save a few bucks on eBay fees that is why!).

When their storefront finally deployed we bought pretty much most of the rest of the kit we needed, which in case you are contemplating doing the same thing we did, was:

• x3 1TB 7.2K SATA 3.5" @ $49 each;
• x3 HP Drive Caddies @ $15 each;
• x1 Dell PowerEdge Rapid Rails Kit @ $35;
• x1 Intel Pro Dell X3959 Dual Port Gigabit Ethernet NIC Card PCI-E D33682 Card @ $36.99

Basically we needed the additional drives in order to create a pair of RAID Volumes - the first one being a 1TB Volume for the OS that consisted of a pair of mirrored drives (plus a dedicated Hot Spare Drive) for the OS and VM Host Server configuration and the first virtual server, which is a network utilities server.

The second Volume was an identical pair of 1TB mirrored volume for the second pair of servers - our Wiki and Database Server and a Testbed Server for online game servers.

To make that work we needed to add another dual-port Gigabit Ethernet Card since each server would need its own dedicated Ethernet and so we needed four Ethernet Ports.

Finally there as the Rails - which you cannot install this beast into a rack without!

The only thing we still need (but there was not enough in the budget for at the time) was one more of the 1TB 7.2K SATA 3.5" drives and an HP Drive Caddies to install it as the designated Hot Spare for the second volume.

See the way that this works - because it is necessary for us to install the server in the NOC Rack far away from humans due to the incredible noise that Enterprise-Level servers tend to make thanks to their massive cooling requirements - trust me this Monster sounds like a 737 taking off when its fans kick in!

So yeah, since it has to be racked away from anywhere that humans are likely to be found, having pre-configured and deployed hot spares makes total sense! What a hot spare does is just what it sounds like: if something goes wrong with one of the two drives that make up the mirrored volume, the RAID controller immediately signals an audible alarm (that nobody will hear) and emails a warning alarm notice to the admin email address (which of course I am likely to see soon enough) while at the same time activating the Hot Spare and rebuilding the Array using it!

That way we never lose the mirror and insurance that it offers, and the server can take the necessary steps to ensure the continued operation of the volume. I get the email and dispatch one of the kids to pull the bad drive - the HP caddies actually have indicator lights on each that will show which drive has gone bad or crashed, but there is also a display panel on the front of the server that will tell them which slot the bad drive is in as well...

There is a drawer in my wife's office desk that among other things contains our supply of eplacement drives for the different RAID equipped systems we have on our network - which basically is Monster plus the A/V rendering server, which we built using fast 1.5TB SATA 2 Drives. So I remove the bad drive from the caddy, and swap in a new replacement, then the kid takes that shoe/caddy back downstairs and locks it back into the server, whereupon the RAID Controller marks IT as the Hot Spare for the Volume, and all is right in the world.

In the interim I go online and RMA the bad drive, then Yvonne boxes it up and ships it off to Western Digital, who then sends me a new or refurbished 1TB 7200RPM replacement which gets placed in the RAID Drawer in the desk to start the whole process all over again as needed, if it every IS needed.

Servers at the MBR

So Monster is fully deployed and our virtual host is providing the virtual servers to our very real network, and in the meantime we have to complete the deployment of the MBR Rack. That involves a few minor and some major tasks!

First there is the KVM Switch that we moved from the NOC Server Rack (Monster has its DRAC so it does not need a head or a foot since we can access it via the DRAC Card), so the KVM Switch is no longer needed downstairs.

Back in the day that KVM Switch - a Belken OmniView 8-Port Switch - was the console interface for all of the servers in the racks - which basically meant all of the network resources plus our crack cluster. Today things are a bit different.

For one thing the ancient SVGA 15” Monitor that sits on a shelf in the NOC Rack is still sitting on that shelf - at least until we can recycle or sell it anyway - because we will not be using either that old monitor OR the mini-keyboard and mouse that sits on a retractable shelf in the NOC Rack.

In its place we have bought a used and refurbished KVM Unit (basically that is a folding display with built-in keyboard and touchpad) that is build as a 2U drawer by APC, which we installed in the MBR Rack. Below the KVM is the OmniView KVM Switch, which is connected to the APC KVM and to each server to be installed in the rack.

Now previously the Wiki Server that was plopped on top of the E-Center was the only Server upstairs other than the A/V Server which is not installed in either the E-Center or Rack. But that will change over the course of the next 2 months or so.

New Servers and the KVM

The guts of what was the Wiki Server are to be swapped into a new 19” rackmount server that includes high-density mounts for hot-swapable hard drives for what will become our NAS Server - NAS translating to Network-Attached Storage.

In this case that server will be built using the FreeNAS suite - which is basically a small turnkey appliance-style OS and utility that fits on an 8GB Thumb Drive. That way all of the hard drive space that is actually installed in the server will be hard drive space.

The FreeNAS Thumb Drive is plugged into a USB Port on the motherboard INSIDE the new case, which means it is nowhere that it can be tampered with or lost. Inside the case we have currently installed all of the spare SATA drives that we had laying around in units of 2 identical drives each, so that we can utilize the FreeNAS RAID File System.

Eventually ALL of the drives in that box will be intentionally-purchased large format drives - probably 2TB minimum but eventually 3TB since that is the current hard limit for formatting using the ZFS File System - though really if you think about it since ZFS and the FreeNAS software sets aside a large chunk of the drive for swap space anyway, we could probably get away with using 4TB drives since it only needs to format the portion of the drives it can actually use as data space (the swap does not count)...

Huh, that is certainly something to think about!

Actually how much hard drive space for NAS is hard-limited by the amount of RAM that is in the system anyway - as FreeNAS requires you to have a minimum of 1GB of RAM per TB of disk space, so for example, if you have a server with 16GB of RAM in it, you would be hard-limited to just 16TB of PHYSICAL disc space. Note the emphasis on “physical” there because the RAM to HDD ration is NOT for the actual storage space but the disks with swap included in the figure.

So for now that means we will have 16TB of physical Hard Drives in the server, and 16GB of RAM. That will translate to around 8TB of Swap and 8TB of NAS. Which is just fine - heck, 8TB of NAS is a LOT of storage space!

Eventually though we will increase the RAM in the server to 64GB, and install 64GB of physical drive space for a total of 32GB of NAS. And that will be so sweet! Because not only will that give the family plenty of space for saving photos and movies they take with their phones/cameras, but it will provide more than adequate space for the automatic backup scripts to backup each of their PCs every other night. So yeah, that happened.

Hollywood...

Almost the final step in the network upgrade will be building a new server called “Hollywood” that will be built into a standard 19” rackmount case, and installed in the MBR Rack, and attached to the KVM. Its name should clue you in on its function...

Hollywood is the home-built Media Server I am creating using Ubuntu Linux Server and a pair of TV capture/tuner cards and as much RAM and Hard Drive space as I can cram in!

Using a subscription-based schedule database available online via the web, the family will be able to access Hollywood either via its webserver or by sending the server an email message telling it what shows that they want it to record.

Hollywood will record the shows that they ask it to - minus commercials - which means that the kids will never miss their favorite shows thanks to time-shifting. See they have reached the point in their young-adult-life where having a job as well as going to uni is part of life. This way they still get their TV fix.

In addition, and thanks to some clever scripting and several apps, we can pop any DVD from our collection into the optical drive on Hollywood and it will automagically rip the contents to the hard drive and add it to the program database and its related web display on the server.

Once a title has been added, the source media can be safely stored and thus never risk being scratched! Not only that, but the show/movie/etc can now be watched from any network-attached device in the house that is capable of playing standard video formats!

So the family can watch Game of Thrones using their iPad or iPhone ON the throne (um, yeah, that is kinda gross but still...), or on the TV in the living room, or on their notebook or desktop PCs while laying in bed, or even on a portable device while sitting in the sun on the deck out back, or... Well, you get the idea...

So there you go - we went from an aging largely failing network and moved to a fast, reliable, modern, and useful network that offers all of the modern conveniences! Yeah, technology is good. But what is even better is that almost all of these improvements and the new network is a product of re-use of existing kit, buying refurbished kit, and making most of it myself from cables to servers.

The consequences of that is a new network that was done on the cheap. And what could be wrong with that?!