Sunday, September 6, 2015

Rebuilding the Network

Back in May of this year I revealed that I was in the process of planning out a network rebuild as a consequence of our home network aging out (the newest network infrastructure components were at least 10 years old).

You can read that post by hitting this link for the a Home Network Revival Storypost.


While we are not each of us likely to choose to admit to matters like hero-worship easily, I have to admit that it is the Zen-like awareness of the character Chris Stevens - and specifically the sub-character he is when being "Chris in the Morning" - a DJ for radio staiton KBHR - which he announces as Kay-Bear...  Proof that figuring out what you want and then making it happen by recycling as much as possible is just plain cool man...

Surveying the Reality of the Network Situation

When you begin a new journey it usually helps when you start at the beginning - and in our case, for the re-imagining of our network - the beginning starts where the wires attach to the house.

Since we previously ran our consulting business from offices in our basement that we had finished off with walls, doors, paneling, and a finished ceiling with proper lighting, the place where the wires come into the house was the most logical place to deploy our network racks.

So in that spot there was (past tense) a Relay Rack (a two-post 19” rack used exclusively for network hardware), besides which was two matching 7-foot-tall standard box-style 19” equipment racks of the sort that are commonly known as “Server Racks.”

The pair of Server Racks we chose back in the day were the then-current model of Tripp-Lite 42U rack enclosures. We bought two of them, with side-panels, but no doors (front or back), no wheels, and no handles. Just the standard leveler bolting scheme that allowed us to drive in concrete receivers so that we could bolt the racks to the floor.

For the Relay Rack we chose a Tripp-Lite SR-2Post-45U Open Frame Rack, with a solid steel 4-way-mount base plate that allowed it to be properly secured to the concrete floor so that it could safely contain up to 800lb of hardware (for which it is rated).

Even when it looks like it is still good, old network hardware is just that - old.

Age and Treachery Require Flexibility!

As we contemplated an entire re-design of our network in order to bring it, kicking and screaming, into the 21st Century, we had to make some tough decisions. The main reason for many of them had more to do with the fact that between 1995 (when the original network was planned out) and 2015, I have lost the use of my legs and require a wheelchair to get around.

As a result of this in 2002 we pretty much moved my office to the ground floor (because the basement office was no longer accessible to me), with the room on the ground floor that was designated as my office being specially adapted.

Because I converted my old and no longer manageable career of NetSec Engineering into Writing, and in particular as more than half of my writing was on the video games beat, that room was given the well known and patented Ikea treatment in order to make it as friendly to my needs as possible.

What that translates to is simple enough. Against the front wall of the room we installed a custom-designed and modular solution based on the "Besta" line of shelving and storage units from Ikea. 

Basically that included a primary Besta Unit with a large open TV shelf off-center to the right, with a bank of shelves along its left side. We then added a pair of large shelf banks, one of which was laid down on its side to function as a base for the main unit (to which it was bolted) that basically provides five identical box-shaped storage openings that match the five similar openings that serve as the base of the main unit.

By doing that the main unit is raised in height of the unit so that when we put the second bank or column of storage shelves on the left-side of the assembled main unit, their heights matched.  What that resulted in was a main unit combination that could easily accept our 52” flat screen TV and plenty of storage. 

The base of the TV space has a row of two very wide but low shelves that are meant to accept entertainment kit like DVD players, game consoles, and other modular kit, while the eight box-shaped openings plus the added five additional shelves from the second column laid on its side provide full customization space.

The total number of storage openings is 18 and they are sized so that you can use modular storage objects from Ikea - such as their conversion kits with backs and doors to turn them into independent storage space, as well as drawer inserts and filing container inserts just to name a few.

We went with a selection of different inserts for the office so that I had plenty of file space, as well as proper shelves for books and game cases.

Right so you now know what the central focus of the office consists of. Positioned directly in front of the E-Center is a large flat bed-like construction upon which I can sit, lay, or recline, while I work, with my notebook PC and game controllers easily accessible.

To the left of that work space is a small desk on which the A/V rendering server is placed, with its keyboard and mouse accessible to use from the work space. Finally along the left-hand wall is a proper desk for my wife's use.

It was into this fully functional and well-designed work space that we inserted - to the right of the E-Center and thus occupying the right-front-corner of the room - one of the 45U Server Racks, into which would be installed all of the hardware that for various reasons I may need direct access to.


The New MBR Rack Space and LAG

We decided to call this the MBR Rack, so the E-Center became the MBR Center.  Into the MBR Rack was initially installed a pair of shelves, and some spacer vent plates. The latter being perforated plates that provide front-coverage to serve as spacers for the various hardware while allowing ventilation so that air-flow was ensured.

A 4U spacer plate was installed into the top of the MBR Rack, and directly under that we installed the first of a matched pair of Netgear GS724TV2 Managed Smart Gigabit Ethernet Switches that would serve as the backbone of our new network (bearing in mind that the original network was left in place and fully functional as we created the new one!).

So the pair of GS724TV2 Switches were installed - on in the MBR Rack, one in the NOC Relay Rack - and these were fully and properly configured. That meant adding the proper configuration settings, and then making two custom CAT6 Ethernet cables that connected Ports 23 and 24 on the MBR Switch to Ports 23 and 24 on the NOC Switch.

These two switches were then configured to use the pair of Cat 6 cables as a single 2GB backbone that connects the upstairs and downstairs switches - as otherwise a single Gigabit cable would be all that connected them.

All fully-managed and most partially-managed Gigabit Ethernet Switches today offer up a backbone protocol of this sort, either using dedicated Network LAG configurations (LAG = Link Aggregation) or, as was the case for our Switches, a dedicated aggregated Trunk connection of two cables and two ports for a total of 2GB backbone speed.

The reason that we chose the Netgear GS724TV2 switches was a combination of price (they can be had on eBay dirt cheap) and the decision on Netgear's part to opt for using Trunking in place of proper modern LAG.

Trunking is preferable largely because in theory - and if needed - we could use up to 8 Ports (for a total of 8GB) for our backbone aggregation if we chose to, whereas most of the modern implementations of LAG protocol limit it to just two Ports - or 2GB.

Because we work with A/V files on a regular basis, and because file storage is planned to be centralized on our network, we need a minimum of a 2GB backbone and possibly a minimum of 4GB - so the ability to expand that as needed? Yeah, that was key.

With the two switches and the backbone connection in place, we tested the data transfer speeds and have come to the conclusion that, as long as no more than three users are accessing A/V media or working with A/V files across the network simultaneously at any given time, the 2GB backbone should be sufficient.

But just in case we made a third Cat 6 cable and attached Port 22 to Port 22 after disabling Port 22 on both routers (to ensure that there is no accidental looping) so a third 1GB connection exists that can instantly be added to the existing 2GB backbone to increase it to 3GB if we ever need to. You know, having the raw cable there really was no good reason not to do that.

Proper Termination

The network downstairs was originally a free-flow cable configuration but, once that got too messy, we ended up adding a CAT5E Patch Panel to it ten years ago, and then custom cabling to Keystone boxes installed at the baseboards for a neater cable management scheme.

So all of the systems that connect to the NOC end of the backbone do so via properly terminated permanent CAT 5E cable runs that utilize a patch panel and patch cables for their end-point connections.

Since we were re-imagining our network anyway, and moving the Server Rack and a Switch upstairs into the office anyway, it seemed like a good idea to duplicate that standard upstairs. So we purchased an industry standard CAT 6 Patch Panel and properly terminated all of the cable runs that connect to it using the standard box-and-keystone configuration, so that, at strategic points in the walls, there are nice clean Ethernet jacks in place of spaghetti cable runs.

At the MBR Rack installed just below the Ethernet Switch is a 48-Port CAT6 Patch Panel that is almost identical to the 48-Port CAT5E Patch Panel in the NOC Relay Rack. So how is that for symmetry?

Power Management

The NOC Relay Rack was originally planned and deployed with a standard, grounded, 8-Port PDU that connected to a 2400KVA UPS installed in the base of the Server Rack next to it, which allowed for the various bits of kit in the Relay Rack to have their power independently controlled, using the toggle switches on the PDU.

Of course at the time I was AB and thus able to walk over to the rack and flick a switch - but once I was crippled, and with the hardware now completely inaccessible to me, if I needed a device attached to the rack down there reset, that meant someone else had to go downstairs and flick the switch off, then on again.

Clearly that was not an ideal circumstance - and with so much of the hardware failing as it aged out, I was having to send switch-flickers downstairs several times a day, EVERY day!

So when the Server Rack was moved upstairs, we detached the mechanical PDU from the Relay Rack and brought it upstairs with the Server Rack, installing it at what would ordinarily be waist-high level in the rack. All of the network kit was then powered via that mechanical PDU.

Downstairs in the Relay Rack things were different. Changed. Better!

Via eBay we purchased an APC model AP7900 Network-Attached 8-Port PDU complete with the same style of web-based admin access that the switches use, so that I could basically recycle the power to any connected device remotely.

Yep, that meant that even if I needed something attached to the NOC Relay Rack reset, I would not have to dispatch a human to do it - I could do it myself! And how cool is that?

The AP7900 is a pretty well-known high-quality piece of equipment. Purchased brand new from APC you are looking at $500 a crack - but used on eBay? Less than $90! So that is totally win-win and within the budget.

In fact I am seriously considering buying a second AP7900 for the MBR Rack and I would if that was not so decadent and lazy a decision :)


Servers and Such

The Server Rack downstairs is part of the solution for our re-imagined network, because it is the home to a new server we purchased and over the course of 4 months (as we could afford it) upgraded until it was ready to be deployed.

I am speaking of the Dell PowerEdge 2950 II server we call “Monster” that functions as the VM Host for our network.

Virtual Servers are a new concept for us here - and while it just might maybe be overkill when you stop to consider that the network we are building is being built to provide Internet Access to the residents of our farmhouse, and work resources for a journalist, it probably IS overkill.

But when we did the math, when you combine the average cost of ownership and the electric bill for running four generic PC-based servers which each have PSUs ranging from 450W to 900W (the file server uses a 900W PSU due to the number of drives in it) it actually makes total sense!

The 2950 II has a pair of redundant 720W PSUs in it, and collectively they end up using less than half of the juice that the four original servers that they replace use.

So our new (it is actually a used and refurbished) PowerEdge 2950 II which we purchased from a server rehab outfit in New York City called TechMikeNY who we originally found on eBay (but who has their own store-site online as well) we basically got very very lucky.

The process of finding reliable, reputable, honest, and knowledgeable suppliers for specialized computer hardware in this day and age is often something of a crap-shoot. And considering that we not only needed a reliable and savvy refurbisher but also a supplier who would be willing to answer all of our questions and provide solid and sound advice pretty much whenever we needed it... Well, that is almost an impossible task these days!

To make this absolutely clear, we have no reservations whatsoever in recommending TechMikeNY as a supplier whether via eBay OR via their website sales center. The blokes working there know their hardware inside and out, can (and more importantly WILL) answer any and every question that you ask - even the really stupid ones - and provide a high-quality refurbishing services for servers that they back up 100%.

The MIT Flea - held third Sunday of the month, April thru October, and the place where we obtain the used and refurbished hardware as well as new batteries for the UPS's

Adventures in Server Acquisition

We felt so comfortable - and happy - with the services that they provided when we originally bought the server as an eBay transaction (which is a good idea for you to use to since using eBay adds a layer of insurance to a purchase like that thanks to the eBay and PayPal policy of going to bat for the buyer whenever issues crop up for such transactions) that we had no problems or concerns with hopping from eBay to their web-storefront to do the rest of the purchasing.

At their well-made website we bought the rest of the kit we needed -- which in this case consisted of the following (with source noted):


MONSTER (Our VM Host Server Build)

PowerEdge 2950 II Initial Server Purchase @ eBay -- Total Cost: $250.74
(Buy-It-Now Price = $215.99 plus Standard Shipping = $34.75)
eBay Order and Purchase Number 221295119685 check it out for yourself!

This was basically a starter-package for a Commercial-Grade Enterprise-Level Virtual Server Host System that included the following bits and bobs:
  • Model: PowerEdge 2950 III
  • Processors: 2 x Intel 2.33GHz E5345 Quad Core
  • Memory: 8 x 2GB PC2-5300, FB
  • Hard Drives: 2 x 1TB 7.2K SATA 3.5"
  • Drive Bays: 6 x hot-swap 3.5"
  • Power Supplies: 2 x Dell 750 Watt (redundant PSUs)
  • Number of Power Cords: x2
  • RAID Controller: PERC 5i w/battery backup installed
  • Network Interface: 2 x Broadcom Gigabit Ethernet
  • Video Card: ATI Technologies Inc ES1000
  • Optical Drive: DVDRW-CDROM+
  • Remote Access Controller: DRAC5
  • OS: Windows Server 2008 R2 Evaluation Edition
  • Warranty: Standard 30 Day Included
Yes, all of that and only $250?! Not only is that a heck of a deal, it is an awesome server! The ting is it is just what we went looking for - a starter kit. We knew going in that we would need to buy additional bits and bobs before we could actually deploy this as our server solution.

So happy were we with the initial purchase that when we found out that TechMikeNY was in the process of building its own web-based storefront, we waited for them to complete it and roll it out rather than doing the rest of the purchases via eBay (because why? Because we could save a few bucks on eBay fees that is why!).

When their storefront finally deployed we bought pretty much most of the rest of the kit we needed, which in case you are contemplating doing the same thing we did, was:
  • x3 1TB 7.2K SATA 3.5" @ $49 each;
  • x3 HP Drive Caddies @ $15 each;
  • x1 Dell PowerEdge Rapid Rails Kit @ $35;
  • x1 Intel Pro Dell X3959 Dual Port Gigabit Ethernet NIC Card PCI-E D33682 Card @ $36.99
Basically we needed the additional drives in order to create a pair of RAID Volumes - the first one being a 1TB Volume for the OS that consisted of a pair of mirrored drives (plus a dedicated Hot Spare Drive) for the OS and VM Host Server configuration and the first virtual server, which is a network utilities server.

The second Volume was an identical pair of 1TB mirrored volume for the second pair of servers - our Wiki and Database Server and a Testbed Server for online game servers.

To make that work we needed to add another dual-port Gigabit Ethernet Card since each server would need its own dedicated Ethernet and so we needed four Ethernet Ports.

Finally there as the Rails - which you cannot install this beast into a rack without!

The only thing we still need (but there was not enough in the budget for at the time) was one more of the 1TB 7.2K SATA 3.5" drives and an HP Drive Caddies to install it as the designated Hot Spare for the second volume.

See the way that this works - because it is necessary for us to install the server in the NOC Rack far away from humans due to the incredible noise that Enterprise-Level servers tend to make thanks to their massive cooling requirements - trust me this Monster sounds like a 737 taking off when its fans kick in!

So yeah, since it has to be racked away from anywhere that humans are likely to be found, having pre-configured and deployed hot spares makes total sense! What a hot spare does is just what it sounds like: if something goes wrong with one of the two drives that make up the mirrored volume, the RAID controller immediately signals an audible alarm (that nobody will hear) and emails a warning alarm notice to the admin email address (which of course I am likely to see soon enough) while at the same time activating the Hot Spare and rebuilding the Array using it!

That way we never lose the mirror and insurance that it offers, and the server can take the necessary steps to ensure the continued operation of the volume. I get the email and dispatch one of the kids to pull the bad drive - the HP caddies actually have indicator lights on each that will show which drive has gone bad or crashed, but there is also a display panel on the front of the server that will tell them which slot the bad drive is in as well...

There is a drawer in my wife's office desk that among other things contains our supply of eplacement drives for the different RAID equipped systems we have on our network - which basically is Monster plus the A/V rendering server, which we built using fast 1.5TB SATA 2 Drives. So I remove the bad drive from the caddy, and swap in a new replacement, then the kid takes that shoe/caddy back downstairs and locks it back into the server, whereupon the RAID Controller marks IT as the Hot Spare for the Volume, and all is right in the world.

In the interim I go online and RMA the bad drive, then Yvonne boxes it up and ships it off to Western Digital, who then sends me a new or refurbished 1TB 7200RPM replacement which gets placed in the RAID Drawer in the desk to start the whole process all over again as needed, if it every IS needed.


Servers at the MBR

So Monster is fully deployed and our virtual host is providing the virtual servers to our very real network, and in the meantime we have to complete the deployment of the MBR Rack. That involves a few minor and some major tasks!

First there is the KVM Switch that we moved from the NOC Server Rack (Monster has its DRAC so it does not need a head or a foot since we can access it via the DRAC Card), so the KVM Switch is no longer needed downstairs.

Back in the day that KVM Switch - a Belken OmniView 8-Port Switch - was the console interface for all of the servers in the racks - which basically meant all of the network resources plus our crack cluster. Today things are a bit different.

For one thing the ancient SVGA 15” Monitor that sits on a shelf in the NOC Rack is still sitting on that shelf - at least until we can recycle or sell it anyway - because we will not be using either that old monitor OR the mini-keyboard and mouse that sits on a retractable shelf in the NOC Rack.

In its place we have bought a used and refurbished KVM Unit (basically that is a folding display with built-in keyboard and touchpad) that is build as a 2U drawer by APC, which we installed in the MBR Rack. Below the KVM is the OmniView KVM Switch, which is connected to the APC KVM and to each server to be installed in the rack.

Now previously the Wiki Server that was plopped on top of the E-Center was the only Server upstairs other than the A/V Server which is not installed in either the E-Center or Rack. But that will change over the course of the next 2 months or so.


New Servers and the KVM

The guts of what was the Wiki Server are to be swapped into a new 19” rackmount server that includes high-density mounts for hot-swapable hard drives for what will become our NAS Server - NAS translating to Network-Attached Storage.

In this case that server will be built using the FreeNAS suite - which is basically a small turnkey appliance-style OS and utility that fits on an 8GB Thumb Drive. That way all of the hard drive space that is actually installed in the server will be hard drive space.

The FreeNAS Thumb Drive is plugged into a USB Port on the motherboard INSIDE the new case, which means it is nowhere that it can be tampered with or lost. Inside the case we have currently installed all of the spare SATA drives that we had laying around in units of 2 identical drives each, so that we can utilize the FreeNAS RAID File System.

Eventually ALL of the drives in that box will be intentionally-purchased large format drives - probably 2TB minimum but eventually 3TB since that is the current hard limit for formatting using the ZFS File System - though really if you think about it since ZFS and the FreeNAS software sets aside a large chunk of the drive for swap space anyway, we could probably get away with using 4TB drives since it only needs to format the portion of the drives it can actually use as data space (the swap does not count)...

Huh, that is certainly something to think about!

Actually how much hard drive space for NAS is hard-limited by the amount of RAM that is in the system anyway - as FreeNAS requires you to have a minimum of 1GB of RAM per TB of disk space, so for example, if you have a server with 16GB of RAM in it, you would be hard-limited to just 16TB of PHYSICAL disc space. Note the emphasis on “physical” there because the RAM to HDD ration is NOT for the actual storage space but the disks with swap included in the figure.

So for now that means we will have 16TB of physical Hard Drives in the server, and 16GB of RAM. That will translate to around 8TB of Swap and 8TB of NAS. Which is just fine - heck, 8TB of NAS is a LOT of storage space!

Eventually though we will increase the RAM in the server to 64GB, and install 64GB of physical drive space for a total of 32GB of NAS. And that will be so sweet! Because not only will that give the family plenty of space for saving photos and movies they take with their phones/cameras, but it will provide more than adequate space for the automatic backup scripts to backup each of their PCs every other night. So yeah, that happened.


Hollywood...

Almost the final step in the network upgrade will be building a new server called “Hollywood” that will be built into a standard 19” rackmount case, and installed in the MBR Rack, and attached to the KVM. Its name should clue you in on its function...

Hollywood is the home-built Media Server I am creating using Ubuntu Linux Server and a pair of TV capture/tuner cards and as much RAM and Hard Drive space as I can cram in!

Using a subscription-based schedule database available online via the web, the family will be able to access Hollywood either via its webserver or by sending the server an email message telling it what shows that they want it to record.

Hollywood will record the shows that they ask it to - minus commercials - which means that the kids will never miss their favorite shows thanks to time-shifting. See they have reached the point in their young-adult-life where having a job as well as going to uni is part of life. This way they still get their TV fix.

In addition, and thanks to some clever scripting and several apps, we can pop any DVD from our collection into the optical drive on Hollywood and it will automagically rip the contents to the hard drive and add it to the program database and its related web display on the server.

Once a title has been added, the source media can be safely stored and thus never risk being scratched! Not only that, but the show/movie/etc can now be watched from any network-attached device in the house that is capable of playing standard video formats!

So the family can watch Game of Thrones using their iPad or iPhone ON the throne (um, yeah, that is kinda gross but still...), or on the TV in the living room, or on their notebook or desktop PCs while laying in bed, or even on a portable device while sitting in the sun on the deck out back, or... Well, you get the idea...

So there you go - we went from an aging largely failing network and moved to a fast, reliable, modern, and useful network that offers all of the modern conveniences! Yeah, technology is good. But what is even better is that almost all of these improvements and the new network is a product of re-use of existing kit, buying refurbished kit, and making most of it myself from cables to servers.

The consequences of that is a new network that was done on the cheap. And what could be wrong with that?!

Wednesday, September 2, 2015

. . . Electronic Entertainment Expo (E3) 2015 Analysis Part 2


(or How I Learned to Love Ta Bomb)
Part 2 of 7 

"Time is what we want most, but what we use worst."

William Penn

The sentiment expressed in that quote reminds me of the attitude that certain editors from certain game sites (both of which shall remain nameless since I like having work) often feel about how the writers choose to spend their time, as opposed to how the editors believe that they should be spending it, and it certainly does sound familiar.

It is odd how this one particular sub-editor, who I happen to know has NEVER been to LA let alone covered E3, somehow knows precisely to the minute how long it should take us to travel from LAX to Union Station.

How long the walk is from Union Station to our hotel; generally how long it should take to check in, and, allowing half-an-hour for us to settle in and freshen up, feels fully justified in setting up meetings for us on the very same day we arrive!  And yet it happened.

Note: As this is really an analysis AND a traditional travel feature article, we include the costs of things where it is appropriate to do so and especially in the case of kit since you may want to incorporate some of the wicked clever and massively convenient tech we use to do our job - just saying...

They say that the journey is half the fun, but from our point of view the journey is half of the frustration...
Planes, Trains, and Automobiles

Traveling to Los Angeles is accomplished in basically one of three ways: Airports, Train Stations, and Highways - Planes, Trains, and Automobiles.

It has been our experience, when the nature of the journey is a cross-country run, that the least attractive method is via the Interstate Highway Network, particularly if you have to do all or some of the driving.

An interesting fact: the first Interstate Highway to be built in the Los Angeles area and that served as a centralized access point for LA was Interstate 110 (aka State Route 110), which also happens to be the highway made famous in the semi-animated motion picture Who Framed Roger Rabbit?

That film was a fantasy interpretation of the real-life conspiracy undertaken by a commercial passenger transportation company called National City Lines (NCL), who along with major corporations like General Motors, Standard Oil, and Firestone set out to purchase and dismantle the tram-based public transport networks in major metropolitan cities.

Between 1938 and 1950 NCL took over commercial public transportation lines in 25 major cities in the USA, and began to dismantle the environmentally friendly electric tram lines in favor of bus lines.

Those use buses. You know, large petrol-powered public transport devices that used gasoline and rubber tyres in great quantity annually? Built and sold by companies like General Motors? Those things?

Even if you are traveling in a modern, air-conditioned, WiFi-equipped bus, getting to LA from anywhere east of the Mississippi River generally means between five and seven days of brutal travel through some pretty inhospitable landscape. We don't recommend it but it is the cheaper way to go.

The only way to fly between Boston and New York City.  Seriously!

21st Century Passenger Rail

The other relatively slow alternative is to travel by train. If you are coming from outside of California in America today, that means you have a single choice: the government-owned Interstate Passenger Train Service commonly known as Amtrak.

Amtrak's real name is the National Railroad Passenger Corporation, and it is a partially-government-funded for-profit corporation that provides medium to long-distance intercity passenger rail service throughout the contiguous United States. Well, sort of. I mean the routes that they serve are pretty restricted and do not include every major US city or even most of them.

For example you cannot take Amtrak from Chicago to Las Vegas, because why? Because they don't service Las Vegas as a destination, that's why. But if you are traveling between cities that they do service, then you could do far worse.

The trains that Amtrak runs are all modern, well maintained, well-staffed, and offer most of the amenities one might expect from their European counterparts. Club and Dining cars, Observation Cars for scenery watching, and of course Bar Cars where you can get a few pleasant libations and seek out the company of like-minded train passengers.

A choo-choo ride on Amtrak can be a very pleasant way to get from Point-A to Point-B - especially if you can afford a private compartment, which during the day contains your seats and at night is converted to pretty comfortable beds.

They have TVs, radios, in-compartment WiFi Internet access, standard 110v grounded power outlets, and food service delivery to your compartment (if you are disabled) - that's pretty spiffy for the 21st Century!

The Club Car for the famous Illinois Central train City of New Orleans, offering regular passenger service from Chicago to New Orleans.  In addition to the Club Car pictured here, it sported first-class and second class dining cars, meal service for third and coach via a snack bar car, and a long ling of Pullman Compartment Cars, each with its own formal attendant to turn down the beds and deliver hot coffee, fresh squeezed and chilled orange juice, and the morning paper with your breakfast.

It was not always so I am afraid. In fact it almost wasn't period. You see the decline of passenger rail began in the mid-1950s and, by the late 1960s it looked like passenger rail service in America was due to meet the same fate as the inner-city tram networks.

But in 1970 President Richard M. Nixon stepped in to save the day, and helped to pass the Rail Passenger Service Act, a new law that basically created Amtrak and, what is more, preserved passenger rail services in America for generations to come.

It may not sound important now, but according to the experts, when gasoline prices rise above $7 a gallon, Americans are going to quickly re-discover passenger train travel in a big way.

You should read up on that, because trust me when I say this, you will be very happy that President Nixon took the steps that he did to preserve passenger rail in this country when you have to refinance your home just to fly from Boston to LA.

Because $7 a gallon is a magic number at which the cost of train travel compared to ANY other form of long-distance public transportation begins to slide towards cheap.

The cost of a train ticket from Boston to New York City at the time that I write this is $52 Coach or $113 Business Class. The difference between Coach and Business is spacious, comfortable seats and an extra-large tray table, 110/120v plugs in your at-seat power outlet, and complimentary non-alcoholic beverages.

What a private compartment between New York City and Los Angeles actually looks like when you travel by train...  The seat folds out into a bed and there is a second bed above.  The room has a small bathroom with shower, TV, fold-out table for meals, and WiFi as well.  It is not all the comforts of home but it is pretty close to it!
Coach, on the other hand, offers a seat with individual reading light, and depending on the car type may include a 110/120v electric outlets right at your seat. Or not.

One key and important detail you may not be aware of though, is that when you opt for Business Class during the reservation process online, that key choice adds any Acela High-Speed Trains on that route to the possible seats you can book. Because Business Class is the starting class for that wicked ride.

Taking the train to New York from Boston Business Class on the Acela will take 3 hours 25 minutes. Taking the same route on a regular train in Coach? That takes 4 hours and 11 minutes.

When high-speed rail is finally added coast-to-coast though, do the math for that difference and you will discover that Business Class from Boston to LA will save you 32 hours (on average) in travel time!

Either way though, taking the train when gas hits $7 a gallon will cost a fraction of the price of a plane ticket, and is estimated (obviously we don't know what sort of car you drive or what gas mileage you get) to be about half the cost in petrol (but probably less than that).

The very slow move to create and adopt high-speed rail to America's aging rail infrastructure started in the New England region, where the dense rail lines are being retrofitted to support bullet trains and so, allow for rail travel at speeds that cut travel time by as much as half.

The reason that the upgrade process began along the New England rail corridor has less to do with logistics and more to do with the fact that, unlike the rest of the USA, travelers in New England never really gave up on using trains to travel.

That is important because it allows the government to study the use pattern and effectiveness of the new train service in an environment that will accurately reflect use in other areas, once those other areas start using rail again. While some real progress has been made, true high-speed train service still does not exists along the whole route. Yet.

When high-speed rail is a reality in the New England corridor, you will actually be able to travel from Boston to New York City or Washington, DC faster by train than you can by plane! Well, faster when you include the usual two-hour delays to get through security, but still.

So yeah, you can take the train to LA - but doing so from the east coast means a minimum four-day travel time, so if you are pressed for time or your boss actually expects you to be at your desk on the Monday after E3? Probably not an option for you just yet.

And that leaves passenger air travel!

"The world is still a weird place, despite my efforts to make clear and perfect sense of it." -  Hunter S. Thompson

Virgin America Business Class - when someone else is paying for it...

The Jet-Powered Butterfly Effect

“The Guide says there is an art to flying", said Ford, "or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”
― Douglas Adams, Life, the Universe and Everything

The International Air Transport Association (IATA) airport code for Los Angeles International Airport is “LAX” - while the International Civil Aviation Organization (ICAO) airport code is “KLAX”.

These codes are used by the air travel industries - and were established to serve as official designators for each airport served by the system - in order to make the experience for passengers more convenient. How?

Well for one thing they are the reason your bags end up arriving with you at Los Angeles International Airport (LAX/KLAX) instead of, say, some other airport in LA like John Wayne Airport (SNA/KSNA) or LA/Ontario International Airport (ONT/KONT).

Back before the industry adopted the IATA and ICAO system that sort of thing actually happened; when two planes were departing from the same airport, both going to different airports in Los Angeles and the bags said Los Angeles it was a crap-shoot.

That said, your only contact with this code system will be on your tickets, and on the baggage tags attached to your stuff at airport check-in desks.

Considering that LAX handles more than 70-million passengers each year, that should be a number large enough to clue you in on the method most people choose to use to visit the City of Angels.

After getting off of the plane and claiming our baggage, unlike the vast majority of the rest of the games journalists in LA to cover E3, we do not make our way to the car rental desks. We do not head for the many taxi stands either. The limo pick-up area for the hotel limos? Nope. And the Towncar kiosks are also not our destination. Oh, we were them once, but after a few E3s you soon learn that there is a better way to handle getting around LA.

If you take the time to learn the light rail lines and snag a 7-day Pass Union Station (pictured above) is THE transport hub for the LA area.  From here you can go anywhere - and it is just three stops to the LACC and E3!
That better way is called “Light Rail” and Los Angeles is literally setting the standard for Light Rail Public Transport in America. To make use of it though, at least until they manage to get a station to the airport itself, means you have to make a little effort.

That starts with exiting the airport and going to the bus stop, located just outside the arrivals access area, and grabbing the bus to Aviation Station - the LAX Metro Connection Station.

From there, you take the Metro Green Line 7-stops to Rosa Parks Station at Imperial / Wilmington, then change to the Blue Line and 11-stops later get off at 7th Street, which is the Metro Center Station.

Change to the Red Line and, 4-stops later you arrive at Union Station. Total elapsed time? Exactly 53-minutes - and mates? You can NOT get from LAX to Union Station faster by Taxi or Limo. We've tried, it just is not going to happen.

Now granted, you can still take the taxi - and pay the average meter rate of $74.39 (plus tip) for the privilege, or an average $160 for the limo unless the big-name hotel you are staying at offers limo service pickup at the airport, in which case your end of that will average $59 (no, it is NOT free!).

Or take the train and it will cost you $8. You do the math but considering the fact that we only get reimbursed for travel expenses AFTER the trip, we choose the train.

Not only can you get back to the airport via that route, but thanks to the light rail access at Union Station and nearby if you want to walk a little in China Town, you can also use it to get to and from E3 at the LA Convention Center.

So when you arrive at Union Station - even before you head to your hotel - hit the TAP machine there and buy a 7-Day Pass -- which is valid for 7 consecutive days from the first tap (use) and expires at 3am on the day following expiration, which means that unless you are staying more than a day after E3 ends, it will also get you back to the airport.

The 7-Day Pass includes all Metro services but you should be aware that additional charges apply to ride the Metro Silver Line and certain Metro Express Buses.

This is good info and will save you massive amounts of money once you download the free LA Metro Transit Guru App for your iOS devices. Transit Guru not only helps you figure out what route you need to take, it will tell you when the next train or bus will arrive at your stop and when you can expect to arrive at your destination. How cool is that?

Technology is awesome!

In Los Angeles the speed limits actually apply to the sidewalk as well as the road.  These nice young men in their matching outfits decided a verbal warning was the ticket, not a ticket, which I was just fine with - and who knew?!

Pre-Show Press Briefings

When we arrived in LA we already had a basic schedule set down in stone months ago... That was the official (though it did change) Media Briefings from the major game Studios and Publishers, and Publishers who also happen to make the devices we game on, like Microsoft, Nintendo, and Sony.

Note: All times are local times, which are GMT-8. Also this list looks neater than it was because we are writing it AFTER the fact. Several of the players changed the time of their briefings on very short notice, so it was not as cut-and-dried as it may look...

Sunday, June 14
  • 1830 Bethesda Official Press Briefing / Event.

Monday, June 15
  • 0930 Microsoft Official Press Briefing / Event.
  • 1300 Electronic Arts Official Press Briefing / Event.
  • 1500 Ubisoft Official Press Briefing / Event.
  • 1800 Sony Official Press Briefing / Event.

Tuesday, June 16
  • 0900 Nintendo Official Press Briefing / Event.
  • 1000 Square Enix Official Press Briefing / Event.
  • 1700 PC Gaming Show Official Press Briefing / Event.

So there you go! A very light schedule. Why, this was almost a holiday! We arrived with practically nothing to do, right? Actually no, not so much really. This is not a paid holiday you see, it is work.

So far we have arrived in LA and, at LAX, made the transition to ground mode, and used the rather extensive light rail network to get from Point-A (LAX) to Point-B (Union Station).

Now that we have arrived at Union Station and picked up our 7-Day Metro Pass, we need to hit the convenience store that is on-site right there and purchase some essentials.

Mostly these consist of snacks and beverages - since it is cheaper to do that than to consume the beverages in our hotel mini-fridge. On that note, we strongly recommend you buy a fairly large quantity since you WILL be missing some meals over the course of the next week thanks to heartless scheduling.

By arranging in advance for reasonably nourishing foodstuff to be on hand back in your room, you allow for quick transition from events to your notebook, where you can write and then file your copy while you graze on your Emergency Food Supply, and quaff beverages that you are NOT paying $5 each for!

These booth babes are following neither the spirit OR the letter of the law when it comes to the costume restrictions at E3.  Just so you know, bikinis?  Right out.  No question about it.
There seriously will not always be an extra hour to duck out and hit a restaurant down the street from the convention center and the last thing you want to do at one in the morning is look for an open restaurant on your way back to the hotel.

If you CAN carve out an hour for a meal though, on-site at E3 this is the list of our chosen, tested, and approved restaurants nearby the convention center that you may find useful:

  • ChocoChicken 403 W. 12th Street (0.4 miles away) American Fusion. You should try it.
  • Fernando's Taco Inn at 815 W Olympic Blvd (0.4 miles away) Cali-Mex Cuisine.
  • Fleming's Prime Steakhouse & Wine Bar 800 West Olympic Blvd (0.4 miles away) 5-Star.
  • IHOP 800 S. Flower St. (0.7 miles) American Comfort Food. Brekkie Anytime.
  • Sprinkles Cupcakes 735 S Figueroa St (0.8 miles away) Comfort Food.
  • Sugarfish 600 West 7th Street (0.8 miles away) Japanese Cuisine. Best. Sushi. Ever.

If you cannot take a full hour to grab food then go outside the main entrance and across the street to the parking lot - where you will find some of the best Food Trucks in LA! Turn right and cross the tracks and there are some more trucks along the side street here (look for the red Coke Sign).

We can firmly recommend:

  • Dogtown Dogs -- Menu Spotlight: Gourmet Hot Dogs.
  • The Grilled Cheese Truck -- Menu Spotlight: Stuffed Mac & Cheese Grilla.
  • India Jones -- Indian Lamb Burrito
  • Kogi -- Menu Spotlight: Korean Fusion-Tacos.
  • Ta Bom* -- Menu Spotlight: Linguica Frita

* This Food Truck has no relationship whatsoever to the sub-title of this article series, “or How I Learned to Love Ta Bomb” which is a reference to a completely different event that will only really make sense if you happen to be a member of the large group of game journos who regularly meet up at events like E3 and CES and who are in on the joke.

Checking In

With emergency snacks and beverages managed, we head to the hotel and check in, claim our messages, and then take all their stuff out of the mini-fridge and pile it on top, and replace it with our stuff (the next day housekeeping basically removed all the stuff piled up on top and left a note telling us that if we needed anything ring the front desk anytime but otherwise the fridge was ours to use!).

Waiting at the desk for us were several notes from a few sub-editors and an editor for the different pubs we were covering E3 for. Peter actually had his own gig for this year's E3 and how cool was that? His editor had set up a hardware briefing for him I ended up piggy-backing on.



The messages basically said “Check your email NOW!” Sadly I must confess we did not in fact instantly check our email. Instead we finished the check-in process, then made our way to our room where we performed a seriously important set of preparation rituals not at all unlike the spells that Hermione Granger cast in the Forest of Dean when they were setting up their campsite:

If we're staying, we should put some protective enchantments around the place," she replied, and raising her wand, she began to walk in a wide circle around Harry and Ron, murmuring incantations as she went.

After safely entering our room and after we transferred our supplies to the mini-fridge, unpacked our bags, we then hang up and drawer our clothes, verifying that certain invaluable objects were present and accounted for.

After Peter moved one chair into the bathroom to serve as my towel and robe holder, I rolled over to the table in the corner and methodically unloaded my Brad Thor Alpha Jacket from SCOTTeVEST (around $200 or so IF you can buy one - they are having trouble keeping them in stock it seems).

The jacket is what is known in journalism and military circles as a pack-it - a piece of clothing with lots of hidden and/or deep pockets designed so that you can store objects in them and they do not imprint (you cannot tell what the object is from outside). The Brad Thor is worth its weight in gold as it doubles as my go-bag AND as my secret carry-on bag.

A lot of airlines today charge you $25 to $50 for any second bag you carry onto the plane with you, and some consider a computer bag as a second bag if you also have a regular carry-on. The pockets of my Brad Thor allow me to carry pretty much all of my electronics - including my notebook PC and my iPad - so that I can use my computer pack for other things.

Setting Up Camp

Peter and I begin the rather complex process of setting up camp - with the table in the corner being sufficiently large enough to serve as the common workstation for both of us. Peter pulls out the surge strip and plugs it into the wall and the notebooks get setup and plugged in to that for a little extra insurance.

That way they can charge-up the battery juice consumed during the trip (the plane was an older 737 that did not have power at the seats) while we cast all of the usual protective charms ala Harry Potter that we use when traveling on assignment.

We start with a “Cave Inimicum” charm, followed by “Fianto Duri” and “Muffliato” -- meaning we set up our D-Link DIR-510L Portable Router ($75 from D-Link) on the desk in the corner attaching its CAT5e wire into the wall jack at the desk and plugging it into its APC P1T 120V Portable Surge Protector ($12).




A quick hop on my notebook and call to the front desk to get the WiFi information - SSID and Key - and the 510L was on their network.

When the Network Services page loaded we opted for the Week-Plan to save a few bucks.

The standard day-plan works out to $15 per day, per device - and as we had six devices to connect in theory that would have cost us $90 a day but “Cave Inimicum” plus “Muffliato” and “Fianto Duri” brought the day-price to $15 if we wanted to go that route.

The Week-Plan was only $75 (a savings of $30 looking at it one way, or a savings of $660 looked at the other way!). Basically what the 510L does is serve the following functions:

  • Single Connection Point: It is the default (and as far as the hotel is concerned ONLY) networked device that will access the Internet from our room.

  • WPA2: It creates a private network for our room (so all of our traffic inside the room stays INSIDE the room) and using WPA2 secures the network - as much as we can claim that WPA2 secures any network really...

  • DHCP Server: It provides DHCP service to any device that knows its SSID and key (it is pre-configured to not broadcast the SSID and the key is pretty freaking long but all of our devices were previously set up and tested to be sure that they could use it).

On top of the 510L we placed our Toshiba Canvio Slim II 500GB portable storage device ($36), plugging its USB3 cable into the 510L. In addition to providing shared portable storage on which to backup our notes and copy.


Each evening we copy the contents of all of the press kits we receive to the drive so that we don't lose them when we lose the tiny thumb drives that are used to distro the packages - and we WILL lose them. Happens every E3.

We then cast “Repello Muggletum” (a Muggle Repelling Charm) by placing on top of the stacked 510L and Slim II a cardboard sign we printed back at the bullpen, before embarking on our trip, that reads “Please do not touch this hardware - it is doing hardware things while we go do human things and does not like to be disturbed!!!”

That must be a powerful charm because is all of the assignments we have used it for, none of the housekeeping have ever touched our kit.

NOW we checked our email, and learned that during our travels the sub-editors had been busy and we had appointments - including for that day - to a number of briefing events. Mostly it seemed that these were briefings for games that were not being showcased at E3 this year - long-range stuff for games that the publishers and studios wanted to talk about but had not integrated into their E3 booth routines.

An Updated Schedule

If you were thinking that the schedule we ended up with only covered the big Pressers that the major studios and the industry players put on in the days just before the show kicks off, well, you'd be wrong...

To help you with figuring out how this works, we'll start by reminding you of the official coverage schedules we started with above - go read it again so it is fresh in your mind - and then keep reading this so you can see what the real schedule looks like AFTER it was fiddled with by El Editors...
The schedule we had before the ramps dropped on the LCA's showed only the official briefings, but now thanks to the sub-editors, we have a much fuller and more complex one:

Saturday, June 13
  • 1100 Brunch with PR from Publisher A - briefing for 2 fantasy titles.
  • 1300 PR from Studio A - briefing for Wargame.
  • 1500 PR from Studio B - briefing for Wargame.
  • 1630 PR from Studio C - briefing for Asian-Based F2P Fantasy Games.
  • 1900 Studio-Hosted (Studio D) Dinner at LA Restaurant with private briefing.
  • 2230 PRs from Publisher B - Private Party at LA Nightclub with hosted mini-briefings.

Sunday, June 14
  • 1100 Brunch with PR from Well-Known Publisher C - briefing for 4 sim titles.
  • 1400 PR from Well-Known Company A - briefing for gaming peripherals.
  • 1530 PRs from Publisher B proper briefings.
  • 1830 Bethesda Official Press Briefing / Event.
  • 2200 PRs from Publisher D - Private Party at LA Nightclub with hosted mini-briefings.

Monday, June 15
  • 0930 Microsoft Official Press Briefing / Event.
  • 1300 Electronic Arts Official Press Briefing / Event.
  • 1500 Ubisoft Official Press Briefing / Event.
  • 1800 Sony Official Press Briefing / Event.
  • 2100 Publisher B's Online Division Dinner at LA Restaurant w/ private briefing.
  • 2300 PRs from Publisher E - Private Party at LA Nightclub with hosted mini-briefings.

Tuesday, June 16
  • 0900 Nintendo Official Press Briefing / Event.
  • 1000 Square Enix Official Press Briefing / Event.
  • 1400 Lunch Briefing; Publisher-Hosted at LA Business Venue.
  • 1700 PC Gaming Show Official Press Briefing / Event.

When you look at this list, we totally understand if you get the wrong impression. That covering E3 must be just one wild trip in which game studios and publishers host booze-fueled parties and dinners, where games journos get smashed every night, eat for free, and ride around town in limos paid for by games publishers and the larger studios.

In other words they spend time living in the lap of luxury... The terrible thing is that, in part, you would be correct! But let me tell you a little secret. The kids who allow themselves to be sucked into that trap? That will be the last E3 they cover for a very long time and maybe even the one and only E3 that they ever get to cover period.



Because I can tell you from personal experience that editors expect you to (a) turn in copy that they can read and more important USE, and (b) you cannot do that if you are drunk or stoned.

Contrary to popular belief, editors are not human.

They don't have any sort of sense of humor at all. They do NOT accept apologies or excuses. And I totally get that - because they are under deadlines just like the rest of us.

So in the interests of transparency and with regrets for bursting your bubble - our experiences with the various appointments that were arranged for us by the sub-editors largely turned out to be nothing more exciting than sipping carbonated water or diet soda while we sat through mini-briefings - many of which we could hardly hear.

The dinners were okay... Actually they were not the sort of food we would have chosen if it were up to us, and mostly we spent our time taking notes rather than eating - because the briefings took place AT dinner. Yeah. Lap of luxury.

The private party briefings were even worse, because it was hard to hear over the music and you got the definite sense from the PRs doing the briefing genuinely resented us.

Resented us because we were not going with the program and scheme that they had set out to make happen -- because it is clear by the way that they treated us that the point to the events was not to brief us on their games, but to do as little as possible and party.

From their point of view what they expected was for us to accept the game PR kit, then hit the open bar, dance with the cute girls I am pretty sure were paid to be there, and dance with us, and basically get sloshed under the guise of work.

While the younger kids did just that, they still had to deal with the old farts and our questions about the games that they did not want to answer. Basically we persisted in asking them until they either gave us the answers we were looking for, or they made it clear said answers would not be provided, at which point we left the events because why stay there when we can write or sleep?
The truly disturbing fact here is not that all this happened, but that it actually took place in the days BEFORE E3 actually opened!

E3 hadn't even officially started - heck, the major press briefings hadn't yet happened! We were already thrust into scheduled meetings and briefings that pretty much guaranteed we would have to operate on short sleep rations the entire week!

If it sounds like I am complaining or whinging, well I am not.

This is the territory we agreed to when we accepted the assignment to cover E3, and we knew that going in. We also understood that the faster we obtained what we were sent there to get, the sooner we could leave. That's the difference between seasoned journos and kids.

The latter want to party all night and meet celebrities, and the former? They want the freaking quotes they were sent to get, and the press package and briefing so that they can hit Mr. Churros on the way home, then file their copy and go to bed!



How to refer to E3 like the cool kids do
 
Put together and put on each year since 1995 by the Entertainment Software Association (ESA), its proper name is the Electronic Entertainment Expo - or E3 for short - and it is without question the world's premier trade show for computer and video games (and related products).

If that was all that there was to the story, we'd be finished now, but hey, it is not. Specifically E3 has a rich and varied history and, like most long-running cultural events, it has experienced a vast array of issues and even scandals.

Heck it would be strange if it had not, but we are not here to air out the dirty laundry of E3, we're here to help you know and understand it in a way that permits you to be cool about it. You see the Cool Kids are not cool because they are cool, they are cool because they know what they are talking about. Put another way, knowledge is power, but among geeks and gamers, knowledge is cool.

So gaining that level of status inside gaming first requires you to understand - and to appreciate - many of the events and issues that served to shape what we think of as |the Electronic Entertainment Expo. Seriously!

When it was founded in 1995 it was christened “The Electronic Entertainment Expo” and just that because it was not clear that there would be another. When that first Expo turned out to be a major success, the ones that followed were named with their year attached.



It wasn't until after “Electronic Entertainment Expo 1996” wrapped up that the ESA began to widely refer to it as “The Annual” in terms of its ID. As in “The Annual Electronic Entertainment Expo” in its promotional materials.

The thing is each successive Expo grew bigger - and wilder - than the last, until the debacle that was “E3 2006” happened. If you plan to be one of the Cool Kids you need to not only know about this, but be able to discuss it.

A series of unrelated events eventually lead to the near-total-demise of the event known as E3, and certainly lead to its temporarily being turned into the shadow of a ghost of its former self!

Knowing that history - and the events that lead up to it - is an important element in knowing E3 - so let's go to school, shall we?

The Disaster that Was E3 2006

Before we rush to judgment it is really important to remember that the disaster that was E3 2006 was not down to any single event. No single powerful disaster or disasters stand out. No smoking gun.

It was, in fact, the culmination of lots of mostly unrelated issues that finally blew up in the face of the event host, the ESA, and the mainstream publishers and studios.

While nothing can be held out as THE THING that marred the image of E3 for all time - what actually happened at E3 that drew the event into an unfortunate position and that provided nearly endless fodder for late-night talk show host monologues and seemingly built into one nasty storm made up of equal parts embarrassment, bad PR, and that late-night humor finally pushed the ESA to the breaking point.

In the months that followed June 2006 the gamer community did a lot of snickering, while certain studio and publisher bigwigs found themselves throwing-up a little in their mouths each time the subject of E3 came up. None of them would have been so public about it though, if they had only understood the consequences that their actions and attitude would end up causing the ESA to do. There was plenty of warning too - this is not a case of hindsight being 20/20 so much as it is the games industry indulging in a combination of irresponsible actions.



What hindsight tells us is that the situation was more a matter of wish-fulfillment and what psychologists and sociologists call “Magical Thinking” that contributed to the disaster that was E3 2006 than anything else! The idea that once the event was over the event was OVER never occurred to anyone outside of the ESA.

The events that finally triggered the avalanche - and half of the blame - is easily placed on the results of the months leading up to E3.

During that period there was a palpable imbalance in The Force (and to be fair that was partly triggered by the combination of economic hardships and certain market segment failures in the games industry over-all) that left the respective gamer communities wielding way too much influence on the industry as a whole.

An obvious consequence of this was that the primary focus for the industry at E3 was:
  • 7th-Gen Consoles - Sony's PS3 and the Nintendo Wii join Microsoft's Xbox 360 as the 7th Gen;
  • 7th-Gen Portables - The Nintendo DS and PlayStation Portable Portables Victory;
  • The New Mobile - Apple's iPod / iPod Touch / iPhone / iPad and Android Invade;
  • 7th-Gen Games - upcoming titles for the 7th-Gen platforms overall.
The industry focus was bass-ackwards! And nobody noticed until it was far too late to do anything about.

It was that skewed focus that lead to some rather unfortunate consequences, the most visible of which was Sony's complete overconfidence that the hardware (PS3) would carry the day. That left Sony largely ill-prepared for the silent reception their press briefing ultimately received, and the humiliation that it transferred to eh ESA.

Sony sailed its PS3 into a massive storm (thanks to its focus on the platforms rather than the games) and when the games were presented, the results were pretty humiliating. It was also prime evidence that they badly and poorly read the market and community.


Now, to be fair, it certainly did not help that the focus of mainstream media was on the pretty obvious and sweeping “win” by Nintendo thanks in no small part to the very impressive game lineup that the Japanese game company put behind both the Wii and DS. Because Nintendo had managed to set the bar so high, Sony's failures appeared to be all the more disasterous.

There was also the safety factor in covering the event that the media felt. If it had been an American company on stage blowing it so badly they might not have chosen to go at them full-throttle, but the issue was being cast as two major Japanese games companies placed at opposite ends of a long rope in a game of tug-o-war that also happened to be easily forced into the roles of David and Goliath.

The media cast Nintendo as David, and Sony as Goliath, and seemed to take unbridled joy in how soundly and roundly David beat the living-heck out of Goliath!

What is worse, the traditional news media was so tightly focused upon the disaster that was unfolding before their cameras and eyes that they totally missed a bigger - and more important - story!

That is how the bloggers and what was then considered to be “new media” (online web-based gamer publications) ended up totally scooping the traditional media in its outing and coverage of the startling inroads Apple and Android were making into the mobile gaming sphere.

New media revealed two very disturbing elements of this new threat to gaming: the measure of control mobile apps were bringing to bear on the market which suggested the threat they posed to the established game pie was far more serious than at first it appeared.

It seemed that, while nobody was looking, a massive paradigm shift was happening - both inside the games industry and, more significantly, in the gamer community.



Amateur game developers and garage-based studios were making increasingly higher quality games for the mobile market, and gamers were noticing, and those same gamers were changing the way that they viewed both their smartphones and touch-devices. What had previously been relegated to a means for making phone calls, tracking schedules, and listening to music was morphing into a pretty powerful games platform.

Then there was the statistical evidence that suggested the market share that these two relatively new game platforms had carved out of the pie was growing at a sustainable rate far greater than the initial projections predicted.

What made this all the worse was that this was happening on a physical platform the control of which was outside the sphere of influence of either the game publishers or the primary games platform makers.

The biggest problem that the established games industry had with this was not that it couldn't create games for the Android and iOS platforms - it could - rather it was the ubiquitous nature of those platforms and the estimates that new media was making suggesting that, due specifically TO the ubiquitous nature of smartphones and tablets they had suddenly become a major threat to the profit margins of publicly traded game companies.

There was every reason to anticipate that they would not only create their own niche in the games market, but that they would erode the position of ALL of the other segments. The reason for that? They were simply too easy to access and use.

The More Things Change

While all of this was going on, meanwhile over at E3 2006 Microsoft was announcing its plans for a new service and capability called “Live Anywhere” that melded the Xbox 360 games console and PC's running Windows Vista into a pretty impressive gaming platform.



This new service was planned to allow multi-player games and game communication between Xbox 360 and the (then unreleased) Windows Vista OS, and despite the major implications that the announcement contained, hardly anyone noticed!

Of course the fact that this announcement at Microsoft's E3 2006 presser would not actually result in anything meaningful until Windows 10 and the Xbox One in 2015 aside, it was still worthy of notice, but the media was busy lambasting Sony and holding Nintendo out to be a hero.

If that was all that there was to the issue, it would not BE an issue, right? Yeah, too right. The problem was not the amateur approach Sony took in its press briefing, or the dominance of Nintendo over all of the other makers at that E3.

The problem was down to the sea of OTHER negative elements that had previously - and were currently being -- uncovered by the media combined with the willingness of political types and special interest groups to twist issues related to the games industry in general, and E3 specifically, in order to blow their own horns.

That is some Hot Coffee!

On the heels of the closing ceremony for the previous year's E3 the news media discovered something that gamers were already well aware of, that being a rather juicy scandal that was perfect for the 90-second sound bites of broadcast journalism, but that had the potential to fill literally pages of a newspaper too.

What are we talking about? Hot Coffee.

Hot Coffee was a piece of abandoned orphan code in the video game Grand Theft Auto: San Andreas in which the player could take a woman back to her apartment and have sex with her. Specifically oral sex among other sex acts - that the player basically controlled and was scored over.

This was a piece of code that was NEVER included in the game, so it was not submitted to the ESRB for consideration when San Andreas went for its rating.

The problem is that while that piece of orphan code was never made part of the game, it was left in the source code FOR the game.



Players of the PC version of San Andreas found it when they were searching through the code - a practice that both the game publisher -- Take-Two Interactive -- and its studio -- Rockstar Games -- had to know was common enough so that the players were very likely to discover that orphan code.

What Rockstar could not predict, however, was that a California state assemblyman would latch onto that issue and use it as a springboard to media attention and a bid for higher office.

Assemblyman Leland Yee (D-San Francisco), did not stop short at simply condemning the presence of the orphan code in the game - he blew it all out of proportion, outright accusing Rockstar of intentionally leaving it in the game, knowing that gamers would find and play it.

He then accused the ESRB of "failing to appropriately rate" the game, implying that they knew of the code when San Andreas received its M-Rating, and suggesting that the board failed to give the game the X-Rating (actually it would have been an AO-Rating, not X, but Yee did not know that) it richly deserved in order to prevent it from being banned from major retailer shelves.

It seemed that Yee believed it was possible that the board would look the other way if the Hot Coffee code had actually been included in the ratings application. Based on his certainty of that sort of action or reaction, he declared that the ESRB had a clear "conflict of interest in rating games" because it was run by the game publishers whose interest would be to sell as many games as possible.



Yee then implied that the federal government should be the one to rate games but, failing that, each state should create its own game-rating board and take over the responsibility. Can you imagine what a nightmare THAT would have created for the gaming community? Yee couldn't.

Basically Yee called into question the very integrity of the voluntary ratings system for video games and worse, seemed to be suggesting that the sex game was left in San Andreas in the hopes that it would be discovered, and thus lead to greater sales than might otherwise have been the case for the game based on the scandal.

ESRB president Patricia Vance responded strongly to Yee and his allegations by announcing that the board would undertake an investigation into the circumstances surrounding the 'Hot Coffee' mod with the goal of determining whether the mod unlocked preexisting code, or was actually a third-party creation.

It turned out that it was in fact orphaned code in the game, and the folks behind the game had taken every link and mention of it out of the game but, unable to remove it completely, the damage was already done by the time the investigation was completed.

In their defense they explained that the code could only be accessed via the PC version of the game, and only then after seriously messed up effort was taken to make that happen. That did not prevent Vance from attacking Yee though, as she rightly pointed out his own ulterior motives for the brew-ha-ha he had practically personally generated over the issue:

"Assemblyman Yee has been on a crusade for years to undermine the integrity of the ESRB, and in so doing, generate support for his legislative agenda," Vance said.

"His latest attempt to win political points is to claim, without any legitimate basis, that a game rated for ages 17 and older, with explicit content descriptors prominently displayed on every box, has been inappropriately rated," she added.

As to the question that the board somehow failed the duty and obligation it owed to gamer parents, Vance allowed that the agenda of the board was above reproach; that she believed that contrary to Yee's claims, the ESRB rating system did its job with the information it had, and further research showed parents overwhelmingly found the ESRB ratings system to not only be a very effective system, but continued to value - and use - it.

"We will do whatever it takes to maintain their confidence in the integrity of the system," Vance declared. That was all well and good - and right - but Hot Coffee appeared to have become one of the nails in E3's coffin.

Booth Baby, Booth Baby, Throw me a Kiss!

The next nail in the proverbial coffin was the issue of Booth Babes at E3.

Surprisingly it was not E3 2006 so much that set this off, but E3 2005 - though the measured response to the new Booth Babe Policy at the 2006 event (or rather the lack of measured response) after the ESA changed the policy certainly was the issue.


To help understand the problems - which were a combination of image and PR - watch the video embedded above and pay close attention at the following times: 0:28 -- 0:35 thru 1:02 -- 1:36 -- 2:35 thru 2:55.

Check out the video below for shots of the dance numbers that drew particularly vitriolic criticism from woman’s rights groups...


After the heat from E3 2005 all of the registered exhibitors for E3 2006 were sent a copy of the rules -- rules that require that all display material -- including live models (meaning booth babes) -- can not include conduct deemed to be sexually explicit and/or sexually provocative, including but not limited to partial nudity and nudity. The rules specifically outlaw bikini bathing suit bottoms and G-strings from the show floor.

Ironically those rules are the same ones that have been in effect since they were amended for E3 2001. They just were not being enforced. For E3 2006 the exhibitors were put on notice that the rules would be enforced.

That would seem to be very clear, yes? Well, despite that clarity, several studios who shall remain nameless chose to outfit their babes in costumes that left rather little to the imagination in terms of the anatomical gifts that God - who clearly was an artist of the highest caliber - had awarded them.

It seems that the letter of the law - but not its spirit - were observed in 2006 - just check out the video from E3 2006 below and see how many rules violations you can spot:


It was the liberal interpretations by the studios and publishers of just what constituted sexually provocative that it seems was the final nail in the coffin for E3. The flaunting of the rules for booth babes was what the board for E3 unofficially cited as the final factor that pushed them into the decision to re-imagine E3 into the what became the E3 Media and Business Summit.

The Dark Ages Descend Upon E3

What had always - up until 2007 - been known by the Cool Kids as “The Show” got a new name after Day 1 of the 2007 E3 Media and Business Summit: Snafu.

It wasn't just the booth babes that were gone from the show either. Most of the attendees were gone as well as the ESA tightened the rules about who could attend so tight that the door count for the entire event barely passed 10,000. And that included the exhibitors!

Let's get some perspective here shall we? Here are the numbers for all of the shows up to and including 2007:
  • 1995: 50,000 / ?** Exhibitors (Location: LA Convention Center)
  • 1996: 50,000* / 400 Exhibitors (Location: LA Convention Center)
  • 1997: 40,000 / 500 Exhibitors (Location: Atlanta at the Georgia Dome)
  • 1998: 73,000 / ?** Exhibitors (Location: Atlanta at the Georgia Dome)
  • 1999: 40,000* / ??? Exhibitors (Location: LA Convention Center)
  • 2000: 45,000 / 450 Exhibitors (Location: LA Convention Center)
  • 2001: 62,000 / 400 Exhibitors (Location: LA Convention Center)
  • 2002: 60,000* / 400 Exhibitors (Location: LA Convention Center)
  • 2003: 60,000 / 400 Exhibitors (Location: LA Convention Center)
  • 2004: 65,000 / 400 Exhibitors (Location: LA Convention Center)
  • 2005: 70,000 / 400 Exhibitors (Location: LA Convention Center)
  • 2006: 60,000 / 400 Exhibitors (Location: LA Convention Center)
  • 2007: 10,000 / 33 Exhibitors (Location: Santa Monica at many locations)
* Estimated Attendee Count.
** No Official Numbers Offered by the ESA.

While most of the industry insiders - and many of the editors who assign journos to cover it - prefer to call it “E3” when the cool kids refer to E3 - and when we say “the cool kids” we are not talking about the American alternative hip hop duo -- they call it by another name.

From 1995 to 2006 they called it “The Show” -- as in, “Jim and I got assigned to The Show this year, are you going to make it?”

In 2007 the name changed to “Snafu” -- as in, “Jim and I tried to get assigned to The Snafu but they denied us!”

Re-named the E3 Media and Business Summit, the 2007 event was moved to a smaller venue in Santa Monica, California, and you could say that it underwhelmed a lot of the attendees, of which there were only 10,000 because the ESA had set a cap and severely restricted who could attend.

That move ended up having consequences nobody could have anticipated including a very negative reaction from the ESA's own members.

2008 E3 Media and Business Summit

Naturally enough, even the Cool Kids were impressed by the measure of success that the ESA attained in making things worse. So much so that they again changed its name yet again - this time to “Tarfun” - which you may want to look up via Google as this being a family-friendly blog we can't actually tell you what it means. Sorry.

But to help you in grasping the impact, here is an example of the proper use of the new name E3 was gifted with for the 2008 show:

“Jim's editor tried to get him credentials to this year's Tarfun, but mine decided not to bother when she found out most of the game studios were not going.”

Yup, you read that correctly. The anger that was generated by this new and not-improved approach that the ESA had taken was not kindly received -- either by its members OR the video game studios and publishers that made the show possible.

In fact they stayed away in such record numbers that, even extending attendance eligibility to family members, the best that they could do for E3MBS 2008 was slightly more than 5K total.



Fortunately the ESA finally understood that the road that they were traveling on was not the road that the rest of the industry wanted to travel on, a response that so angered the ESA board, they threatened to move the event to a storage locker in Compton for the 2009 E3 Media and Business Summit! Okay just kidding!

Thankfully cooler heads prevailed and the Expo was changed back to its pre-2007 format, and the industry decided to let bygones be bygones, and returned to the LA Convention Center with the original format restored, and vowing to relax their enforcement of the rules as long as no serious protests happened.

The fact that a restored E3 2009 somehow managed to pull in over 41,000 attendees despite the show having its smallest number of exhibitors (216) ever for an Electronic Entertainment Expo.

Still 216 was a major improvement over the 33 exhibitors at the 2007 E3 Media and Business Summit, and just 39 at the 2007 E3 Media and Business Summit.

Coming Up...

Coming up as we make our way through the days leading up to the opening bell for E3 2015 including how the Digital Divide applies to gaming, the spectacular disaster that was the Nintendo Digital Event for E3 2015 (aka an hour of our life we want refunded), what is it like to rub elbows (and other body parts) with the famous and infamous at E3, and we may discover why Mickey Mouse wears a Julian Assange watch, but probably not.