by Scott Kantner, October 15th, 2009 in Human Factors
Remember classic puzzles like “What’s wrong with this picture?” from Highlights Magazine? Tell me, what’s wrong with this picture:
Yes that’s right, clowns and coffee don’t go together, particularly this clown. We seem to know that instinctively, yet the blogosphere reports that we are apparently ignoring sound instinct in the name of price. The Clown is cheaper than Starbucks, and so we are enticed away from something of known, predictable quality to something, shall we say, less predictable.
McTreachery
Despite being a loyal Starbucks customer, last week I too swung by the local golden arches for a McLatte. Here’s how it went down at the drive-up’s McSpeaker box (imagine Charlie Brown’s teacher’s voice as you read the McSpeaker’s part):
McSpeaker: Hello, may I take your order?
Me: Yes, I’d like a medium cafe latte with skim milk.
McSpeaker: Would you like whole or non-fat milk?
Me: Non-fat please.
McSpeaker: Hot or Cold?
Me: Hot.
McSpeaker: That will be $2.39. Please pull around.
So far so good, despite my deliberate avoidance of the word “McLatte” or my faux pas of asking for skim rather than non-fat McMilk. I pull around to window #1, pay and receive my change, and proceed to window #2. At window #2 I wait an eternity for the window to open. Eventually a young, ponderously pierced McDude opens the window and presents me with a see-through plastic cup holding a milky substance with ice cubes in it. A new dialogue ensues:
Me: I’m sorry, this should have been made as a hot latte. (said very politely)
McDude: Uh…really?
Me: Yes. (The petulant McDude now checks the overhead order display for confirmation.)
McDude: Uh…OK…one minute.
As I settle in for another eternal wait, my server surprisingly appears in less than 20 seconds with the proper looking drink: a brown paper McCup with a black plastic McLid. I drive away, yet something seems wrong. The cup should be warm even though there’s a McSleeve to prevent me from suing them over a burned hand. One sip, and I realized I’ve just been had. The devious McDude has poured the cold latte into a different cup and simply ditched the ice cubes. Surely there must be something in the Geneva Conventions about messing with a person’s morning caffeine fix. Where is Jackie Chiles when you need him?
So, disgusted but realizing I should have known better, I pointed my truck toward Starbucks. I paid a little more, but got exactly what I asked for with exactly the quality I was expecting.
The McLesson
Clearly, coffee is not Ronnie Mac’s forté. This was not my first failed attempt at getting a fancy coffee McDrink though I’ve tried on multiple occasions. Each and every time they have either botched it badly or been visibly irritated to have to break their burger making ritual in order to do obeisance at the latte machine. They pretty much bat 1.000 on the burgers though, because that’s what they’re really good at.
The lesson? Stick to what you do well and let the rest to somebody else, because you can’t do everything well. Not even if you have the deep pockets of Ronnie Mac. Industry type doesn’t matter either, as history shows us that Novell made this same mistake in 1994 when they bought WordPerfect. They strayed away from what they did best at the time (file and print sharing), got into applications, and everything went downhill from there. Their slow descent into mediocrity is well chronicled, but it all started when they took their eyes off of what they were really good at. Tragically, they are not alone.
R.I.P. Wordperfect. We knew thee well.
McApplication
There are only so many things you or your business can do with excellence – probably less than four – and usually only one in which you can truly excel. If, like the vast majority of businesses today, yours is a consumer of IT infrastructure rather than a provider, it will quite naturally be difficult and more expensive for you to try to deliver IT as effectively as a professional provider can. There is simply too much to know. It may even be a major annoyance like the McLatte machine. Why? Because it’s not your sweet spot. Your strengths will suffer while you’re focusing on things best delegated to others.
Servers, storage, networking infrastructure continue to evolve into increasingly more complex creatures. Unless IT is the sweet spot of your business, it doesn’t make sense to try to keep chasing infrastructure on your own, for much the same reasons you don’t keep factory-trained mechanics on staff to fix company cars. It’s too much, too costly and certainly not worth it. IT is no different. You absolutely need to have technology in your business, but there is no reason to bear the burden of it yourself.
Like Starbucks, professional IT providers may cost a fistful of dollars or just a few dollars more, but the results are reliable, predictable, and therefore very much worth it because it frees you to focus on what you do best. Just make sure your IT provider doesn’t start selling coffee.
Why the movie references to Fistful of Dollars and A Few Dollars More? Well it turns out that Clint Eastwood’s most famous line came after a bad cup of Joe. If gun violence offends you, please don’t go here, otherwise…take a three minute action coffee break with Dirty Harry.
Warning, this is a long post on a controversial subject. I’d recommend you refill your coffee cup before diving in.
We get a lot of questions about “Green IT.” Is your data center Green? What’s your Green strategy? What do you think of Cap and Trade? And so on. With all the bacteria in the air about global warming and the associated hyper ventilation going on in the media, it’s becoming difficult not to catch the disease and lose perspective. Do we have really have to add “being Green” to our list of worries in the data center? Little did I realize a favorite childhood classic would be a relevant stress reliever so many years later.
Do you like green eggs and ham?
There are many shades of green. I was reminded of that one day when my Dad asked me whether I was waiting for the traffic signal to turn avocado before I was going to pull out. But in terms of going Green in the data center there several flavors, such as reducing so called greenhouse gas emissions, buying RoHS compliant products, recycling old assets properly, etc. Since power for equipment and cooling is the most critical resource for IT, the Greenhouse Gas Police are our primary concern. Over on SearchDataCenter.com, we find this in a piece entitled “Get Ready For A Carbon Tax:”
Today, the U.S. government and other countries are taking carbon emissions seriously. The Environmental Protection Agency last week formally declared carbon dioxide and five other greenhouse gases to be harmful air pollutants. It’s a move that is thought to set the stage for a carbon tax of some kind.[emphasis added]
Kai Reichardt, data center manager for UniCredit Group, an Italian bank, said his company recently built a data center over a canal [raucous laughter added] so it could use free water cooling instead of expending energy for mechanical cooling. And he said the company will also investigate other opportunities to save.
“You have to define rules and implement them,” he said. “You have to punish the polluters and push the innovators.” [more emphasis added]
Well isn’t that rich? You, Mr. Data Center owner, are a polluter that needs to be punished. Whenever the discussion turns from the issue itself to name calling, you know that rational discussion has ended. Let’s assume the title of Evil Polluter for a moment and consider how we might mend our ways.
Would you like them with your server? I would not like them with my server.
Servers are the first potential power hogs that come to mind. To Green our servers quickly, we have to replace or virtualize them. IBM claims their new x3650 M2 class Intel servers boast $100/year savings on power. Yippee skippee. Most companies are not going to be excited about spending thousands on a new server just to save $100 annually on the electric bill. But it adds up in a big data center, you say. Well yes, but if you can show me a budget in this economy that can swap out gear in numbers large enough to make these power savings seem attractive, I’ll show you a budget where that savings is round-off error. $100/yr by itself is not a compelling story, given the capital expense and upheaval to our operations that always comes with putting new boxes in place. The sensible solution here would be to buy Greener hardware as old servers naturally come to end of life. But we’d be doing that anyway without consciously trying to be Green. New hardware from IBM, for example, becomes more energy efficient over time without us asking for it.
If we virtualize, we still have to spend money on product and labor to make it happen, and then we have service outages to incur, end-user politics to negotiate, and all the risks inherent in shutting down and moving healthy systems. Measuring the hard dollar savings of this maneuver from a Green perspective is like trying to weigh a chicken with a yardstick. Virtualization is a Very Good Thing, but using Green alone as the cost justification certainly isn’t going to cut it. Justification is going to found in reduced costs realized from hardware consolidation, new hardware avoidance, and to some degree software license reductions, among other things.
So unless Green means something substantial to the bottom line, who’s actually going to be interested?
Would you like them here or there? I would not like them here or there, I would not like them anywhere.
We could also try to replace or virtualize storage to save power. Kantner’s General Theory of Storage states that:
The rate of storage growth is inversely proportional to the amount of free storage available.
In other words, we’re always going to need more storage at the most inopportune times. Once we advance beyond spinning platters for storage (e.g. SSD), perhaps storage will become more power efficient. But regardless, storage virtualization leads only to a deferral of additional storage purchases through better space utilization, not a reduction of powered up hardware. Ultimately, storage purchases are justified by business requirements, not by an appeal to better power efficiency.
What about network gear? Because of the business impact of disruptions that can occur when swapping out major network components, no one is going to dive into that pool until it’s absolutely necessary. Those big old Cisco 6500 power supplies are going to continue to glow like the sun.
Green really needs to mean something more to the bottom line.
Would you, could you, in your data center? I would not, could not in my data center.
Short of taking a major outage or starting over, what practical facility changes can one realistically expect to cost justify? Few can accord to rip/replace chiller plants, UPS systems, or generators with more efficient units. That said, we are in fact looking at ways to shut our chiller plant down during the winter and just run off the cooling tower loop, but we’re doing that to reduce costs – Green is not the driver, but rather deregulation of the electrical utility.
Sure there are some practical things we can do, but extreme measures are hard to justify. We hear of folks running their cold aisles at temps over 90°F. Anyone smell silicon burning? (Incidentally, we’ve seen equipment get toasted. We like 72° at the equipment inlet for good reason, and the Uptime Institute agrees.)
But let’s get back to our alleged title of Polluters. Here at DSS we don’t have smoke stacks towering out of the data center. Chances are, neither do you. Why? Because the vast majority of us obviously don’t generate our own power. We outsourced that to the electric utility industry a long time ago. Certainly we can turn things off and use less, but the generator plant down the street is not going to run any less and will still make power the same way. And if that way is anything other than nuclear fission, there are still nitrogen oxides, sulfur oxides, dust, and carbon dioxide wafting into the atmosphere. Greening our data centers isn’t going to change that. Clearly there is something else driving the Green monster.
I do so like green eggs and ham, Thank you, Uncle Sam-I-Am!
Business cares about the bottom line, not greenhouse gas. Uncle Sam-I-Am understands this. Could it be that the clarion call to save the planet is rooted in the discovery of green ham? My hunch is that Uncle Sam is all about the green ham, namely the cold hard cash that will come out of Cap and Trade, or any other similar program. And we get the green eggs of trying to minimize our financial exposure to those taxes…err…programs. Cap and Trade, boiled down to it’s essence, is about government revenue, and it looks like Uncle Sam has done his homework. My bet is that Archie Bunker would smell a rat too. (If you’re a Dem, please don’t be offended – the title should really be “Archie Bunker on Government”).
As major consumers of power, those of us with data centers are squarely in the cross-hairs. Faced with confiscatory financial punishment, we suddenly have an interest in global warming, whether it’s reality or not. If the revenue from Cap and Trade were only intended to replace legacy power plants with nuclear or other clean power, then the idea would be more palatable, but as they’re designed now, these plans look like just more sources of pork to be spent as Uncle Sam sees fit. One need look no further than the UK for confirmation.
You do not like them so you say. Try them! Try them! And you may.
While I’m not alone in my contrary view on Green, the IT jury is still out on what Greening the Data Center really means from a practical standpoint. But let me end with some positive suggestions for both camps. For those truly concerned about going Green, consider mothballing your server room or data center and move your gear to a professional hosting facility. Take your shop off the grid and put it in a cloud somewhere that can, because of scale, do it with more power efficiency that you would be able to achieve on your own. If global warming theory rings true for you, this should have tremendous and obvious appeal. If it doesn’t, and you just want to avoid the hassles of Cap and Trade, you too might consider moving your gear to a professional data center. Let us Evil Polluters smell the sulfur of green eggs and send the ham to Uncle Sam.
Since only Robinson Crusoe had the luxury of getting everything done by Friday, the rest of us have to come up with other strategies to get all of the things done necessary to properly serve our customers. To help with this in our own data center, we’ve have a pseudo-tradition called Infrastructure Friday. This is not to be confused with Redneck Tuesday:
On Infrastructure Fridays, members of the data center team who normally don’t work out on the floor put down email, IM and PDAs, roll up their sleeves, and step into the data center to help get some of the “real” work done. To keep IT running smoothly, we sooner or later have to stop talking about it and actually go and do something about it. That “something” we do includes taking care of ongoing operational details and implementing new functionality that maintain or improve reliability.
Like many other things, excellent performance in the data center is all about execution and details. Focus on the details and the big picture will take care of itself, or as Mel Gibson advised his young son in The Patriot, “Aim small, miss small.” What sort of details are we talking about on Infrastructure Friday?
Not just performing rack inspections, but actually correcting any problems found.
Not just noting network latency issues, but getting the right people involved to isolate and resolve them.
Not just checking that critical monitoring systems in the NOC are healthy, but verifying they are actually working by simulating failures.
Not just verifying that operational documentation is current and complete, but actually updating it if it’s not.
Not just checking parts inventories (patch cables, cable management supplies, etc), but placing the orders to replenish supplies.
Not just validating that data center standards are being followed (equipment mounted for proper air flow, floor tile placement, etc) , but actually correcting violations.
Not just noting that wire management is shoddy, but actually making it better.
Not just complaining that critical patch cables aren’t labeled, but actually getting out the label machine and doing the labeling.
Not just finding hot spots in the electrical system, but scheduling the downtimerequired to avert a future disaster.
Hopefully the theme is obvious. On Infrastructure Friday, the goal isn’t to grouse about problems, it’s to fix them.
On a happier note, what sort of cool new functionality might we install on Infrastructure Fridays to improve reliability? That’s a shorter list probably not worthy of a set of bullets, but it typically involves installing new or upgraded monitoring capabilities in the NOC, adding additional monitoring instrumentation out on the floor, improving the quality and types of information on the master dashboards, and continuing to implement automated processes to lessen the chance of unplanned downtime. But again the theme is the same: take action.
In the day-to-day blur of activity required to keep a live data center running, the Oughta List of things (we ought to do this, we ought to do that) that would improve reliability grows week by week, but never seem to get done because of the tryanny of the urgent. We find ourselves officially declared Too Busy to work on the Oughta List and before we know it, an outage occurs and the Oughta List suddenly becomes an embarrassing Shoulda List.
Infrastructure Friday is designed to overcome Oughta List inertia. With a “try me” cost of zero, it has pretty good ROI.
//spk
Page 1 of 1
About DSS
DSS is a company that exemplifies the innovative spirit, delivering Data Center solutions that drive business value. From our Tier III Data Center to our consulting and support that DSS provides, we continually strive to produce high-quality, beyond standard solutions to exceed the expectations of our customers.
