Monthly Archives: February 2011

The Documentation Graveyard

by Scott Kantner, February 14th, 2011 in Operations

When asking for documentation, how many times have you heard “I’ll have to dig that up for you?” For all intents and purposes, documentation has only two effective states: either it exists or it does not. A no-brainer statement to be sure, but let’s add a corollary:  Documentation that you can not find or can not access is the same as no documentation at all.

Let’s prove it with a real-world, uber non-IT  example. Suppose you’re the superintendent of a cemetery (I promise this won’t be too macabre), and you’ve just received a call from the funeral home – Mr. A has just died and you need to mark the spot where he is to be buried. The burial company is relying on you to mark the right spot for the backhoe operator. To make this happen you need some documentation, specifically:

Who is being buried? Mr. A, the funeral home told you that.

Does Mr. A have a cemetery lot? This info is contained in a 3-ring notebook you keep on your bookshelf.

In what position on the lot should Mr. A be laid to rest? Lots hold up to 8 people, and you make the determination by looking at your notebook.

Where is the exact location on the cemetery? There is where it gets tricky. You have a map showing where all the lots are located, and from that you can determine the general area of Mr. A’s lot, but you must now locate the exact location in the real world. Because all of the lots immediately adjacent to each other, you must find the right location within 1-2 inches of accuracy, otherwise you could accidentally put Mr. A on someone else’s lot, or worse, on top of Mrs. B. Clearly, not Good Things.

The rest of your documentation is on the cemetery itself. All lots are 16′ square and are marked at the corners with 6″ cornerstones. The top of the stones have the owner’s last name initial carved on the top and are flush with the ground. No sweat, you say, as you drive to the cemetery. But when you arrive you find this:


12″ of pickaxe-class snow and ice, and no cornerstones visible. The last piece of your documentation is missing, and unless you can find it, this mission is a FAIL. Because you’ve been on the cemetery before, you have a clue where the cornerstones (main documentation) might be, but what if you’re in Colorado and you’re trying to talk your second-in-command back in Pennsylvania through this via cell phone?  In either case, the job is going to take much longer because the documentation is not readily available. Moreover, the cornerstones you’re looking for might be buried under the turf as well as the snow, so you may also need to reference cornerstones from adjacent lots (related documentation).

Because you’ve done this before (i.e. disaster recovery exercise), and can piece it together from the partial documentation you have available, you at least have a clue where to start, and our story has a happy ending.

But it doesn’t always work out this way, and so I must ask the uncomfortable question: where is your documentation? Is it buried somewhere, literally? Here are a few questions to ponder:

  1. Have you settled on a system of cornerstones to document your shop?
  2. Are all of your cornerstones in place? Is the documentation written? All of it?
  3. Are your cornerstones visible?  Can they be found in a crisis?
  4. Does your staff know how to find the cornerstones?  Do they know where everything is, and do they have proper credentials to access it?


What kind of system are you using for your own documentation? Please share your secrets below with a comment!

When The Eggs Hit The Pan

by Scott Kantner, February 7th, 2011 in Disaster Recovery

Virtualization is a wonderful thing. It leads to better hardware utilization and less hardware overall. Sounds like goodness, and it is, but could there be a pitfall?

Without careful planning, consolidating physical servers could have the unwanted side effect of consolidating core network services. If you’ve virtualized your infrastructure, you might want to take a quick mental inventory. Do you have physical diversity for the following core services?

  • Directory Services (Active Directory, LDAP)
  • Name Services (DNS)
  • DHCP
  • ACS, RADIUS or other central authentication
  • Documentation

Much like putting all one’s eggs in one basket, inadvertently consolidating all core services into one virtual basket is something best avoided.

I did really mean to list documentation as a core service. Without run-books, passwords, IP addresses and such, recovery may not be impossible, but it will definitely take a lot longer.

Keeping hardcopy documentation current is a constant struggle of questionable worth, since it’s usually out of date before the ink dries, and I’m not suggesting you keep multiple copies, or even one copy on paper unless that’s your preference. But it might be very handy to maintain a copy in the cloud of your choice, perhaps Dropbox, SkyDrive, or a similar service. The beauty of putting documentation “up high” in a cloud is that you can see it from anywhere you happen to be – in your primary data center, or in the Homewood Suites during disaster recovery.

And of course, it’s best to do all this before an egg gets fried.