Tag Archives: documentation

The Documentation Graveyard

by Scott Kantner, February 14th, 2011 in Operations

When asking for documentation, how many times have you heard “I’ll have to dig that up for you?” For all intents and purposes, documentation has only two effective states: either it exists or it does not. A no-brainer statement to be sure, but let’s add a corollary:  Documentation that you can not find or can not access is the same as no documentation at all.

Let’s prove it with a real-world, uber non-IT  example. Suppose you’re the superintendent of a cemetery (I promise this won’t be too macabre), and you’ve just received a call from the funeral home – Mr. A has just died and you need to mark the spot where he is to be buried. The burial company is relying on you to mark the right spot for the backhoe operator. To make this happen you need some documentation, specifically:

Who is being buried? Mr. A, the funeral home told you that.

Does Mr. A have a cemetery lot? This info is contained in a 3-ring notebook you keep on your bookshelf.

In what position on the lot should Mr. A be laid to rest? Lots hold up to 8 people, and you make the determination by looking at your notebook.

Where is the exact location on the cemetery? There is where it gets tricky. You have a map showing where all the lots are located, and from that you can determine the general area of Mr. A’s lot, but you must now locate the exact location in the real world. Because all of the lots immediately adjacent to each other, you must find the right location within 1-2 inches of accuracy, otherwise you could accidentally put Mr. A on someone else’s lot, or worse, on top of Mrs. B. Clearly, not Good Things.

The rest of your documentation is on the cemetery itself. All lots are 16′ square and are marked at the corners with 6″ cornerstones. The top of the stones have the owner’s last name initial carved on the top and are flush with the ground. No sweat, you say, as you drive to the cemetery. But when you arrive you find this:

Snowbound

12″ of pickaxe-class snow and ice, and no cornerstones visible. The last piece of your documentation is missing, and unless you can find it, this mission is a FAIL. Because you’ve been on the cemetery before, you have a clue where the cornerstones (main documentation) might be, but what if you’re in Colorado and you’re trying to talk your second-in-command back in Pennsylvania through this via cell phone?  In either case, the job is going to take much longer because the documentation is not readily available. Moreover, the cornerstones you’re looking for might be buried under the turf as well as the snow, so you may also need to reference cornerstones from adjacent lots (related documentation).

Because you’ve done this before (i.e. disaster recovery exercise), and can piece it together from the partial documentation you have available, you at least have a clue where to start, and our story has a happy ending.

But it doesn’t always work out this way, and so I must ask the uncomfortable question: where is your documentation? Is it buried somewhere, literally? Here are a few questions to ponder:

  1. Have you settled on a system of cornerstones to document your shop?
  2. Are all of your cornerstones in place? Is the documentation written? All of it?
  3. Are your cornerstones visible?  Can they be found in a crisis?
  4. Does your staff know how to find the cornerstones?  Do they know where everything is, and do they have proper credentials to access it?

 

What kind of system are you using for your own documentation? Please share your secrets below with a comment!

When The Eggs Hit The Pan

by Scott Kantner, February 7th, 2011 in Disaster Recovery

Virtualization is a wonderful thing. It leads to better hardware utilization and less hardware overall. Sounds like goodness, and it is, but could there be a pitfall?

Without careful planning, consolidating physical servers could have the unwanted side effect of consolidating core network services. If you’ve virtualized your infrastructure, you might want to take a quick mental inventory. Do you have physical diversity for the following core services?

  • Directory Services (Active Directory, LDAP)
  • Name Services (DNS)
  • DHCP
  • ACS, RADIUS or other central authentication
  • Documentation

Much like putting all one’s eggs in one basket, inadvertently consolidating all core services into one virtual basket is something best avoided.

I did really mean to list documentation as a core service. Without run-books, passwords, IP addresses and such, recovery may not be impossible, but it will definitely take a lot longer.

Keeping hardcopy documentation current is a constant struggle of questionable worth, since it’s usually out of date before the ink dries, and I’m not suggesting you keep multiple copies, or even one copy on paper unless that’s your preference. But it might be very handy to maintain a copy in the cloud of your choice, perhaps Dropbox, SkyDrive, or a similar service. The beauty of putting documentation “up high” in a cloud is that you can see it from anywhere you happen to be – in your primary data center, or in the Homewood Suites during disaster recovery.

And of course, it’s best to do all this before an egg gets fried.

//spk

Personnel DR

by Scott Kantner, July 10th, 2009 in Disaster Recovery, Hosting

Are you prepared for the departure of a key technical resource in your operation?  Someone who holds the infamous “keys to the kingdom?”   Typically there is a least one person on a company’s IT staff who achieves deity status in regards to physical and logical access. Sometimes key skills also reside in just that one person. If such a person leaves, either voluntarily or involuntarily, how would your critical operations fare?

vista_help_icon_by_thoosje

Now would be a good time to take a fresh look at both your internal documentation and your skills matrix. Things to consider:

  1. Are all sysadmin userids and passwords documented somewhere, somehow?
  2. Are all critical architectures documented in excruciating detail?  (SAN, virtualization, LAN/WAN, disk replication, backup/restore systems). You want to see how things are connected and how they are intended to interact. You want to see things like IP addresses, subnet addressing schemes, WWN numbers, hard and soft zoning information and the like. You’ll know you have all of the information you need when you can hand it to a new engineer and he doesn’t have any questions. Seem impossible? Strive for it, and the result will be good enough.
  3. Where does the above documentation live if you do already have it?  Hopefully it’s not on your staff’s laptops.  If you think you already have it in a shared on-line space, are you sure you have all of it?  And is it being backed up?
  4. Do you have runbooks for all of your servers?  Are they current?  Where are they? Are they backed up?
  5. How many people have practical working knowledge in each area of your critical infrastructure?  Do you have more than one VMWare tech?  More than one SAN person?  How about Active Directory or Exchange? Ideally you’ll want three in each area. Contract for it if you need to.

 
I could go on, but I think you’re getting my point. This process is somewhat like writing a will.  It’s a real drag to write up, and everybody knows that they need to take care it, but yet it often gets ignored until it’s too late. And just like a will, all of this documentation needs to be updated on a regular basis or it may end up being worthless at crisis time.

Alternatively, you could move the responsibility for a large portion of this to a professional hosting facility.   Why not limit your exposure to just your applications and let us worry about how all the  plumbing is hooked up?

//spk