Avoiding Guesswork in Complex Environments
Published by Alex Gorbachev March 24th, 2008 in UncategorizedTags: complexity, documentation, knowledge, no guesswork.
One of the main reasons why it’s difficult to troubleshoot problems in complex environments is lack of documentation. This post is long overdue because it was inspired a while ago by Dan Fink’s post (again). Be sure to read it first.
From time to time, I see situations when one person leaving the company leads to a disaster that puts in danger the whole existence of multimillion businesses. Why? Because there is nobody else who knows how system works and how all its moving parts are connected. How does that happen? No knowledge sharing.
The most universal method of knowledge transfer is documenting stuff and I would argue that, most of the times, one hour of documenting brings to a business more than two hours developing new functionality in PL/SQL.
Actually, there might not even exist such know-it-all-wizard. Complex systems are not created in a matter of days or even months. Often, extreme complexity is the result of years or even decades of evolution. All those workarounds, bug fixes, re-designs, pressing deadlines — those friends of entropy win time from documentation tasks.
How many times it happened to you:
- Why are there forty-two underscore parameters in every new database you create?
- Because it’s our standard.
- Can I look at it?
- {proudly} Yes. It’s in every init.ora file we have!
Or another variation:
…
- Can I look at it?
- {proudly} Yes. Here is 42 pages document how to create new database:
Performance boosters:
_db_aging_touch_time=…
_db_aging_stay_count=…
_db_writer_chunk_writes=…
_discrete_transactions_enabled=…Bugs:
_kgl_latch_count=…
_kgl_bucket_count=…
_lm_rcfg_timeout=…and etc…
- When was this document started?
- {proudly} Oh! Forty-two years ago!
- When was it last time updated?
- Eh… I’m actually quite new here…
What are the reasons behind missing or inadequate documentation?
Just to name a few:
- It’s boring.
- We have no time for it now and it’s not a priority (it never will be) — we still missing couple reports.
- I’ll do it tomorrow.
- Project manager is not paid by number of pages of documentation produced (not that this is a good idea either).
- I’m the only one who knows this system — they must be idiots to get rid of me.
I’m sure readers can name plenty of others and please feel free to do so.
Sometimes you can avoid complexity and make systems simple by design but in real life it’s more often when you need to deal with complexity. Paraphrasing Dan — documentation will not make systems simpler but will reduce perceived complexity.
If you’ve got to support and/or develop a complex system — how do you make sure you dedicate enough time to documentation? What are the strategies to keep it up to date? How to explain the importance of documenting to the management? If you have anything to say — the comment box is at the bottom of this screen — use it. If you think that you have enough to say for the blog post — just let me know.
Find more about complexity documentation knowledge no guesswork
Amen!
Good points Alex! Exactly the last situation I was partly involved with.
In my experience, setting up a successful system, either complex or simple, requires a systematic approach from the start. And, which cannot be said enough, simplicity!
A database needs to be adjusted as little as possible. Sourcecode needs to be version controlled, Even in very small simple environments. Production environments need software releases, explicitly tagged from the version control. Not a few files to be fiddled with in order to make it work again.
Happy easter!
Alex,
Nice thoughts.
I agree completely.
Your last reason is very important. People, many times, do not share knowledge, due to insecurity. Sounds funny, but it is a fact. They (probably) get “married” to the company and whenever they are acknowledged as S.P.O.C. (Single Point-Of-Contact), it soothes their ego.
Such people will do everything they can, to sabotage any plans to prepare / update documents.
Documentation is very important and under-used in today’s IT shops.
To help remedy that, here’s my recent post on system administrator site documentation with docbook, and a few “intro to docbook” links that were helpful to me.
Good point. Thanks Aleksey.
The biggest difficulty I found is to motivate engineers to write docs AND keep them up to date.
Excellent points Alex.
I would also add that writing the documentation is only useful if you ensure it’s properly distributed or can be easily found. Finding the correct location for the documentation and ensuring that it’s organised in such a way to make retrieval easy is not as simple as it could be in a large company(at least that’s what I’m observing where I work atm).
Thanks Graham. I might be biased or just used to Oracle Database documentation style but after comparing Oracle database documentation with documentation for the products that Oracle acquired — I can only attest to your point. The ability to navigate quickly and effectively within a large documentation volumes is a must.
But it’s not only the documents creators who should invest efforts, readers must invest enough to be familiar with organization and knowledge domain. Even when search functionality works great, you would still need to know what to search for.
Thanks again for the feedback.