One of the main reasons why it’s difficult to troubleshoot problems in complex environments is lack of documentation. This post is long overdue because it was inspired a while ago by Dan Fink’s post (again). Be sure to read it first.

From time to time, I see situations when one person leaving the company leads to a disaster that puts in danger the whole existence of multimillion businesses. Why? Because there is nobody else who knows how system works and how all its moving parts are connected. How does that happen? No knowledge sharing.

The most universal method of knowledge transfer is documenting stuff and I would argue that, most of the times, one hour of documenting brings to a business more than two hours developing new functionality in PL/SQL.

Actually, there might not even exist such know-it-all-wizard. Complex systems are not created in a matter of days or even months. Often, extreme complexity is the result of years or even decades of evolution. All those workarounds, bug fixes, re-designs, pressing deadlines — those friends of entropy win time from documentation tasks.

How many times it happened to you:
- Why are there forty-two underscore parameters in every new database you create?
- Because it’s our standard.
- Can I look at it?
- {proudly} Yes. It’s in every init.ora file we have!

Or another variation:

- Can I look at it?
- {proudly} Yes. Here is 42 pages document how to create new database:

Performance boosters:
_db_aging_touch_time=…
_db_aging_stay_count=…
_db_writer_chunk_writes=…
_discrete_transactions_enabled=…

Bugs:
_kgl_latch_count=…
_kgl_bucket_count=…
_lm_rcfg_timeout=…

and etc…

- When was this document started?
- {proudly} Oh! Forty-two years ago!
- When was it last time updated?
- Eh… I’m actually quite new here…

What are the reasons behind missing or inadequate documentation?
Just to name a few:
- It’s boring.
- We have no time for it now and it’s not a priority (it never will be) — we still missing couple reports.
- I’ll do it tomorrow.
- Project manager is not paid by number of pages of documentation produced (not that this is a good idea either).
- I’m the only one who knows this system — they must be idiots to get rid of me.
I’m sure readers can name plenty of others and please feel free to do so.

Sometimes you can avoid complexity and make systems simple by design but in real life it’s more often when you need to deal with complexity. Paraphrasing Dan — documentation will not make systems simpler but will reduce perceived complexity.

If you’ve got to support and/or develop a complex system — how do you make sure you dedicate enough time to documentation? What are the strategies to keep it up to date? How to explain the importance of documenting to the management? If you have anything to say — the comment box is at the bottom of this screen — use it. If you think that you have enough to say for the blog post — just let me know.

Find more about  , , ,

7 Responses to “Avoiding Guesswork in Complex Environments”  

  1. 1 Christian Antognini

    Amen!

  2. 2 Frits Hoogland

    Good points Alex! Exactly the last situation I was partly involved with.

    In my experience, setting up a successful system, either complex or simple, requires a systematic approach from the start. And, which cannot be said enough, simplicity!

    A database needs to be adjusted as little as possible. Sourcecode needs to be version controlled, Even in very small simple environments. Production environments need software releases, explicitly tagged from the version control. Not a few files to be fiddled with in order to make it work again.

    Happy easter!

  3. 3 Narendra

    Alex,

    Nice thoughts.
    I agree completely.
    Your last reason is very important. People, many times, do not share knowledge, due to insecurity. Sounds funny, but it is a fact. They (probably) get “married” to the company and whenever they are acknowledged as S.P.O.C. (Single Point-Of-Contact), it soothes their ego.
    Such people will do everything they can, to sabotage any plans to prepare / update documents.

  4. 4 Aleksey Tsalolikhin

    Documentation is very important and under-used in today’s IT shops.

    To help remedy that, here’s my recent post on system administrator site documentation with docbook, and a few “intro to docbook” links that were helpful to me.

  5. 5 Alex Gorbachev

    Good point. Thanks Aleksey.
    The biggest difficulty I found is to motivate engineers to write docs AND keep them up to date.

  6. 6 Graham Oakes

    Excellent points Alex.

    I would also add that writing the documentation is only useful if you ensure it’s properly distributed or can be easily found. Finding the correct location for the documentation and ensuring that it’s organised in such a way to make retrieval easy is not as simple as it could be in a large company(at least that’s what I’m observing where I work atm).

  7. 7 Alex Gorbachev

    Thanks Graham. I might be biased or just used to Oracle Database documentation style but after comparing Oracle database documentation with documentation for the products that Oracle acquired — I can only attest to your point. The ability to navigate quickly and effectively within a large documentation volumes is a must.

    But it’s not only the documents creators who should invest efforts, readers must invest enough to be familiar with organization and knowledge domain. Even when search functionality works great, you would still need to know what to search for.

    Thanks again for the feedback.

Leave a Reply


  • does lower blood pressure
  • safest ppi
  • can drink caffeine while taking
  • brand name buy online
  • a prendre quand
  • adverse reactions
  • skin itch cream
  • how long to wait for surgery after
  • when does it start working
  • withdrawal gastrointestinal
  • 75 asian flush
  • buzz
  • good results
  • bula medicamento
  • sexual side effects women
  • buy non prescription online
  • gram positive rods
  • energy jobs
  • and penicillin together
  • taking first time
  • journal week 1
  • daily
  • muscle pain side effects
  • buy online
  • anyone bought online
  • in early pregnancy
  • does make you feel high
  • affecting menstrual cycle
  • urine tests
  • 5 20 side effects
  • delirium
  • safe take entire pregnancy
  • 5 mg
  • 500 mg posologia
  • anticholinergic
  • can you take with
  • get rid spam emails
  • turisti per caso bella vacanza
  • kopen
  • er serum levels
  • show up in a drug test
  • should namenda taken together
  • cost of vs pradaxa
  • success with irregular periods
  • can taken empty stomach
  • pregnancy statistics
  • more than once a day
  • how does work for anxiety
  • golf redmond
  • or during cycle
  • how to take medication
  • for tapeworms
  • a wit c
  • used epstein barr
  • dental use
  • official site
  • leg muscle pain weakness
  • there substitute
  • hct recall
  • alcohol drink
  • how do i wean off 30 mg
  • side effects of 150mg
  • otc in us
  • mania symptoms
  • vs. vs.
  • and cancer
  • is generic safe and effective
  • side effects spotting
  • xtc
  • cranberry interactions
  • quem engravidou no 1 ciclo
  • dose 5 year old
  • cagliari voli low cost
  • benefits of stopping
  • pregnancy drug class
  • n og nn
  • pulse pack
  • many milligrams does come
  • o
  • how long to get out of system
  • pharmacy 2u
  • when to take for men
  • swollen feet
  • 2 years
  • for parkinson's disease
  • to dose equivalence
  • does treat prostate infections
  • price in kerala
  • good cat allergies
  • can you take and together
  • and fet
  • withdrawal symptoms diarrhea
  • spiagge foto
  • mizzoni
  • kidney damage
  • for baby side effects
  • allergic reaction
  • heart issues
  • liver toxicity
  • dolo capsulas
  • convert to
  • cheap and
  • can you take acetaminophen while
  • generic vs
  • and pregnancy third trimester
  • expiry date
  • fda approval
  • and
  • compare
  • and iud insertion
  • jr tablets
  • interactions
  • when will i see results
  • 1 g\/125
  • triangle gold coast
  • guatemala precio
  • 2 comprimidos
  • instructions taking
  • pentasa versus
  • what's better or
  • children caffeine
  • cancer prevention
  • effects
  • cost australia
  • nerve pain
  • other names
  • dosage seizures
  • can i take gaviscon with
  • pain management
  • z pak d
  • tablets and pregnancy
  • post nasal drip
  • en francais
  • can cause stomach pain infants
  • dermatologist houston
  • para que serve a medicaAA?o
  • cvs d generic
  • ointment used cold sores
  • to treat nausea
  • how long does take to lose weight
  • side effects of 40 mg tablets
  • cream
  • birth control canada
  • prolong 150
  • new warnings
  • can be crushed
  • ask a crack hoe
  • can you take while breastfeeding
  • list of foods you can't eat on
  • safe during pregnancy
  • stopping suddenly
  • alternative
  • skin care
  • 5mg dosage
  • commercial bicycle
  • for male infertility
  • side effects flushing
  • can take acetaminophen
  • hcl vs
  • rite aid
  • side effects fever
  • mode of action
  • clinical trials
  • together side effects
  • and ed
  • use during outbreak
  • side effects of taking without ed
  • get you high
  • rite aid
  • para que sirve el
  • guaifenesin interaction
  • back pain cause
  • ultram
  • therapy side effects
  • armour conversion chart
  • to treat ulcers
  • hypertension
  • tardive dyskinesia td
  • hfa 200 metered inhalations
  • levo mechanism of action
  • cold medicine
  • tratamentul cu la copii
  • rash pictures
  • what happens if you stop taking suddenly
  • legal purchase
  • pills
  • d ready tabs
  • strep coverage
  • children recall
  • o algo similar
  • use menopause
  • package insert stabilizer
  • pcp treatment
  • prolong 50 mg vademecum
  • interactions medications
  • despues de tomar cuanto tiempo dura el sangrado
  • does increase your blood pressure
  • is a controlled substance
  • tudo sobre o remedio
  • can you switch from to
  • period pain but no period
  • e anticoncepcional
  • changing from to
  • during breastfeeding
  • diarrhea
  • buy online no prescription canada
  • taking with
  • over counter strength
  • 150 mg for yeast infection
  • swelling
  • can i exercise while taking
  • can d raise your blood pressure
  • over the counter us
  • zoloft vs sexual side effects
  • how long to lose weight after stopping
  • versus tylenol kids
  • mode of action
  • recreational
  • z-pak dosing
  • after taking 5 years
  • o que significa bd no
  • tired fatigued
  • 90 mg biverkningar
  • ear congestion
  • asma embarazo
  • pink for women
  • risks
  • safe dose
  • rapid 12.5 directions
  • list foods vitamin k
  • generic problems
  • gmail hack
  • duration after bare metal stent
  • y muerte subita
  • cvs caremark prior authorization
  • there black box warning
  • cabbage vitamin k
  • c quoi du
  • quanto custa o medicamento
  • common severe side effects
  • quanto tempo
  • can take
  • gel neck
  • long can left out fridge
  • stops pulmonary edema
  • generic cheap
  • capsulas para que sirve
  • using tylenol and together
  • safe take daily
  • side effects fainting
  • and rsd crps
  • what is highest dose of for infants
  • umc clinic las vegas
  • burning scalp
  • price india
  • obat sr 75 mg
  • hct free trial
  • efek pemakaian
  • weaning off success stories
  • how to take after a steroid cycle
  • dose for trenbolone
  • mycoplasma hominis
  • failure rate
  • can't sleep
  • stopping hair loss
  • interaction
  • litigation
  • induced lupus
  • xr how supplied
  • heat exhaustion
  • schedule drug
  • y prurito
  • starting dose
  • g-tube
  • side effects
  • stevens johnson syndrome
  • z pak #1
  • using after expiration
  • used cystitis
  • verkningstid
  • does cause weight loss
  • alcohol interaction
  • 40 mg side effects
  • lp vidal
  • dosage in pregnancy
  • std
  • thrush oral
  • hemodialysis dosing
  • weight gain
  • blood in urine after taking
  • single dose of for gonorrhea
  • 100 mg
  • versus
  • greece
  • eye problems
  • substitute for hct
  • y omifin
  • for skin rashes
  • allergy hives
  • tosh.o gay challenge
  • results after 3 weeks
  • what is it
  • 12h jarabe
  • para que sirve el 300mg
  • ovulos de nistatina
  • 500mg tab
  • taper off 20 mg
  • basal cell
  • no prescription
  • gel headaches
  • side effects of for women
  • civil suit
  • inhaler kids
  • user stories
  • vs migraine
  • 20 mg tablets
  • is itchy skin a side effect of
  • 9006681662
  • taking no appetite
  • oral
  • generic 20 mg
  • e agitazione
  • drug information
  • dose 9 lb baby
  • foros o
  • partir 2
  • 37.5 reviews
  • 3723776372
  • dosage for migraines
  • can you take get high
  • together
  • fda approved replacement drug
  • keramag nr. 1 plan wc
  • drug
  • cream over the counter u.k
  • price generic
  • best time day take
  • maximum single dose
  • bioavailability vs levo
  • plm unam
  • can you take 2 days in a row
  • indicaAA?o e contra indicaAA?o
  • does cause hair loss
  • para que es el medicamento
  • prescription cost australia
  • good for tooth infection
  • e cipralex
  • time to work
  • is it legal to buy online in australia
  • cutting cycle
  • coverage staph
  • best brand in india
  • sale canada
  • beta hemolytic strep
  • can take vicodin
  • cost comparisons
  • prix du 10 mg bayer
  • appetite stimulant dosage
  • a alkohol
  • splitting tablets
  • y consumo de alcohol
  • capsules 4mg
  • xarope bula
  • thigh hematoma
  • efeito amoxicilina
  • 75 safe while pregnant
  • days 3-7 or 5-9
  • and insomnia
  • rhodiola
  • extended protocol
  • 5287008747
  • para que sirve el 850
  • difference between and
  • effects baby
  • side effects blurry vision
  • overdose treatment
  • rp 400 mg
  • compared to
  • ipratropio 0.25
  • emedoutlet
  • does vitamin c affect
  • financial assistance
  • prostate cancer
  • klonopin combination
  • side effects
  • tinea cruris
  • build up tolerance
  • $4 copay participating pharmacies walgreens
  • medicinenet
  • quick does work
  • pashol von mp3 free download
  • there difference between
  • tylenol vs advil vs aleve vs
  • better anxiety zoloft
  • opened packets
  • oral thrush babies
  • natural substitute
  • steroids
  • normal dosage
  • e gambe gonfie