One of the main reasons why it’s difficult to troubleshoot problems in complex environments is lack of documentation. This post is long overdue because it was inspired a while ago by Dan Fink’s post (again). Be sure to read it first.

From time to time, I see situations when one person leaving the company leads to a disaster that puts in danger the whole existence of multimillion businesses. Why? Because there is nobody else who knows how system works and how all its moving parts are connected. How does that happen? No knowledge sharing.

The most universal method of knowledge transfer is documenting stuff and I would argue that, most of the times, one hour of documenting brings to a business more than two hours developing new functionality in PL/SQL.

Actually, there might not even exist such know-it-all-wizard. Complex systems are not created in a matter of days or even months. Often, extreme complexity is the result of years or even decades of evolution. All those workarounds, bug fixes, re-designs, pressing deadlines — those friends of entropy win time from documentation tasks.

How many times it happened to you:
- Why are there forty-two underscore parameters in every new database you create?
- Because it’s our standard.
- Can I look at it?
- {proudly} Yes. It’s in every init.ora file we have!

Or another variation:

- Can I look at it?
- {proudly} Yes. Here is 42 pages document how to create new database:

Performance boosters:
_db_aging_touch_time=…
_db_aging_stay_count=…
_db_writer_chunk_writes=…
_discrete_transactions_enabled=…

Bugs:
_kgl_latch_count=…
_kgl_bucket_count=…
_lm_rcfg_timeout=…

and etc…

- When was this document started?
- {proudly} Oh! Forty-two years ago!
- When was it last time updated?
- Eh… I’m actually quite new here…

What are the reasons behind missing or inadequate documentation?
Just to name a few:
- It’s boring.
- We have no time for it now and it’s not a priority (it never will be) — we still missing couple reports.
- I’ll do it tomorrow.
- Project manager is not paid by number of pages of documentation produced (not that this is a good idea either).
- I’m the only one who knows this system — they must be idiots to get rid of me.
I’m sure readers can name plenty of others and please feel free to do so.

Sometimes you can avoid complexity and make systems simple by design but in real life it’s more often when you need to deal with complexity. Paraphrasing Dan — documentation will not make systems simpler but will reduce perceived complexity.

If you’ve got to support and/or develop a complex system — how do you make sure you dedicate enough time to documentation? What are the strategies to keep it up to date? How to explain the importance of documenting to the management? If you have anything to say — the comment box is at the bottom of this screen — use it. If you think that you have enough to say for the blog post — just let me know.

Find more about  , , ,

7 Responses to “Avoiding Guesswork in Complex Environments”  

  1. 1 Christian Antognini

    Amen!

  2. 2 Frits Hoogland

    Good points Alex! Exactly the last situation I was partly involved with.

    In my experience, setting up a successful system, either complex or simple, requires a systematic approach from the start. And, which cannot be said enough, simplicity!

    A database needs to be adjusted as little as possible. Sourcecode needs to be version controlled, Even in very small simple environments. Production environments need software releases, explicitly tagged from the version control. Not a few files to be fiddled with in order to make it work again.

    Happy easter!

  3. 3 Narendra

    Alex,

    Nice thoughts.
    I agree completely.
    Your last reason is very important. People, many times, do not share knowledge, due to insecurity. Sounds funny, but it is a fact. They (probably) get “married” to the company and whenever they are acknowledged as S.P.O.C. (Single Point-Of-Contact), it soothes their ego.
    Such people will do everything they can, to sabotage any plans to prepare / update documents.

  4. 4 Aleksey Tsalolikhin

    Documentation is very important and under-used in today’s IT shops.

    To help remedy that, here’s my recent post on system administrator site documentation with docbook, and a few “intro to docbook” links that were helpful to me.

  5. 5 Alex Gorbachev

    Good point. Thanks Aleksey.
    The biggest difficulty I found is to motivate engineers to write docs AND keep them up to date.

  6. 6 Graham Oakes

    Excellent points Alex.

    I would also add that writing the documentation is only useful if you ensure it’s properly distributed or can be easily found. Finding the correct location for the documentation and ensuring that it’s organised in such a way to make retrieval easy is not as simple as it could be in a large company(at least that’s what I’m observing where I work atm).

  7. 7 Alex Gorbachev

    Thanks Graham. I might be biased or just used to Oracle Database documentation style but after comparing Oracle database documentation with documentation for the products that Oracle acquired — I can only attest to your point. The ability to navigate quickly and effectively within a large documentation volumes is a must.

    But it’s not only the documents creators who should invest efforts, readers must invest enough to be familiar with organization and knowledge domain. Even when search functionality works great, you would still need to know what to search for.

    Thanks again for the feedback.

Leave a Reply


  • shy wife sex auditioin
  • chita girld naked
  • black amateur madison tube movie
  • glamour stocking legs sex
  • Open sex in india
  • 7958774158
  • my pussy juices running down
  • shaved pussy close up videos
  • pictures of teens with big breasts
  • reality outdoor pussy
  • 4212021250
  • shaved teen girls pussy orgasm pics
  • teen destruction sex videos
  • free lesbian mature seduce young video
  • hot teen hardcore fucked
  • filipina nude young pics
  • celebrite pics
  • web cam girls showing tits
  • better sex wife
  • girls stripping for webcams
  • colombian sex videos
  • video aria and lee stone
  • lesbian teens having sex after party
  • Hentia sex movies
  • amateur lesbian wives
  • teenies huge fuck
  • hg mature video
  • wife adult free video
  • angelique leclair anal sex movies
  • indain group sex
  • Large female breast
  • boobs norma video
  • grandpa drops his load in pussy
  • naked betrayal movie
  • teen orgy facial video
  • sex eduion topics
  • Head in girls pussy
  • kathryn rogers bikini pics
  • ebony star jasmine shy
  • Upskirt sex videos
  • free video babysitter exploited
  • black big nipples
  • the girls next door videos
  • teen girlfriend handjob amateur
  • lindsay lohan sex video blowjob
  • young sleeping teen blond videos
  • Lesbian finger licking
  • Mother sex hot
  • xvideo tiny girl pussy cum
  • free shemale anime galleries
  • Rihanna nude images
  • japanese beauties galleries
  • jordan capri pics
  • huge vacum pussy
  • pussy fucked socks
  • video post shemale
  • lick cum on pussy
  • pussy powered by phpbb
  • Sex and pics
  • girls in hardcore
  • Home made sex scenes
  • teen anal nude
  • Litle girl pussy
  • torrent teen pussy fist
  • amateur milf sleep sex videos
  • sexy fake tits
  • rough group sex movie
  • plumper anal slut pictures
  • naked girlssex videos
  • shemale gallerys
  • 3003655166
  • horny mature shaved pussy
  • rough hardcore anal
  • free xxxx amature wife videos
  • free ex girlfriend cumshot hardcore pics
  • mature women seducing little girls slutload
  • rubber toys to convert to pussy
  • hardcore sex gie style
  • femdom blowjob videos
  • teen girl photo thumb sites
  • shasta high videos
  • hairy mature pussy hardcore
  • home made sex movies for sale
  • girls rubbing tits
  • girl with big tits getting fuck
  • photos of anal sex damage
  • Perfect pussy licking
  • free young old girl sex
  • legs glamour pics
  • movie previews
  • anime with huge dicks
  • amanda logue actress movies
  • Huge ass black
  • asian lesbian videos
  • sexy girl drinking a milkshake
  • x videos amateur
  • daughter sex story erotic
  • video girl precum licking
  • tight bleached pussy
  • ass fuck teen cumshot
  • free amaeture pictures
  • sexy south seas island girls
  • sweaty boobs picture
  • topless girls gallery
  • free amateur orgy nude gallery
  • indian pussy in fayetteville nc
  • large shemale dick
  • mature adult group sex
  • group watchin couple have sex
  • high quality videos of teen girls
  • Romantic hot sex
  • inch dick girl st time
  • free online hardcore
  • marina amateur star
  • 2040016555
  • high school bikini sexy
  • big hard black shemale cock
  • naked girls petite
  • Hugh fucking tits
  • madison young fuck
  • Candid teen bikini pics
  • young black women fucking
  • miley cirus topless vanity fair pics
  • streaming erotic movies free
  • Naked sport girls
  • blacks fuck straight guy
  • sexy girls in tall socks photos
  • anal fist fuck
  • petite asian women twan
  • black girls shaking dat ass naked
  • japan teen idol s
  • ebony lesbians with strapons
  • pale busty blonde teen
  • 2588540285
  • how to recieve oral sex
  • peachy forum stars videos
  • black mailed to fuck
  • girl in black basque
  • amazing anal
  • big tit fuck tube
  • sex pictures pic hunter
  • young enema
  • quickie handjob videos
  • Hairy pussy flash
  • nonude ebony treteen
  • pussy videos dailymotion
  • young people sex stories photo
  • Girls peeing
  • teen movies from max hardcore
  • casual mature sex
  • aol fuck me hard black women
  • free download of indian videos
  • Hollywood sex scenes movies
  • black ops hardcore kill teamates rules
  • naked americain girls being tickled
  • thai sex tour for lesbians
  • redhead girl having sex
  • free young skinny girl pictures
  • asian girl getting stuffed xtube
  • 4924022709
  • free oral lesbian stories
  • unrestricted free sex picture search engine
  • kendra wilkinson sex tape tube video
  • online japanese girls videos
  • petite asian videos
  • starlets naked in movies
  • free streaming indian nude videos
  • ameture anal sex
  • 8994482573
  • ex teen girlfriends naked
  • sex oral tongue ring meaning
  • white hard on black chic pictures
  • girl getting her breasts kissed
  • illegal naked teen girls
  • Tamil online sex movies
  • young school girls having sex
  • mature young couple suck
  • black knight gallery
  • drunk girls fucking videos guam
  • transexual butt fucking
  • super models gallery bikini lingerie
  • Granny sexy ass
  • hot petite teen lesbos nude
  • 2829437017
  • woman sticks fish in pussy
  • teen fashon model gallery
  • xxx ebony free movies
  • first anal video
  • movies huge jugs
  • orangi town sex videos
  • girls young pharaoh
  • amateur videos squirt
  • mature gallery long
  • black andalusian pictures
  • Nude skiing pics
  • hardcore horny latina pussy fucking
  • wet shaved bent over pussy
  • skinny granny puffies video
  • caned ass movies
  • sexy chubby girl naked
  • juicy cumming pussy
  • naked girl tuxedo
  • black white gallery
  • Sexy naked boobies
  • interracial black sluts
  • husband tapes wife having sex
  • uk teen sex vids
  • mom fucks young boy movies
  • black beauties
  • free teen lesbian anal bead movies
  • wife gangbang stories xxx
  • Sexy girl in diapers
  • 977733086
  • Lesbian streaming
  • wife withholding sex husband has affair
  • amateur teen pussy pictures
  • Ebony cum mouth
  • young naked girls shaved
  • free shemale sex gallerys
  • videos of shemale cumshots
  • pussy lips and butthole
  • n tubes teen tit pictures
  • buff teens naked
  • free very big tits movies
  • black beauty eggplant seeds
  • ordered him to lick pussy
  • skinny young asian girl
  • video flashing teen
  • Sex positions during pregnancy pics
  • Female ass fuck
  • fuck chubby girls
  • mature moms pussy
  • old sluts for young blackmen
  • perfect fake tits pictures
  • massive orgasm instruction video
  • coaks pussy pictures
  • sex movies free video
  • brooke adams ass videos
  • bi mmf ches wife
  • vintage sex twat ass
  • Nude swedish teens
  • free amature hardcore sex movies
  • please fuck my wife tube
  • orgasm toying thick dildo video
  • Ass sex movie
  • hot sexy shemale outfits
  • petite teen young tube
  • Girl girl lesbians
  • young teen homemade pussy play
  • couples having sex on live webcams
  • girl stripped naked and whipped
  • teens babysitting sex
  • upload sex movies
  • 5496054078
  • videos s mexicanos sexo anal gratis
  • free quicktime movie
  • husker girl flashing
  • oral sex for partners
  • extra long sex movies
  • sex moms tits
  • black lesbian tube licking ass
  • ass gaping videos
  • college girls have orgasm
  • small asian dick
  • free black lebians
  • free long grandpa sucks pussy
  • white moms interracial black men
  • bump lovers bareback fuck pussy
  • jr teen beauty pageant nudist download
  • long cock in pussy
  • free boys like girls pictures
  • xxx blonde ass young
  • pussy video tube
  • ebony black pussy pics
  • The vagina monolgues
  • 9465323746
  • classic movies debbie does
  • freeky anal sex
  • my cock ion my girls pussy
  • xxx girls cash money for sex
  • Sex my video
  • cannibal sex movie
  • asian anal miyagi mimi
  • humiliate suffer fuck rough punish
  • skinny ebony gallery
  • met art tiny shaved pussy
  • black ass drillers
  • amateur photo gratuite sexe
  • free fulllength indian movies
  • der romantic sex women videos pussy
  • the holic church and oral sex
  • lesbian penthouse hardcore sex
  • All teen videos
  • drunken hairy pussy
  • Asian teens hardcore
  • christy canyon fuck pics
  • i compatible xxx movies
  • tanda chicks movies
  • xxvids indian
  • teen beauty hardcore
  • free girlfriend blowjob videos
  • young teen girls gang bang
  • Types of pussy pictures
  • shared fuck movie
  • Amateur teen cams
  • girls with large natural breasts
  • song from girls video on afv
  • legal naked young girles
  • sweet teen babes victoria