One of the main reasons why it’s difficult to troubleshoot problems in complex environments is lack of documentation. This post is long overdue because it was inspired a while ago by Dan Fink’s post (again). Be sure to read it first.

From time to time, I see situations when one person leaving the company leads to a disaster that puts in danger the whole existence of multimillion businesses. Why? Because there is nobody else who knows how system works and how all its moving parts are connected. How does that happen? No knowledge sharing.

The most universal method of knowledge transfer is documenting stuff and I would argue that, most of the times, one hour of documenting brings to a business more than two hours developing new functionality in PL/SQL.

Actually, there might not even exist such know-it-all-wizard. Complex systems are not created in a matter of days or even months. Often, extreme complexity is the result of years or even decades of evolution. All those workarounds, bug fixes, re-designs, pressing deadlines — those friends of entropy win time from documentation tasks.

How many times it happened to you:
- Why are there forty-two underscore parameters in every new database you create?
- Because it’s our standard.
- Can I look at it?
- {proudly} Yes. It’s in every init.ora file we have!

Or another variation:

- Can I look at it?
- {proudly} Yes. Here is 42 pages document how to create new database:

Performance boosters:
_db_aging_touch_time=…
_db_aging_stay_count=…
_db_writer_chunk_writes=…
_discrete_transactions_enabled=…

Bugs:
_kgl_latch_count=…
_kgl_bucket_count=…
_lm_rcfg_timeout=…

and etc…

- When was this document started?
- {proudly} Oh! Forty-two years ago!
- When was it last time updated?
- Eh… I’m actually quite new here…

What are the reasons behind missing or inadequate documentation?
Just to name a few:
- It’s boring.
- We have no time for it now and it’s not a priority (it never will be) — we still missing couple reports.
- I’ll do it tomorrow.
- Project manager is not paid by number of pages of documentation produced (not that this is a good idea either).
- I’m the only one who knows this system — they must be idiots to get rid of me.
I’m sure readers can name plenty of others and please feel free to do so.

Sometimes you can avoid complexity and make systems simple by design but in real life it’s more often when you need to deal with complexity. Paraphrasing Dan — documentation will not make systems simpler but will reduce perceived complexity.

If you’ve got to support and/or develop a complex system — how do you make sure you dedicate enough time to documentation? What are the strategies to keep it up to date? How to explain the importance of documenting to the management? If you have anything to say — the comment box is at the bottom of this screen — use it. If you think that you have enough to say for the blog post — just let me know.

Find more about  , , ,

7 Responses to “Avoiding Guesswork in Complex Environments”  

  1. 1 Christian Antognini

    Amen!

  2. 2 Frits Hoogland

    Good points Alex! Exactly the last situation I was partly involved with.

    In my experience, setting up a successful system, either complex or simple, requires a systematic approach from the start. And, which cannot be said enough, simplicity!

    A database needs to be adjusted as little as possible. Sourcecode needs to be version controlled, Even in very small simple environments. Production environments need software releases, explicitly tagged from the version control. Not a few files to be fiddled with in order to make it work again.

    Happy easter!

  3. 3 Narendra

    Alex,

    Nice thoughts.
    I agree completely.
    Your last reason is very important. People, many times, do not share knowledge, due to insecurity. Sounds funny, but it is a fact. They (probably) get “married” to the company and whenever they are acknowledged as S.P.O.C. (Single Point-Of-Contact), it soothes their ego.
    Such people will do everything they can, to sabotage any plans to prepare / update documents.

  4. 4 Aleksey Tsalolikhin

    Documentation is very important and under-used in today’s IT shops.

    To help remedy that, here’s my recent post on system administrator site documentation with docbook, and a few “intro to docbook” links that were helpful to me.

  5. 5 Alex Gorbachev

    Good point. Thanks Aleksey.
    The biggest difficulty I found is to motivate engineers to write docs AND keep them up to date.

  6. 6 Graham Oakes

    Excellent points Alex.

    I would also add that writing the documentation is only useful if you ensure it’s properly distributed or can be easily found. Finding the correct location for the documentation and ensuring that it’s organised in such a way to make retrieval easy is not as simple as it could be in a large company(at least that’s what I’m observing where I work atm).

  7. 7 Alex Gorbachev

    Thanks Graham. I might be biased or just used to Oracle Database documentation style but after comparing Oracle database documentation with documentation for the products that Oracle acquired — I can only attest to your point. The ability to navigate quickly and effectively within a large documentation volumes is a must.

    But it’s not only the documents creators who should invest efforts, readers must invest enough to be familiar with organization and knowledge domain. Even when search functionality works great, you would still need to know what to search for.

    Thanks again for the feedback.

Leave a Reply


  • black bulls white pussy stories
  • mature black cream pies
  • interracial sex on tube
  • brazil teen fucking hugh cocks movies
  • realistic dildo amateur xxx
  • squeeze nipples video
  • facial muscle pictures
  • ebony sucking xxx
  • videos free nude girl
  • philipina teen sex
  • girls that like fingering themselves
  • jen stefani pussy
  • black sabbath singels japan
  • Interracial gangbang videos
  • video and and sexe and amateur
  • movie character fantasy sex stories
  • pussy with big labias
  • mature milfs free pics
  • anal teen first
  • dash rip rock pussy whipped
  • young couple teen tube fuck
  • Girls breasts pictures
  • pornstar jamie lynn money talks orgasm
  • thai asian pussy
  • sexe amateur video
  • free women undressing porn video galleries
  • free fine ebony video
  • twin sisters lesbians sex
  • Naked spring break video
  • german elevator sex video
  • tyranny fuck girls
  • Ali lohan nude pics
  • two girls office in panties pics
  • xxx mature women with hairy
  • ex girle sex pictures
  • in side young wet pussys
  • pussy blond hair
  • free teen sex video sharing
  • free ethic sex videos
  • blowjob and deepthroat porn
  • serious amateur porn
  • sexy hot bikini gallery you
  • fingering her tight pussy
  • junior teen amateur
  • fat pussy hd
  • Big home movies sex
  • busty mature pics
  • sex mature free
  • girls rubbing and squashing there boobs
  • russian sex rollplay movie
  • porno tube sex videos
  • little boy gets pussy
  • home fucking sex vids
  • asian sexual practices
  • Beach sex videos
  • anal sex photos
  • young girl sucks dys cock
  • Nude teen gallaries
  • homemade swingers party videos
  • Big ass parade video
  • Eminem shake that ass lyric
  • man wife home sex tape
  • Pitures of hot girls
  • pussy pics of celebrities
  • Hot lesbians in skirts
  • Matthew mcconaughey naked pics
  • college girl fuck
  • blood from pussy
  • sexy ass pranks
  • nude girls free blowjob small cock
  • videos on girls getting ate out
  • matures having group sex
  • golf fuck videos
  • fingers in her pussy
  • ass movie cuba
  • oily ass pictures of indian
  • free lesbian clean sex vieos
  • fre erotic sex stories
  • wife wants to try anal
  • hoousewife sex galleries
  • the sexiest free shemale videos
  • anastacia pussy dolls
  • amateur se videos
  • younger girls with older men videos
  • amateur mature pictures
  • pool porn two girls one boy
  • hot teens naked on hien cameras
  • stolen homemade porn movies
  • hairy teen pussy up close
  • japanese streaming sex
  • age young teen video
  • hardcore black babe
  • young spanish school girl porn
  • xxx young brutal girls
  • black fuck milf wiesbaden mainz georgina
  • mature korean bitch
  • Sweet latin pussy
  • interracial granny porn pics
  • Sex with asian girl
  • Photos of girls naked
  • mature couple girl
  • blackmen having sex with black men
  • pornstars fuck ordinary guys
  • lacey black tits
  • black women bikini pictures
  • virgin pussy spreader
  • girls holding other girls boobs
  • hot naked jewish girls
  • circe sex girls
  • girls who like brutal sex
  • Big tit adventures
  • amateur canadan girls nudes
  • interracial breeding videos
  • Sex in jungle movie
  • Sexy gymnast ass
  • teen sex ging
  • freee sex movies tube
  • oral sex training free video
  • sexy cheryl coles porn pics naked
  • Xem phim sex hot
  • Big black tits bouncing
  • mature korean thumbs
  • Teen vagina movies
  • young teen lesbians porn
  • Sex xxx boobs
  • free blowjob video mpeg
  • group sex on the yacht
  • anal lesbian hairy busty streams tubes
  • young girl virgin
  • Naked pics of people
  • Hot pussy girls
  • teen young girl taboo sex porn
  • Teen hard nipple
  • free pictures of long nipples
  • sex love babes
  • group sex class
  • asian lesbians teens
  • video ass xxnx
  • rough gangrape black cocks free video
  • bikini girls index
  • sex machines fuck
  • Girlfriend sex porn
  • Hot sex in ass
  • fat teen girl masturbating
  • free girl fucking guy movies
  • tanning topless video
  • sex aict pics
  • hd amateur porn
  • petite piece concertante
  • only mature sex
  • Black teens fucking dildo
  • men engaged in anal sex
  • anal brazilian free pic sex woman
  • Pussy licking cum
  • Paula jones nude pics
  • meet japan girls
  • cross species lesbian sex
  • free nigerian ebony pics
  • Girls touching boobs
  • blowjob blonde wife
  • blonde milf pussy slut
  • beach pictures wife
  • free black homemade amateur sex videos
  • old granny pussy pics
  • young black teen virgins having sex
  • ffm teen hardcore video
  • piss video mouth porn free
  • drunk girls have sex on boat
  • pictures sister sex
  • errotic pussy movies pictures
  • white men on black pussy
  • Crazy cum shot
  • wife swap xxx stories
  • hollywood party girl
  • twin lesbians teens
  • hairy wife galleries
  • hot teens naked outdoor free pics
  • little teens forced to fuck
  • urban sex party
  • russian girl nudegallery
  • momson fuck videos
  • unusual sex galleries
  • ameteur sexy ass
  • xxx mature group swingers
  • video of asin group sex
  • girls lick girls anus
  • anal granny vids
  • bikini fights cock pictures
  • Naked girls riding bikes
  • free videos of shemale fucking female
  • Nude photos of female
  • epic boobs girl
  • absolutly free black pussy videos
  • sexy glamourous blonde model pics
  • Huge teenage boobs
  • Chubby pussy pic
  • Blond girl big boobs
  • best squirting pussy
  • teen role playing sex videos
  • free heels lingerie amateur porn
  • amatuer webcam teen gallery
  • my free amatuer sex videos
  • Street sexy girls
  • lucypinder topless nude video
  • free porn videos home
  • flat petite girl
  • free amateur pornvideo
  • babes xxx streaming ybot
  • asian sex dating
  • ebony teen white dick
  • webcam carmen naked nipp es
  • fat young girl
  • sluty young girls naked
  • dancing bears videos porn
  • anal teen porn free movies
  • young nude redhead girls free
  • amerture teen girls
  • korean nude girls
  • crystal porn movies
  • british mature sex tubes
  • tiny teen pussy huge cock
  • naked girls having fun with guys
  • used blonde porn
  • japan young business men wearing watches
  • watch a sex change video
  • hermaphridite porn gallerys
  • xnxx party girl spanking
  • video of real oral sex
  • bachelorette party sex free video
  • free pics black booy
  • free naked pics of kesha
  • Hardcore pics teen
  • xxx closeup milf mpeg
  • girls pooping gallery
  • teen titans hentai sex
  • teen movie toplist
  • black beauty basting
  • andi pink ass gallery
  • japanese double anal lesbian schoolgirls
  • lesbian woman fucks little girl
  • the most beautifull porn movies
  • erotic young thai girls
  • geile girls in thye subway porn
  • video naked babes
  • sexy mature moms video tube
  • my wife has a flat butt
  • Hot ebony girls nude
  • preview asian movie
  • Real hairy pussy
  • naked trampolining porn
  • teen titan porn videos
  • old fuck black cock video
  • young teens sucking old cocks
  • interracial snuff movies
  • finnish ass pussy pantyhose upskirt
  • girl ride bike to orgasm video
  • Old man teen pussy
  • nude teen pussy model pics
  • xxx porn stars porn stars
  • pussy in pussy
  • sexiest girl singing
  • brunette girl picture
  • porn erotic search tube
  • cute teenies fuck movies
  • home video facial
  • teen boys and girls sex
  • Movie sex scene hot
  • nude cougar sex video
  • Big black ass sexy
  • flashing for sex videos
  • teen girls nude in locker room
  • redkneck bikini pics
  • girl sweet ass licked out
  • younger big breast blonde girls
  • sexy girls of jmu
  • black bitch galleries
  • amatuer mature bdsm videos
  • girls fucking s free videos
  • free interracial hardcore porn
  • beach topless young little
  • chubby young ladies nude pics
  • sex with korean girls
  • heavy girls blowjobs and fucks videos
  • fabulous young naked teen boys legal
  • sexy teens in boyshorts
  • Ass black girl
  • free matuer nude sex videos
  • girls with pasties gallery
  • hot girl anal dildo
  • etnies sex babes
  • So hot sex
  • young girl fucked by brother
  • sexy massage for couples bangkok
  • granny ebony pussy cum
  • dream of dani anal porn torrent
  • Hot nude chinese girl
  • pussy real good song
  • porn videos porn hub
  • hardcore screaming porn and muff diving
  • baby girls butt
  • girls teen parties
  • sexy young bridesmaid
  • video hard sex big dick
  • tawnee stone blowjob gallery
  • mmf sex chester
  • free lil baby porn videos
  • young girls wiyh hairless cunts
  • free long hd sex movie
  • free hardcore porn anime
  • lard ass pictures
  • girdles for petite ladies
  • naked workout pics
  • cum in fat girl pussy
  • old men free porn movies
  • asian sex delights
  • Sexy girls from israel
  • Hot lesbians in panties
  • red tube hairy wet amature pussy
  • free lesbian girls pon videos
  • forbien teen sex
  • hien girls lockeroom webcam
  • boobs river pictures
  • girl band topless
  • real rough anal pics
  • jessics sipson sex pics
  • young latina anal
  • chetta girl pictures
  • mature sex video channels
  • wifes first anal gangbang
  • Big boob milf videos
  • big cock teens anal
  • Fuck teen grils
  • young pussy gush
  • ebony pornstar lilbaby free vids
  • humongous natural tits sucking many dicks