One of the main reasons why it’s difficult to troubleshoot problems in complex environments is lack of documentation. This post is long overdue because it was inspired a while ago by Dan Fink’s post (again). Be sure to read it first.

From time to time, I see situations when one person leaving the company leads to a disaster that puts in danger the whole existence of multimillion businesses. Why? Because there is nobody else who knows how system works and how all its moving parts are connected. How does that happen? No knowledge sharing.

The most universal method of knowledge transfer is documenting stuff and I would argue that, most of the times, one hour of documenting brings to a business more than two hours developing new functionality in PL/SQL.

Actually, there might not even exist such know-it-all-wizard. Complex systems are not created in a matter of days or even months. Often, extreme complexity is the result of years or even decades of evolution. All those workarounds, bug fixes, re-designs, pressing deadlines — those friends of entropy win time from documentation tasks.

How many times it happened to you:
- Why are there forty-two underscore parameters in every new database you create?
- Because it’s our standard.
- Can I look at it?
- {proudly} Yes. It’s in every init.ora file we have!

Or another variation:

- Can I look at it?
- {proudly} Yes. Here is 42 pages document how to create new database:

Performance boosters:
_db_aging_touch_time=…
_db_aging_stay_count=…
_db_writer_chunk_writes=…
_discrete_transactions_enabled=…

Bugs:
_kgl_latch_count=…
_kgl_bucket_count=…
_lm_rcfg_timeout=…

and etc…

- When was this document started?
- {proudly} Oh! Forty-two years ago!
- When was it last time updated?
- Eh… I’m actually quite new here…

What are the reasons behind missing or inadequate documentation?
Just to name a few:
- It’s boring.
- We have no time for it now and it’s not a priority (it never will be) — we still missing couple reports.
- I’ll do it tomorrow.
- Project manager is not paid by number of pages of documentation produced (not that this is a good idea either).
- I’m the only one who knows this system — they must be idiots to get rid of me.
I’m sure readers can name plenty of others and please feel free to do so.

Sometimes you can avoid complexity and make systems simple by design but in real life it’s more often when you need to deal with complexity. Paraphrasing Dan — documentation will not make systems simpler but will reduce perceived complexity.

If you’ve got to support and/or develop a complex system — how do you make sure you dedicate enough time to documentation? What are the strategies to keep it up to date? How to explain the importance of documenting to the management? If you have anything to say — the comment box is at the bottom of this screen — use it. If you think that you have enough to say for the blog post — just let me know.

Find more about  , , ,

7 Responses to “Avoiding Guesswork in Complex Environments”  

  1. 1 Christian Antognini

    Amen!

  2. 2 Frits Hoogland

    Good points Alex! Exactly the last situation I was partly involved with.

    In my experience, setting up a successful system, either complex or simple, requires a systematic approach from the start. And, which cannot be said enough, simplicity!

    A database needs to be adjusted as little as possible. Sourcecode needs to be version controlled, Even in very small simple environments. Production environments need software releases, explicitly tagged from the version control. Not a few files to be fiddled with in order to make it work again.

    Happy easter!

  3. 3 Narendra

    Alex,

    Nice thoughts.
    I agree completely.
    Your last reason is very important. People, many times, do not share knowledge, due to insecurity. Sounds funny, but it is a fact. They (probably) get “married” to the company and whenever they are acknowledged as S.P.O.C. (Single Point-Of-Contact), it soothes their ego.
    Such people will do everything they can, to sabotage any plans to prepare / update documents.

  4. 4 Aleksey Tsalolikhin

    Documentation is very important and under-used in today’s IT shops.

    To help remedy that, here’s my recent post on system administrator site documentation with docbook, and a few “intro to docbook” links that were helpful to me.

  5. 5 Alex Gorbachev

    Good point. Thanks Aleksey.
    The biggest difficulty I found is to motivate engineers to write docs AND keep them up to date.

  6. 6 Graham Oakes

    Excellent points Alex.

    I would also add that writing the documentation is only useful if you ensure it’s properly distributed or can be easily found. Finding the correct location for the documentation and ensuring that it’s organised in such a way to make retrieval easy is not as simple as it could be in a large company(at least that’s what I’m observing where I work atm).

  7. 7 Alex Gorbachev

    Thanks Graham. I might be biased or just used to Oracle Database documentation style but after comparing Oracle database documentation with documentation for the products that Oracle acquired — I can only attest to your point. The ability to navigate quickly and effectively within a large documentation volumes is a must.

    But it’s not only the documents creators who should invest efforts, readers must invest enough to be familiar with organization and knowledge domain. Even when search functionality works great, you would still need to know what to search for.

    Thanks again for the feedback.

Leave a Reply


  • pakistani girls free porn videos
  • eat her own mothers pussy
  • amber forever big black tits
  • Sexy nude girls making out
  • how fuck wife ass
  • subtitles for asian movies
  • free amatuer oral sex homemade videos
  • free audition girl fuck
  • sexy hot little girls
  • ass kiss bet sex story
  • Extreme anal fuck
  • explicite asian sex world
  • candy girl video jasmine grey
  • female orgasm anal
  • free flash porn video
  • beautiful girls the movie
  • mature lesbian teen pics
  • anal boob suck pee
  • Nude lesbians licking pussy
  • senior erotic stocking pics
  • sick anal pictures
  • pictures throat cancer graphic
  • Amateur girls having sex
  • Madoona sex scene
  • teen big boob pics
  • russian schoolgirls free porn
  • hindi movie about sex
  • lesbian mature powered by phpbb
  • hot beautiful blue eyed girls naked
  • naked pictures of girls making out
  • amateur community porn
  • christie amateur allure videos
  • crazy mature videos
  • japanese reporter blowjob
  • Nurse lesbian videos
  • long play amateur porn
  • Nude aunty boobs
  • burnette teen shows pussy
  • watch cartoon girls show their boobs
  • Anal sex pleasure
  • little april friend pussy
  • ameture black pretty pussy
  • Lady tsunade nude
  • voyeur wife gang bang videos
  • star trek porn pics
  • porn pics for mature women
  • hot ebony girl
  • little pussy art
  • natural blonde phottos xxx
  • personalized little girl birthday party favors
  • adult anal videos
  • british blonde on black
  • kim vassey topless pics
  • she loves cum in her pussy
  • Nude chicks images
  • Percentage of teens having sex
  • free site for oral porn
  • cute older women with shaved pussy
  • magazins for teen girls
  • sex video csfb
  • hot thin black chicks nude
  • ebony porno pics
  • glamour girl pictures
  • school girls deep throat
  • mature stocking and suspenders sex
  • video of girls bras
  • fat omen sex videos
  • uncensored girls next door calendar pics
  • nude young ones pics
  • mini teen fuck small teen
  • does pornstar morgan leigh do hardcore
  • full length black pussy vids
  • porn movie talk about white chicks
  • black girls for old white cock
  • Man vs girl sex
  • free ebony teenies
  • Naked fuck sex
  • antient german sex videos
  • eating granny pussy pornhub
  • Bigger cum shots
  • beoncye naked sexy
  • asian teen movie thumbnails
  • deep throat teen galleries vid
  • interracial amuater sex
  • hardcore amateur pictures and
  • best free cumshot facial videos
  • young asslicking girls
  • free xxx couples pissing
  • his first sex galleries
  • softcore bbw galleries
  • girl with nice butt gives blowjob
  • wei pussy porn
  • mature sex tubes amateur
  • virgin anal galleries
  • hottie couples sex
  • rate picture couples
  • girl friends boobs
  • pornstar ange venus pictures
  • girls touching each others boobs
  • young young girls having sex
  • youg teen girls
  • milf amateur video galleries plus
  • sexy hot photos of japan women
  • amature sex fee vids
  • sexy foreplay oral
  • Teen ebony video
  • sexy girls getting naked videos
  • pics of women with shaved vaginas
  • free teen amatuer galleries
  • petite bassit giffin
  • maria bello sex naked
  • free black creampie videos
  • sex massage lick my clit
  • finger my wet pussy stories
  • schoolgirl topmodel fuck
  • free pussy lick tubes
  • upclose pussy photos
  • bouncing boobs private video
  • Nice ass panties
  • mile school pussy
  • free saggy mature tit movies
  • handjob explosions movie thumbs
  • hot sexy babes ass tits
  • topless workout video
  • asain schoolgirl sex
  • topless south african girls
  • free asian facial videos
  • british girl fucked sex video
  • amateur adult movie retailers
  • free download of teen fucking movies
  • naked college girls vide
  • dailymotion black boobs
  • hot pussy sex video
  • girl shower naked nude undress undressing
  • free pale girl pics
  • free grandma amateur sex tube
  • jana horova fuck video
  • girls kissing butt
  • Big sex fuck
  • pocket pussy red tube
  • male teen fuck
  • cute hairstyle pictures for teen girls
  • free porn of couples
  • pics of bald little pussy
  • montage hot bikini girl
  • moms having anal sex
  • sex afraid to orgasm
  • first anal sex tops
  • lesbian anal lick orgys
  • family nudism gallery
  • pics of girls makin out
  • teens girls movies mpg
  • petite princess boutique
  • free hairy pussy video
  • asian housewife mature
  • bald little young pussy
  • naked teen posts
  • porn mom vid list gallery
  • sex movies of old and young
  • Jennifer lopes sex scene
  • free chubby nude teen girls
  • mature younger lesbian foreign
  • three anal suck
  • babysittter young girls stories
  • girl on girl porn and free
  • fingering into the navel pictures
  • Tila tequila nude pussy pics
  • gaa player porn movie
  • black men and your wife
  • Lesbian strap on movies
  • free blonde shemale porn
  • gp hardcore xxx download
  • xxx old mature tube thumb
  • ass twysties girl
  • schoolgirls fuck and suck
  • pornhub veronica lane anal
  • naked lesbian sex slaves
  • mario lopez sex videos
  • naked girls gym
  • women pics mature
  • teenage girl milking
  • free full video asian nude
  • naked latina video
  • big ginger pussy faye valentino
  • wife sister threesome
  • hardcore bondage movie
  • Bizzare sex porn
  • black stockings lonely housewife
  • Sexy girl dawn
  • sex asian video free
  • Chubby sex videos
  • Irani sex girl
  • Lesbian beach girls
  • Hard core sex video
  • hampster lesbian porn
  • wet young milf pussy
  • photos of harajuku girls of japan
  • sexy mature women seduction videos
  • xxx sluts naked outside
  • beautiful couples movie
  • free gilf milf porn video
  • lolly badcock anal movies
  • instructions wife for black men
  • nice ass lesbian ya
  • free sex cunts mature
  • home sex video upload
  • pussy licking old young
  • my wifes asian ass
  • streaming mature videos hampster
  • teen fucked in the pussy
  • Lesbin hot sex
  • porn black light beauty
  • anal sex stimulation
  • jap teacher sex videos
  • vanessa hudgens contraversial naked pics
  • curious teens hardcore
  • sex girls blacks and blonds
  • interracial pregnancy creampie videos
  • porn actress blonde retired
  • bikini iranian girl houston
  • japanese public fuck tubes
  • free long young couples sex videos
  • free pica mature wives flashinf
  • asian free videos porn
  • jew girls sucking big black cocks
  • Hot nude chiks
  • Ass shake in thong
  • junior boobs pics
  • girl scout teens nude
  • leticia cline pussy nude
  • ols mwn young girls
  • round granny ass xxx
  • sexy teacher fuck
  • Leslie ann warren nude photos
  • red haired girl strips gallery
  • nude teenies anal
  • lesbian strap on hardcore anime
  • joanna garcia topless pics
  • christina aguilera young girl dont cry
  • large dicks young girls
  • asian lesbian strapon big time
  • asian grope videos
  • pastors wife naked
  • the sexiest black porn stars
  • thumbnail gallery of young teen girls
  • wholesome naked girls
  • threesome movie thumbs
  • Teens flash boobs
  • black dudes on white bitch
  • big lip pussy hi def
  • haley paige porn videos
  • slutload little girls pussy
  • skinny girl xxx
  • xxx amatuer porn photos
  • petite busty africans
  • young black hot porn girlz
  • erotic fiction for women sexy stories
  • school girl fuck stories
  • Teenagers sex video
  • denying wife sex advances
  • dirty erotic sex stories and pictures
  • redtube lesbian blonde
  • naked sexy arab photos
  • free nude legal ebony teens
  • extreme young bikini pics
  • free video girl fucked
  • Xxx sex big tits
  • amauer sex videos
  • jenna jameson video porn
  • naked women showing off goodies videos
  • spiderbabe hardcore goth free pics xxx
  • White teen pics
  • free drunk teen girl porn
  • Ass perfect teen tight
  • big juicy teen butts
  • hardcore psp porn
  • sex with threesome
  • men boy fuck videos
  • bathroom sex vids
  • daily porn galleries free
  • slut eating my pussy
  • Nude iranian girls
  • order porn videos
  • shemale teens live
  • publis sex teen pics
  • lesbian porn penis boe
  • horror movie babes dead
  • amateur young porn tube
  • japanese mom sons porn fuck
  • amateur stranger sex
  • girls round butts nudity
  • free pussy no sign up sites
  • old naked pussy
  • Iran sex girls
  • pics of mature fat women
  • horny mature video galleries
  • huge tits video riding bouncing free
  • drunk party girls for free
  • roxanne milana pussy
  • pussy hair cut styles
  • ass n feet galleries
  • pictures of throat cancer
  • Old fucking girl
  • Hot white pussy
  • Cum shot ass
  • videos de tiffany teen
  • flashing pussy in library
  • Pictures of small breasts
  • petite blue jeans
  • s fuck sons porn pics
  • shy girl naked audition slut load
  • latina teen bikini models
  • pussy ready to eat closeup
  • asiann girl with huge tits
  • beautiful girls ith beautiful tits
  • sexy butt movies
  • Anal big fuck
  • girl undressing free video
  • fucking pussy ass sex
  • blonde wives post
  • jenna jammeson handjob video galleries
  • my big wet pussy video
  • teen swimming party ideas
  • unblocked topless pictures
  • my sexy wife maria
  • free hot red heads pussy
  • Daily sex videos
  • fireside pussy avi
  • really petite girls nude
  • wife posts erotic
  • free petite yng bbs movies
  • blonde babes sex galleries
  • aka naked women pussy
  • free asian sex vidoes
  • xxx twink porn free
  • black woman giving blowjob
  • pic of black butts
  • nude women porn free pics
  • naked black guys with big dicks
  • Hot ass young
  • amateur twink porn
  • destroyed wife pussy creampie
  • hot girl party hardcore
  • lengthy sex xxx free
  • poast amateur free nudes of wife
  • amateur pussy free movies orgasm
  • pokegirl sex league
  • pale white girls sex
  • legs video boobs
  • Nude beauties video
  • first time anal pornhub
  • dipa sex movie tube
  • momma shaved my pussy slut load
  • indian sex lesbian
  • free babes porn movies
  • Sweet innocent teen
  • black dicks latins chicks
  • sex group xxx
  • czech chicks anal
  • asian japanese squirt black torrent
  • Thick white girl dancing
  • Hot sex teen in germany
  • lacey glamour girls gone bad
  • ass point video
  • young teens nude hairy
  • Paul donahoe nude photo
  • fingering girls in there sleep
  • ass shakin sex
  • big tit handjob free videos
  • sexy newspaper girls
  • friends pulling off girls bikini top
  • huge bottle pussy insertions
  • Hot girl shoes
  • Chris brown big ass dick
  • can males suck their own dick
  • older people having oral sex
  • streaming amateur free video
  • black beauty tomato seed
  • Xxx crossdresser teens
  • big tits round ass long videos
  • skinny bikini teen girls
  • sexy young kdz
  • nude naked new videos
  • free violent fuck videos forced sex
  • girl screams fuck me harder
  • naked dutch girls in wooden shoes
  • illigal xxx teen
  • very young model galleries
  • masturbation in very young girls
  • pornhub white bitch wants to fuck
  • free full japanese porn
  • long black dick fuck mother daughter
  • xxx threesome videos gtube
  • mature pics forum
  • hentai young pictures
  • utube cream pie pussy
  • young girls first blowjob sex videos
  • video laure manaudou naked
  • fuck movies t home
  • little young teen sex