One of the main reasons why it’s difficult to troubleshoot problems in complex environments is lack of documentation. This post is long overdue because it was inspired a while ago by Dan Fink’s post (again). Be sure to read it first.

From time to time, I see situations when one person leaving the company leads to a disaster that puts in danger the whole existence of multimillion businesses. Why? Because there is nobody else who knows how system works and how all its moving parts are connected. How does that happen? No knowledge sharing.

The most universal method of knowledge transfer is documenting stuff and I would argue that, most of the times, one hour of documenting brings to a business more than two hours developing new functionality in PL/SQL.

Actually, there might not even exist such know-it-all-wizard. Complex systems are not created in a matter of days or even months. Often, extreme complexity is the result of years or even decades of evolution. All those workarounds, bug fixes, re-designs, pressing deadlines — those friends of entropy win time from documentation tasks.

How many times it happened to you:
- Why are there forty-two underscore parameters in every new database you create?
- Because it’s our standard.
- Can I look at it?
- {proudly} Yes. It’s in every init.ora file we have!

Or another variation:

- Can I look at it?
- {proudly} Yes. Here is 42 pages document how to create new database:

Performance boosters:
_db_aging_touch_time=…
_db_aging_stay_count=…
_db_writer_chunk_writes=…
_discrete_transactions_enabled=…

Bugs:
_kgl_latch_count=…
_kgl_bucket_count=…
_lm_rcfg_timeout=…

and etc…

- When was this document started?
- {proudly} Oh! Forty-two years ago!
- When was it last time updated?
- Eh… I’m actually quite new here…

What are the reasons behind missing or inadequate documentation?
Just to name a few:
- It’s boring.
- We have no time for it now and it’s not a priority (it never will be) — we still missing couple reports.
- I’ll do it tomorrow.
- Project manager is not paid by number of pages of documentation produced (not that this is a good idea either).
- I’m the only one who knows this system — they must be idiots to get rid of me.
I’m sure readers can name plenty of others and please feel free to do so.

Sometimes you can avoid complexity and make systems simple by design but in real life it’s more often when you need to deal with complexity. Paraphrasing Dan — documentation will not make systems simpler but will reduce perceived complexity.

If you’ve got to support and/or develop a complex system — how do you make sure you dedicate enough time to documentation? What are the strategies to keep it up to date? How to explain the importance of documenting to the management? If you have anything to say — the comment box is at the bottom of this screen — use it. If you think that you have enough to say for the blog post — just let me know.

Find more about  , , ,

7 Responses to “Avoiding Guesswork in Complex Environments”  

  1. 1 Christian Antognini

    Amen!

  2. 2 Frits Hoogland

    Good points Alex! Exactly the last situation I was partly involved with.

    In my experience, setting up a successful system, either complex or simple, requires a systematic approach from the start. And, which cannot be said enough, simplicity!

    A database needs to be adjusted as little as possible. Sourcecode needs to be version controlled, Even in very small simple environments. Production environments need software releases, explicitly tagged from the version control. Not a few files to be fiddled with in order to make it work again.

    Happy easter!

  3. 3 Narendra

    Alex,

    Nice thoughts.
    I agree completely.
    Your last reason is very important. People, many times, do not share knowledge, due to insecurity. Sounds funny, but it is a fact. They (probably) get “married” to the company and whenever they are acknowledged as S.P.O.C. (Single Point-Of-Contact), it soothes their ego.
    Such people will do everything they can, to sabotage any plans to prepare / update documents.

  4. 4 Aleksey Tsalolikhin

    Documentation is very important and under-used in today’s IT shops.

    To help remedy that, here’s my recent post on system administrator site documentation with docbook, and a few “intro to docbook” links that were helpful to me.

  5. 5 Alex Gorbachev

    Good point. Thanks Aleksey.
    The biggest difficulty I found is to motivate engineers to write docs AND keep them up to date.

  6. 6 Graham Oakes

    Excellent points Alex.

    I would also add that writing the documentation is only useful if you ensure it’s properly distributed or can be easily found. Finding the correct location for the documentation and ensuring that it’s organised in such a way to make retrieval easy is not as simple as it could be in a large company(at least that’s what I’m observing where I work atm).

  7. 7 Alex Gorbachev

    Thanks Graham. I might be biased or just used to Oracle Database documentation style but after comparing Oracle database documentation with documentation for the products that Oracle acquired — I can only attest to your point. The ability to navigate quickly and effectively within a large documentation volumes is a must.

    But it’s not only the documents creators who should invest efforts, readers must invest enough to be familiar with organization and knowledge domain. Even when search functionality works great, you would still need to know what to search for.

    Thanks again for the feedback.

Leave a Reply


  • Housewife sex videos
  • girls beauty contest
  • gorgeous bikini girls nude
  • cute young girls in thongs
  • big girls with small tits
  • old shemale free videos
  • teen and older guy threesome
  • sexy erotic mature tube
  • amature black teen galleries
  • girls with dreds porn
  • black female mature photos
  • cup sizes of average teenage girls
  • lesbian gang bang porn gallery pics
  • asian celebrity sextape
  • strieght guys having anal sex
  • asian voyeur free movies
  • porn teacher videos
  • parental tube fuck videos
  • erect men pics for girls
  • sex video femme avec chien
  • amazing ass hot blonde porn videos
  • blonde vs brunette pornstars
  • Lesbian foot fucking
  • Lesbian sex with strapon
  • black big sex and aisa
  • health problems from oral sex
  • sex pics tits
  • free mature sex videos full lenght
  • girl strips and shows pussy
  • sexy erotic secretary office boss
  • sex movies panties
  • white wife sucking black guy
  • forced teen sex vids
  • Sexy china girl picture
  • black and white erotic photos
  • free girl riding vids
  • yuu asakura porn pics
  • Hot naked sexy girls
  • naked sister on webcam
  • hot teen sex ffm
  • israel porn sex video
  • amateur porno hardcore
  • sluttube mother blackmail sex
  • naruto cartoon sex videos
  • upskirt granny boy hardcore pics
  • tube anal sex
  • two girls cup video thatsphucked
  • Sexy hot black lesbians
  • Nude college girls movies
  • sexy girl sybian nude
  • harley davidson fuck
  • emma watson breasts pics
  • sexy young free galleries
  • homemade xxx anal
  • black women big tits sex
  • sexy bbw wife
  • Hot college girls videos
  • wildest amateur anal
  • skinny white girl ass
  • mickey mouse naked girls
  • taylor swift pussy porn naked
  • sexy womens videos black panty hose
  • girls with dilldoes in the pussy
  • pics of naked males of india
  • shemale in sexy panties pics
  • sexy women playing with their boobs
  • porn asian black indian
  • gianna michaels interracial galleries
  • asian mom son porn
  • hardcore girls photo pictures
  • beautiful blonde with black lover
  • amateur senior women sex
  • latin chicks fucking black dicks
  • xxxdesert free sex galleries
  • young latina fuck video
  • son eating moms pussy
  • How to finger pussy video
  • drunken naked girls hardcore
  • girls gone wild videos unsenserd
  • petite tiny tits xxx pix
  • Hard sex pussy
  • nude asian beach girls
  • mothers teaching girls sex porn
  • worlds sexiest beaches bridget goes topless
  • Hot naked girls dancing
  • facial abuse free videos
  • free shemale futanari cumshot pictures
  • petite teen xoxox
  • black chicks in bondage
  • Pretty young black girl
  • russian hairy anal teen tubes
  • babes in black lingerie movies
  • mature lesbians free picture archives
  • Top sex movies
  • Hot naked lesbian video
  • xhamster older plump black woman fuck
  • joanna garcia porn videos
  • obs uk xxx mature jan
  • Sexy teens skirts
  • wife husband debts black
  • Albanian sex video
  • black girls with fat ass
  • petite blonde cheerleader
  • Sexy naked lesbians fucking
  • page girls pictures
  • nude wife stripping vids
  • black pussy lip pics
  • free shemale domintion sex movies
  • pictures of shaved pussy spreading
  • japanese sex orgy
  • Free online teen sex videos
  • free naked videos of couples
  • annie clancy anal sex
  • hellsing sex video
  • asian sex free trailers
  • a thing of beauty analysis keats
  • xxx teenager girls
  • lesbian teen red head porn videos
  • South indian housewife sex
  • skiiny black girl sex
  • teen amatuer pussy close up
  • sex movies meen and nasty
  • free hot sex moves with lesbians
  • teen pussy porn movies
  • teen boys and teen girls fucking
  • couples marriage pictures
  • sex neighbors wife falls church va
  • teen family sex pics
  • japanese girl shaved
  • free nude pics of girl jailbait
  • best full sex movies
  • asian free sexy video
  • shemale armitage videos
  • lesbians free sex video
  • slim black pussy pics
  • amateur home made sex videos
  • ebony nudes pics in forum
  • hollaback girl japanese
  • wife streaming sex videos
  • curvy blonde teen
  • hot sexy older mature videos
  • naked black girl pins man
  • mature black female nude
  • swingers sex jelsoft enterprises ltd
  • amateur stickam webcam girls
  • hot sexy young teen virgin
  • free anal toy movie
  • teen boygirl sex vids
  • erotic digital teen
  • Latina huge boobs
  • femail orgasm pictures
  • nasty blowjob porn pics and vids
  • hamsters mature big tit porn videos
  • Hot indians sex
  • sex hardcore free pics
  • krystal forscut naked pics
  • gallery girls bound
  • free video audio girls
  • what pronstar has the tighest pussy
  • mature wife naughty video
  • hot blonde southern girls naked
  • best sybian sex video
  • teen xxx les videos
  • free tiny tight teen sex video
  • the best pussy i ever ate
  • home sex videos porn sensitive
  • young african girls nude
  • big dick shemale fuckers
  • homemade videos of mature moms fucking
  • orgasm tube movies
  • wife loves to suck my dick
  • homemade porn wifes interracial breeding session
  • nice curved bikini women pics
  • old lady fuck with young boy
  • i lesbian sex videos
  • nonnude girl pics
  • Does anal sex feel good
  • sexy mature black women photos
  • free pics of babes in ligere
  • open pussy soaked tight
  • songs from jack ass the movie
  • Best lesbian sex movies
  • new porn sites xxx
  • petite girl fucked hard outdoors
  • sexy naked mexican girl
  • nude virgins video
  • young japaneese school girls
  • topless movie thumbs
  • asian schoolgirls in uniform
  • wifes pussy gets abuse by friend
  • full lesbian teen sex movies
  • videos of pussy waking
  • xxx huge teen tits
  • bulgarian bitch xxx
  • deep adult ebony anal
  • sharing home made sex videos
  • obama cheating wife video
  • Huge teenage boobs
  • cool teen movies with sex scenes
  • black women hair and photo gallery
  • Sexy ass in tight jeans
  • naked oops gallery
  • pregnant girls make adult movie
  • soft porn ebony
  • hot professional asian girls
  • german goo girls blonde
  • girls age pussy
  • sexy european blonde
  • female prisoner porn videos
  • cum anal naked
  • georgia teen pussy
  • girl next door hardcore
  • free mature pantyhose galleries uk
  • solo just girls boobs
  • Hot thai girls video
  • black ladies sex free view
  • hairy pussy sucking
  • fat ass black lesbos
  • black gloved girls
  • sexiest blonde woman
  • naked schoolgirl ass
  • pizza delivery girl anal fuck
  • black bbw tits
  • asian theme appetizer party
  • free hardcore home fucking videos
  • very fat sex movies
  • sexy mature soles
  • young girl old xxx
  • amature wife share fuck
  • wife rough sex tubes
  • Big breast of india
  • young young teenage girls naked
  • Vietnamese models nude
  • teen seducing mature for lesbian
  • fat fat fat women pussy
  • feminist porn gallery
  • fuck my dick video movie
  • haley wilde sex movie
  • best sex with my wife
  • english soft porn movies
  • girls get picked up reality
  • mature movie tube galleries
  • pissing inside of pussy
  • highschool fuck pics
  • japaneses teens white pantie thumbs
  • amateur anal tv
  • slut wife stories and pics
  • teen erotica porn gallerys
  • gloria vanderbilt petite size amanda jeans
  • slutload massage erotic lesbian
  • hardcore black pussy and white guys
  • cumming inside her pussy
  • sexy latin teens
  • lesbiam mature young tube
  • young teens in bikinis
  • hottest shemale sex movies
  • mature english escort videos
  • black perfect boobs
  • fuck videos force dominance
  • immature young girls
  • free hot young teen porn sex
  • sexy young ebony pics
  • dare group sex
  • free shemale video porn
  • ebony handjob videos
  • lesbian girls sucking tits
  • free long videos with creampie sex
  • teen hardcore anal xxx
  • old man young teens porn
  • party dresses girls
  • share fuck wives
  • free daily sex video gallery
  • French models nude
  • free amateur natural galleries
  • Ghetto girls xxx
  • asian transexuals in panties
  • milf fucks daughter black boyfriend video
  • young girls swimming naked
  • Love big tits
  • black snake moan pictures
  • young lesbien porn pics
  • free porn videos amateur long
  • nice legs japanese girls
  • amateur thrity year old underwear pictures
  • tribal real sex rituals pics
  • black xrated movies free
  • whats sex with black woman like
  • pool sex orgasm
  • xxx vintage porn tube movies
  • indian girl naked photo
  • young girl anal mpegs
  • free teen girls in panties
  • black bbw pussy pics
  • lesbian teen slut
  • How to masturbate female video
  • izle shemale sex
  • free fake force sex videos
  • big tits cock sucker videos
  • amateur texas girls
  • candid bikini nude videos
  • hot asian girls french kissing
  • cartoon girls fingering
  • jem stone free porn movie
  • licking pussy lesbos
  • Hot naked girls with big boobs
  • rough black female anal
  • sexy redhead babes
  • asian moms pics
  • Thick white girls dancing
  • sexual orgasm g spot
  • sex machine free video
  • ebony scoreland xxx
  • Asian nude pictures
  • sex on asian wmv
  • asian large dildo video
  • orgasm faces and videos
  • adult sex pussy close up
  • free hardcore amature porn tube
  • deep throat tube young
  • retro classic pussy eating videos
  • rough home sex videos
  • wife rough interacial sex video
  • old couples sex with teens
  • Teen anal porn movie
  • latina black sex pics
  • fussian naked little girls
  • old fuck videos
  • keeley hazells spread pussy
  • Naked girls in showers
  • amateur submitted party pics
  • womens beach volleyball bikini pics
  • suckin girl dick
  • muscular women with big boobs pictures
  • audio for pussy dolls beep
  • free double penatration anal videos
  • la petite ferme wallpaper
  • free download of short sex videos
  • black naked free
  • three blonde lesbians fingering and fisting
  • teenager girl models
  • mature cartoon tpg gallery
  • licking pee pussy
  • Rica peralejo sex scenes
  • couples amateur videos
  • divine breasts full videos
  • my first sex teacher ebony
  • Lesbians sex positions
  • asian girl femdom s
  • black rouge cops free movie
  • sucking on little girl boobs
  • brazil amateur porn video
  • little girls naked pussy
  • brittany spears pussy flash
  • young couples sex exgirlfriend
  • house wife threesome galleries trailer
  • pics of student sex
  • ulrika johnsonn topless pics
  • pro photography oh couples having sex
  • Sexy nude images
  • cassidy blue gang bang squad video
  • girl fucks new neighbor sex
  • petite spreading pussy
  • clebrity topless videos
  • young girls showing their bare feet
  • the fuck teens
  • free large mature galleries
  • top anal pornstars
  • sandra bullock pussy shots
  • free cunt sex mum wife public
  • why do women enjoy anal sex
  • young erotic teens nudes
  • free video better sex
  • pic of black sex teen
  • oral sex explicit instructions free video
  • free teen school girl sex videos
  • Drink girl sex
  • brazilians getting fucked hardcore on xvideos
  • petite video amateur cul gratuite extrait
  • frre pics of women haveing sex
  • mom sex free videosw
  • fuck young teen movies
  • mile school girls boobs
  • men in pain free erotic video
  • permanent increase with pussy pump
  • college girl at party
  • Black ass gallery
  • teen pantie porn pictures
  • strapon lesbian threesome pics
  • free videos teen blowjobs seductive
  • ex girlfriend big tits
  • monstrous sex movies
  • mature sex tubes vids
  • free full length frat sex movies
  • portable shower orgasm video
  • bikini sex torrent
  • nude girls shaved pussy pictures
  • Hot teen perfect tits
  • big boobs hentai pov video
  • ass sex teen year
  • tillicum kills trainer amateur video