One of the main reasons why it’s difficult to troubleshoot problems in complex environments is lack of documentation. This post is long overdue because it was inspired a while ago by Dan Fink’s post (again). Be sure to read it first.

From time to time, I see situations when one person leaving the company leads to a disaster that puts in danger the whole existence of multimillion businesses. Why? Because there is nobody else who knows how system works and how all its moving parts are connected. How does that happen? No knowledge sharing.

The most universal method of knowledge transfer is documenting stuff and I would argue that, most of the times, one hour of documenting brings to a business more than two hours developing new functionality in PL/SQL.

Actually, there might not even exist such know-it-all-wizard. Complex systems are not created in a matter of days or even months. Often, extreme complexity is the result of years or even decades of evolution. All those workarounds, bug fixes, re-designs, pressing deadlines — those friends of entropy win time from documentation tasks.

How many times it happened to you:
- Why are there forty-two underscore parameters in every new database you create?
- Because it’s our standard.
- Can I look at it?
- {proudly} Yes. It’s in every init.ora file we have!

Or another variation:

- Can I look at it?
- {proudly} Yes. Here is 42 pages document how to create new database:

Performance boosters:
_db_aging_touch_time=…
_db_aging_stay_count=…
_db_writer_chunk_writes=…
_discrete_transactions_enabled=…

Bugs:
_kgl_latch_count=…
_kgl_bucket_count=…
_lm_rcfg_timeout=…

and etc…

- When was this document started?
- {proudly} Oh! Forty-two years ago!
- When was it last time updated?
- Eh… I’m actually quite new here…

What are the reasons behind missing or inadequate documentation?
Just to name a few:
- It’s boring.
- We have no time for it now and it’s not a priority (it never will be) — we still missing couple reports.
- I’ll do it tomorrow.
- Project manager is not paid by number of pages of documentation produced (not that this is a good idea either).
- I’m the only one who knows this system — they must be idiots to get rid of me.
I’m sure readers can name plenty of others and please feel free to do so.

Sometimes you can avoid complexity and make systems simple by design but in real life it’s more often when you need to deal with complexity. Paraphrasing Dan — documentation will not make systems simpler but will reduce perceived complexity.

If you’ve got to support and/or develop a complex system — how do you make sure you dedicate enough time to documentation? What are the strategies to keep it up to date? How to explain the importance of documenting to the management? If you have anything to say — the comment box is at the bottom of this screen — use it. If you think that you have enough to say for the blog post — just let me know.

Find more about  , , ,

7 Responses to “Avoiding Guesswork in Complex Environments”  

  1. 1 Christian Antognini

    Amen!

  2. 2 Frits Hoogland

    Good points Alex! Exactly the last situation I was partly involved with.

    In my experience, setting up a successful system, either complex or simple, requires a systematic approach from the start. And, which cannot be said enough, simplicity!

    A database needs to be adjusted as little as possible. Sourcecode needs to be version controlled, Even in very small simple environments. Production environments need software releases, explicitly tagged from the version control. Not a few files to be fiddled with in order to make it work again.

    Happy easter!

  3. 3 Narendra

    Alex,

    Nice thoughts.
    I agree completely.
    Your last reason is very important. People, many times, do not share knowledge, due to insecurity. Sounds funny, but it is a fact. They (probably) get “married” to the company and whenever they are acknowledged as S.P.O.C. (Single Point-Of-Contact), it soothes their ego.
    Such people will do everything they can, to sabotage any plans to prepare / update documents.

  4. 4 Aleksey Tsalolikhin

    Documentation is very important and under-used in today’s IT shops.

    To help remedy that, here’s my recent post on system administrator site documentation with docbook, and a few “intro to docbook” links that were helpful to me.

  5. 5 Alex Gorbachev

    Good point. Thanks Aleksey.
    The biggest difficulty I found is to motivate engineers to write docs AND keep them up to date.

  6. 6 Graham Oakes

    Excellent points Alex.

    I would also add that writing the documentation is only useful if you ensure it’s properly distributed or can be easily found. Finding the correct location for the documentation and ensuring that it’s organised in such a way to make retrieval easy is not as simple as it could be in a large company(at least that’s what I’m observing where I work atm).

  7. 7 Alex Gorbachev

    Thanks Graham. I might be biased or just used to Oracle Database documentation style but after comparing Oracle database documentation with documentation for the products that Oracle acquired — I can only attest to your point. The ability to navigate quickly and effectively within a large documentation volumes is a must.

    But it’s not only the documents creators who should invest efforts, readers must invest enough to be familiar with organization and knowledge domain. Even when search functionality works great, you would still need to know what to search for.

    Thanks again for the feedback.

Leave a Reply


  • dry creek petite sirah
  • nude young girls movie
  • petite teenager forum
  • amatuer wife orgasm
  • Can you get stds from oral sex
  • topless pictures of helen flanagan
  • rate my wife sexy
  • free chubby chicks in clothes galleries
  • suck blow pictures
  • kerry fox blowjob video
  • pictures of a young hitler
  • fat black as xxx
  • close up young pussy
  • friends mom sex pictures
  • black eyed susan seed pod picture
  • Huge big asses
  • black shemales on black shemales movies
  • animated teen girls
  • crazy girls bare naked hotties
  • japanese tied up girls
  • fuck teen girl asses
  • teen nude pussy video tube
  • mature xxx porn tube
  • girl ass crack thumbs
  • Lesbian sex scence
  • lovely wet brown pussy
  • free mature full lenght videos
  • pictures of pornstar iga wyrwal
  • teen cutie is seduced by swingers
  • french terry misses petite pants
  • titty flashing compilation videos
  • amateur black shemales
  • carie prejean sex video
  • ass closeup pics
  • white dicks in black chick video
  • mature and black cock tubes
  • blonde girl in white dress ghost
  • black gals fingering
  • video amateur free
  • disney character porn vids
  • stories anal teen xxx
  • baylee asian porn
  • dorothy oz porn naked cartoon wizard
  • fort smith arkansas pussy
  • family sex fuck tubes
  • wife a secret lesbian
  • naked girl spanking
  • emotional orgasm video
  • hot pussy fuck porn
  • Lesbian threesome sex
  • teen lesbian femdom pic
  • hot naked man sex
  • porn young french girls
  • black butt slutls slutload
  • naked early teen boys pictures
  • mature lesbiens and young girls
  • Lesbians sex vidoes
  • free extreme interatial gang bang videos
  • Verne troyer sex tape pictures
  • blow job free japanese girls
  • free hardcore xxx movie
  • black cock pink pussy
  • hot young sexy ladies
  • pussy in the rain
  • hannah montana bikini pictures
  • interracial island sex porn tube
  • free young blonde movies
  • naked blondes pics
  • cartoon ass porn gallery
  • naked erotic teens
  • shemale anal stretch
  • xxx massage of a dick pussy
  • teen girls wet daipers
  • shemale sex archive
  • free teen porn thumb movie gallaries
  • young teen amature on cam
  • handjob sex video tube
  • lesbian sex vidoes
  • sexy video of hot girls
  • hardcore party clubs ny
  • younger babes porn amatuer
  • liz claiborne petite pants
  • erotic silent movies
  • china girl bikini
  • Hot sexy young lesbians
  • nude young oriental girls
  • gang bang big tits movie
  • Sex fucking boobs
  • nothing butt teens movies
  • candid webcam girl pics
  • black girl on line
  • ny giants girls flashing
  • Nude pictures of eva green
  • all asian pussy
  • husband spys on wife video
  • girl point of view sex
  • pussy hair red video
  • old lady fuck movies
  • preparing for anal sex video
  • Sexy girl tummy
  • Hot sexy collage girls
  • amateur video download
  • sexy girl tongue video
  • fuck me chuck movie
  • sex of black pussy
  • stephanie kramer in a porn movie
  • old guy young girl porn videos
  • flexible slim sexy babes
  • naked pics paris tale
  • all the way naked girls
  • blood sex virgins
  • strip video complete teen white
  • young female learns to fuck
  • homemade housewife anal
  • party fuck up video
  • black hot sex teen
  • fucking pussy pon
  • nude bikini model video
  • Bridget marquardt pictures nude
  • naked ex girlfriends free videos
  • free lesbian lick pussy
  • girls eating girls xxx preview
  • teens only bikini pics
  • more teenage girls into fisting
  • guy aroused girlfriend porn
  • hardcore sex with shift stick
  • fine ass tis and pussy
  • suck little dick
  • younger hot teens
  • xnxx free sex movies
  • real wife homemade videos
  • seveteen girl blowjob
  • Sex for couples
  • sex free pictures
  • amateur sex ex girl
  • slose up pussy
  • teen disgraced video
  • teen sex audition fuck video
  • video girl fucks
  • all black porn movies
  • hot latina pussy videos
  • muriel montosse group sex video
  • free hardcore sex video mpeg
  • amateur porn tube mature
  • amateur pussy orgasms
  • asian girls naked
  • megavideo anal tube
  • lesbians with natural huge tits
  • puckett young girl
  • indian free sex girl pictures
  • Tits and ass thumbs
  • mature pussy fetish
  • sex stories of little girls
  • ethnic pussy tv
  • Hot lesbians licking feet
  • free female nude butt pictures
  • Lesbian hairy pussy
  • tiffany mulheron naked pictures
  • clone a pussy
  • petite teenager real teen models
  • hardcore mature lesbian strap on sex
  • girl greasing ass
  • love making movie porn
  • hunting young college girls
  • naked pictures of celebrites
  • show galleries babes
  • girl handcuffed and fucked porn
  • free young close up pussy pics
  • kim kardashian playboy pictures naked nude
  • Girl vagina pictures
  • dead girl sex video
  • you porn lesbian in car
  • elderly fuck partys
  • sexy girls with finger in vagina
  • panty mature sex
  • black history videos online
  • three blonde girls posing nude
  • Penis inside vagina diagram
  • amateur porn stolen video
  • video ameteur sex
  • Sex in girls locker room
  • sexy huge pics of naked girls
  • old girlfriend porn films
  • little fuck doll xxx
  • Sex girls hyderabad
  • girls doing bad stuff naked
  • free very young porn tubes
  • real wife fucks neighbors video
  • suck pictures anal tit
  • legal blonde videos
  • homemade biker chicks porn
  • light skinned teen girls
  • bizarre fuck movies
  • free transexuals porn videos
  • young virginz pussy galleries
  • petite mal seizures in ren
  • asian creampie gangbang blackcock
  • ashleys candy anal videos
  • Sex actress photo gallery
  • gallery leaf porn
  • Little asian tits
  • loretta young home movies
  • real girls naked groups
  • blonde hair with red highligts pictures
  • hot girl speading pussy
  • blonde british babes porn
  • Indian gujarati sex
  • young teens fuck older women
  • free pics babes topless
  • best erotic movies of all times
  • blonde girlfriend porn
  • video streaming tits big lesbian american
  • amateur adult video orgy
  • hot blonde great sex
  • sexy large pointy breasts
  • man eating pussy gallery
  • humiliated punished nude teen girls stories
  • naked babysitter pics
  • Sexy ass ever
  • sexiest teens naked
  • cool pictures of the powerpuff girls
  • hot college girl breasts
  • white girls interracial pregnancy
  • naked girls with nice boobs
  • Imvu naked girls
  • Teen cheerleader porn
  • Herpes oral sex
  • girls masturbating on webcam
  • shy wife sex auditioin
  • my wife cheating black pics
  • chubby sexy nude teens
  • porn on blackberry
  • korean wife swap
  • forum lesbian sex
  • girls next door kendra porn
  • redt tube young pussy
  • agota castings sex videos
  • sexy naked butt gallery
  • anal teen angels free movies
  • sex with anorexic teen video
  • tight sexy babes
  • imogen big tits pics
  • young girl sex pree
  • Beautiful girls fuck
  • free porn biggest dicks
  • homemade amateur real sex tapes
  • keeze movie porn
  • Indonesian girls sex
  • girls at sex party
  • video free wife
  • cover girl xxx
  • Teen hairy armpits
  • mcgraw davidson reality
  • black dick distroy ass
  • teen girl stripper
  • gang bang friends black mom slutload
  • Sexy desi sex
  • young ameater teens have sex
  • Arabe lesbian teens hijab anal free porne movie
  • anal granny movies
  • daily nutrition for teenage girls
  • sex toy home demo party
  • pussy to small for dick
  • naked hogtied girls
  • amateur nude teen webcams
  • vary young porn s
  • free wild porn video sleep
  • pictures of young breasts
  • post amateur porn photos
  • novelty small trees weeping pussy willow
  • asian extreme videos
  • buy teen tranny porn movies
  • free sex fucking young videos
  • teen girls sex site
  • wet panty cameltoe ass pics
  • video francais sexe amateur b randi
  • amateur xxx casting
  • economy analysis after black friday
  • Cheerleaders nude video
  • sex fuck pussy ass
  • black guys on white girls fucking
  • free video teen sex porn
  • free pussy spanked videos
  • exercise teenage girls
  • ebony male butts pictures
  • canada bbw teen who seek sex
  • Teen beauty pagents
  • beach babes sex naked
  • hot blonde teens girls
  • hommade sex movies
  • very young girls cloth
  • photos of anal sex damage
  • daphne fuck scooby idrawporn
  • skinny teen girl porn
  • sex toy party rochester ny
  • girlfriend pillows with breasts and hole
  • ebony love cum in there pussy
  • pornstar friday movie list
  • naruto sexy jutsu with boobs pictures
  • free streaming videos of pornstar tinkerbelle
  • fine teen ass pics
  • pictures of old fat tits
  • young blonde tight cummy pussys
  • Get wet pussy
  • oral teens galleries free
  • Teen tits bounce
  • wide hips natural nude erotic wife
  • teen gy style fuck
  • pakistani girls tits
  • sex virgins annal
  • free onion booty porn videos
  • black girls fucked by white cock
  • pictures black mathematicians
  • smaal virgin pussy