Oct 05 2009

The Next Five Years of Storage

[NOTE: This essay was commissioned by a client in December 2006. It's the second in a series of old-yet-relevant position-papers whose exclusivity has expired, that I'm editing and posting. Things for the next five look "similar". There is no formal "conclusion", as this is one section of a larger piece.]

Over the next five years, gross storage needs will double every other year, sparked by industry trends that avoid deleting anything, ever; continued bloat in software programs; increased user demand for larger-file storage; increased user demand for indefinite storage; increased user, corporate, and industry expectation of system-side backups and frequent snapshots; and the enabling factor of meteoric-disk-size -to- paltry-disk-cost ratios.

Since the late 1990s, we have seen rapid acceleration of infinite data life. While storage vendors will use terms such as “information life-cycle management”, “information archiving” or “data warehousing” – they all converge onto the premise that corporate data life is no longer finite. The value of this is dubious, but irrelevant to argue: financial workers expect to be able to look at historical data for modelling purposes; draft and product workers expect to be able to look at long-dead projects that might now be of value with new knowledge; in the throes of bankruptcy, competent managers (and lawyers) will want to mine the archives for something… anything that may provide some value. Everything your organization has ever known is expected to be retained, indefinitely.

The average 10-page MS Word document in 1995 was 13K in size. The average 10-page MS Word document in 2006 is 1.4MB. While that size may still seem small, it’s indicative of a growing trend of software generating vastly wasteful content because they can. Software vendors don’t need to worry about their data fitting onto floppies anymore, so they don’t. Multiply this across dozens of applications, add in media, and you have truly huge data files with only a few pages of actual content.

Similarly, the users want ever-larger files. Gone are the days of compressing graphics, video and audio to the Nth degree: users want full-quality content. They don’t want a 120×120 “thumbnail” video, they want something that takes some real-estate on their oversized monitor. As bandwidth increases, so will the user-desire for better content faster. They then want to save that same content to their network volume. They want it backed up in case of catastrophe (or their own error). What was a 3MB MP3 file is now a 45MB FLAC or WAV file sitting in your database.

The increase in user-end space (desktop harddisks) has led users to demand not only more and more space from their storage providers, but also indefinite storage. Users no longer have to selectively delete their e-mails to stay in a predefined space, so they keep them all, forever. They expect the same from the rest of their digital attics: they expect every bad poem, doodle, patent-idea-on-a-napkin, picture of their grandkids, etc. to be immediately available, forever.

Forever. Even if your disks die. Even if they accidentally delete them. Even if a meteor pummels your datacenter. The old standard of weekly backups have long passed the borders of Being Prudent, travelled through the Fields of Marginally Acceptable, and have entered the Mountains of Irreparable Harm to Your Reputation. Users, customers, regulators, etc. are barely tolerant of losing a day of data, and this will get worse. In the next half-decade a truly monumental shift into multi-media backups, near-real-time data snapshots, and 100% protection of data assets will be fully realized, requiring several multiples more mixed-media backup storage than live data storage.

On the up-side, disk sizes are sky-rocketing, costs are plummeting and the reliability of the new serial ATA (SATA) architected drives have come up to a level that allows anyone to build in or expand networked disk with a trivial investment. A new generation of storage vendors are coming up and challenging the old way of thinking about networked storage, and adopting technologies with more agility than their behemoth competitors. We’re quickly on our way to 1TB disk drives, flash-based storage continues to be refined and is nearing enterprise-grade, holographic storage is being commercially realized for some applications, and all of these technologies are driving the cost per megabyte down.

Sep 29 2009

Ruminations on “Has the Glory Gone Out of Working In IT?”

A recent Ask Slashdot pondered the following:

… writes to wonder if the glory has gone out of IT… Has a more pervasive technical culture forced our IT gurus into obsolescence?

This question is unfortunately very loaded, but I’m going to step into the minefield: The only segment of IT impacted negatively by this evolution are user support staff. The image of the Helpdesk technician riding the Silver Steed of Only I Can Solve Your Problem and being lavished with riches, lovers, and their own parking spaces went the way of the dodo a long time ago… Probably about the time Yahoo! hit their stride.

In truth, the more pervasive an adept “the users” are to technology, the more “IT gurus” are needed. What has changed, is where the bar to be a “guru” is set. It used to be that anyone who could add RAM to their own computer was labelled a guru by neophytes. As IT has evolved, it has joined the ranks of other venerable classes in that your proficiency (or guru-ness) can only be measured by qualified peers. Thus impressing the luddite by showing them how they can just hit Ctrl-B to make things bold, and having them fawn all over you doesn’t count.

But it does. User support has been increasingly marginalized as the user bar has gone up, but it is still exceptionally important, and will remain so for quite some time. Ineffective user support leads to all sorts of problems within an organization, and excellent user support enables the entire organization to evolve and move forward without tripping on the little things.

The advice I give to people who have become sullen about their profession is the same that I give to everyone: Evolve your skills. If you’re surprised that the same skills you had 15…10…5 years ago aren’t getting you the love today, tough. Unless you’re a philosopher, theologian, or geologist, your skills need to evolve. You’ve chosen a profession that isn’t static: you’re not pondering the past in IT, you’re supporting the future.

Sep 28 2009

Love Is The Hundredth Giftshop, Curse A Meridian

As of Saturday, I have seen The Tragically Hip play live 13 times- that’s 1 more than I’ve seen Alanis Morissette- which means they now hold the trophy as the Most Prolific Live Artist In My Life. I’m sure Gord would love knowing that.

I’ve seen a lot of concerts, and only really bother writing about the exceptional ones, but wow- The Hip at the NAC on Saturday was hands-down one of the best shows I’ve attended. Mid-sized venue, great seats, the usual antics, ringing ears the next day (I didn’t need that auditory spectrum, anyhow), and one horrendous case of vertigo (my left ear is telling my brain I’m falling to the right, which causes me to fall to the left trying to compensate) today: All the hallmarks of a great live concert!

Sep 25 2009

The Next Five Years of Bandwidth

[NOTE: This essay was commissioned by a client in December 2006. It's the second in a series of old-yet-relevant position-papers whose exclusivity has expired, that I'm editing and posting. Things for the next five look "similar", yet scaled up in some areas. There is no formal "conclusion", as this is one section of a larger piece.]

Over the next five years, datacenter bandwidth will level off for a bit. With the 10GigE standard behind us we can finally pull our backbones up to a level where they’ll be able to breathe easier for a while. Storage speeds are still being gated by the storage devices themselves, and until either solid-state media becomes cost effective or disks rotate twice as fast as they are now, that isn’t going to change much. Aggregating virtual systems is actually causing an interesting bandwidth phenomena that I’ll address later. Regardless, a 10Gig, or Nx1Gig backbone should be able to breathe well for the next half-decade. Planned year-over-year demand increases of 5-7% should be expected.

Desktop network speeds have been about the same for the last five years, and will largely remain unchanged. A 32-bit computer system running a commercial desktop operating systems has too many architectural limitations, still, to be make use of more than 60-85Mb/s of bandwidth. While some vendors are running 64bit processors, they generally are using bus architectures that aren’t that wide, thus gating peripheral speeds back to 32bit. In the next five years that will clean up a bit, and 64bit “extensions” to the 32bit processors will become more common place, but still not impacting the network noticeably due largely to OS and bus architectural issues.

Environments consolidating onto virtualized systems are seeing an interesting gross decrease in datacenter network bandwidth use. Not surprisingly, they’re also seeing peak utilization well above what they had prior to consolidation. The latter is easily explained by virtualized systems generally “netbooting” their OS from the storage network or a bootserver, and now more than ever embracing networked storage completely. The gross decrease has been unexpected because of the higher demands on the network, but is explained by architectural constraints. We’re now seeing 10-15 virtual servers sharing one or two network connections, where previously each had one or two of their own. This has somewhat of a levelling effect on network use, but isn’t dramatically impacting service performance as one would expect. The network is more important in these environments, but as a whole not as taxed.

It was largely believed that mobile “broadband” availability and use would be much higher by now, but we have yet to see a real platform for use. The Palm Treo series is getting an overhaul “soon” and rumored platforms by Google and Apple may change that landscape. In general, even if fully realized, the network demands by these users will largely have no impact on the greater network, or on datacenter network needs. The next-generation, “4G”, will be changing that, but I don’t expect to see that kind of horsepower in a phone until late-2010-to-2012: the processors are still just too slow.

What will change dramatically will be the bandwidth access for remote users. While not directly impacting the datacenter we’re going to see dramatic growth in the cable/DSL/satellite “broadband” space. Internet-facing applications may see a 20-30% rise in client demands as users become less tolerant of waiting for application loads due to their expectations of “faster” service, on the order of 200-250% more bandwidth. It is expected that OSP asymmetrical provisioning will continue.

Sep 24 2009

Disposable Appliance Computing

[NOTE: This essay was commissioned by a client in February 2007. It's the first in a series of old-yet-relevant position-papers whose exclusivity has expired, that I'm editing and posting]

The hosted systems industry has turned another critical point. Several years ago we eschewed large mainframe systems in exchange for commodity servers that could divide load and work together to provide services without single-vendor lock-in and without a single piece of “iron” waiting to fail. The computing power of a $2,000,000 mainframe was dwarfed by the implementation of $80,000 in commodity hardware. With virtualization coming-of-age- with Intel and AMD putting hooks into their processors and chipsets to allow virtualization to be fully realized and not just a software-only hack- we’ve seen those same commodity systems hosting dozens of virtual systems reliably and at near-metal efficiency. The cost per virtual system is a number rapidly approaching zero.

New offerings from Sun, IBM and HP/Compaq are emphasizing something that “the server guys” haven’t needed to care much about: infrastructure. Historically, your network engineers and analysts worried about interconnection, route redundancy, and ensuring the bits could flow where they needed, reliably and sufficiently; and your system engineers worried about everything up to the point the bits hit “the network”. Moving forward, that is almost a debilitating dichotomy. Traditionally, in the post-mainframe era, a physical system did one or two things and its exclusion from the network or its under-performance on the network was a minor issue. With a physical system possibly hosting dozens of virtual systems- all with unique networking requirements, cross-talking requirements, and of course: networked storage requirements- your system engineers must be well-versed in network engineering. “The Network Is The Computer” is not just a Sun tag-line, or a lame cliche’. We’re now fully realizing the potency of that statement. Every system offering from the Big Three contains significant “infrastructure” features: Network features.

By pushing more and more network features into server systems- IBM servers with Cisco “swrouters” built-in, for example – the server itself has become more important and less relevant at the same time. Keeping it up and running well will require a new kind of system engineer because “the box” is now more complex: But at the same time, collections of “boxes” should be able to self-heal and adapt to the failures of others. Each system has now become disposable.

A large swath of the architectural literati are already deploying quantities of self-healing farms that take over the work – the very virtual machines – of failed or failing physical systems. Virtualization on its own wasn’t a game-changer. Virtualization with processor support and recognition sparked real potential. Virtualization on top of “infrastructure”-aware (e.g. heavily networked) physical systems has dramatically shifted the value of hybrid “networked systems engineers”, raised the bar for the “server guys” to get up to speed on the real internals of networking, and has provided the unprecedented opportunity to deploy redundantly resilient systems that can in-practice achieve five-to-seven “nines” of reliability.

Sep 13 2009

Anniversary Dinner

My parents’ anniversary was September 10th. While their son is a [culinary] genius, they have only allowed me to cook for them about three times in my life. My mother doesn’t approve of all sorts of things I do, including quite a bit in the kitchen. Well, anyhow, this year they accepted an offer for me to take care of their celebratory evening. Below are the recipes.

Appetizer: Wine Poached Pears

Recipe was from an old friend of mine (from memory), melded with one from a cookbook for precision.

1 1/2 cup red wine or burgandy
1 cup raw sugar
2 tbsp lemon juice
2 cinnamon sticks
6 cloves (whole)
4-6 pears (whole, peeled)

In a pot combine all ingredients except the pears over med-high heat until boiling. Reduce to low and simmer, covered for 3-5 minutes (should be noticeably thickened). Lop off the bottom of each pear so they will sit upright on the plate for serving later.

Add the pears on their side if possible, and cook uncovered for about 15 minutes or until tender, turning frequently.

Remove the pears to small serving plates, upright. Bring reserve sauce to a boil, uncovered, over med-high/high heat, until reduced to a thick glaze, ~5-8 minutes. Remove cinnamon sticks and cloves with tongs. Drizzle over pears and serving plate.You may serve chilled if climate dictates.

Serves: 6. Prep time: ~2 minutes.Cook time: 25 minutes.

Salad: Goat’s Medley w/ Walnuts

I can’t overstate the value of baby spinach, dandelion greens, and clover sprouts. Simply the best salad you can make. The ratios can be varied for taste.

1/2 lb baby spinach
1/2 lb dandelion greens (shredded or chopped)
1 bunch radishes (sliced, chopped, or shredded)
2-3 carrots (sliced or shredded)
1/2 lb sun-dried tomatoes
1 1/2 cups freshly cracked walnut pieces

Combine. Toss. Serve with your favorite version of raspberry vinagrette.

Serves: 6-8 humans, or 1 goat or llama. Prep time: 20-15 minutes.

Entree: Peppercorn-rubbed Filet Mignon

Purists and book-wise chefs would crucify you for rubbing down a tenderloin filet with anything more than simple salt. I’m neither.

Peppercorn Rub

Regular peppercorns can be subbed-in here, but it changes the dynamic. This rub is great on any cut of beef. Double, triple, etc. as needed. Keeps well in sealed container for several weeks. Omit sugar until just in time, to keep for several months.

2 tbsp szechuan peppercorns (whole)
2 tbsp coriander (whole)
1/2 tbsp raw sugar
1 tsp allspice (whole)
1 tsp sea salt

In iron skillet, over grill or med-high burner, cook szechuan peppercorns and coriander until you can smell them well, about 2-3 minutes. Remove from heat, combine with rest of ingredient in a food processor or spice grinder. Pulse-grind until evenly-blended but coarse.

Coats 4-6 filet mignons lightly. Prep time: ~2 minutes. Cook time: 3 minutes.

Grilled Filet Mignon

If you don’t know the difference between tenderloin cuts, that’s ok. Filet mignon is the tip of the tenderloin: very small, and very pricey. I recommend 2″ cuts because that yields about 7 oz of meat. If you can’t afford all mignons (will run you about $50 for 6 little cuts) then you might want to purchase a whole tenderloin, which any reputable butcher will gladly slice up for you: 1″ cuts of the upper tenderloin with a single 2″ mignon at the bottom will run you about $40, but net 6 good-sized steaks plus the mignon. Do not buy packaged filets wrapped in bacon. If you’re going to pay for tenderloin, make sure it’s being chopped fresh just for you.

4-6 2″ thick filet mignon cuts of tenderloin beef
or
4-6 1″ thick upper tenderloin cuts

Preheat grill on high. Trim visible fat from filets. Rub down filets by hand with peppercorn rub, coating evenly but lightly. Turn grill down to medium. Meat should be flipped every 5 minutes. Should be served medium-rare (135 degrees), medium (145 degrees) at most… but well-done brutalizations are also possible (165 degrees). All temperatures are taken at the center of the meat.

Serves: 4-6. Prep time: ~5 minutes. Cook time: varies by thickness and doneness, 7-25 minutes

Dessert: Raspberry Tart

This lovely was made from scratch by my lovely, following a copyrighted recipe I can’t legally reprint here. Excellent ending to a great meal.

Sep 10 2009

Ruminations on “The unspoken truth about managing geeks”

Jeff Ello wrote a great piece in Computer World about managing “geeks”. It’s getting a lot of press, good and bad, and really sparked me back into this conversation. If you replace “geek” and “IT pro” in the article with “knowledge worker”, it’s almost eerie how in sync our opinions are. Jeff and I both have a lot of dual experience- seeing organizations from the inside-out as employees, and seeing them from the outside-in as consultants. I’ve written a lot about my pinings for knowledge workers, including an essay here. Jeff’s editorial really takes a good, honest jab, and one I want to “Amen” as loudly as I can:

AMEN

I really want to call attention to some of the finer, most important, high-impact parts of his essay. The focus of this is management, but if you’re in a relationship, or friends with a knowledge worker, you’re generally just as likely to be impacted for all the same reasons as someone in the same IT establishment.

R-E-S-P-E-C-T

Those whom they do not believe are worthy of their respect might instead be treated to professional courtesy, a friendly demeanor or the acceptance of authority.

My closest friend once warned a newcomer, “if Matt was consistently nice to you, you’d be irrelevant.” This is not a concious thing, but it is very real. Arguments, seemingly unprofessional behaviour, witty banter, are all ways of negotiating decisions and divining truth and logic. No knowledge worker wastes energy or logic on people that aren’t worth the investment. We’ll take bullets for those we respect, but wouldn’t lift a finger for those we don’t. That respect is earned by a confluence of aptitude, attitude, and appreciation. I’ll fall over myself trying to help a bright person who bakes me cookies, or someone who admits they’ve got a problem and is willing to do what it takes to solve it. But bright and ungrateful, or troubled and unwilling to learn – forget it. I’ve got better things to do.

IT pros are [not] antisocial. On the whole, they have plenty to say.

The biggest misconception you can make is that a knowledge worker’s lack of engagement has anything to do with being anti-social. Sure, there are some socially awkward people in any group, in any industry, but they’re not the mass. Again, it has to do with the value of communication. Don’t expect a network architect to ramble on about dulcimer trees and virtually redundant paths to your HR director- They can assess, instantly, the subject-matter competency of their audience and will prefer to say nothing and hover by the punch bowl over engaging people who won’t understand anything they have to say. This works the other way as well. Knoweldge workers don’t care about accrual systems nor the intricacies of handling the corporate Christmas Fund.

Self-organization

IT pros always and without fail, quietly self-organize around those who make the work easier, while shunning those who make the work harder, independent of the organizational chart.

Look at your IT group:

  • Do they hang-out voluntarily outside of work? Knowledge workers don’t generally leisure with anyone they don’t respect.
  • Do they invite managers to do things? A manager who is being included by a team has their respect.
  • Do they invite managers outside of their chain-of-command to do things? You may have a synergy there you’re not realizing the potential of.
  • Who do they eat with? Meal time is leisure time. Geeks don’t eat with people that give them indigestion.

The more synergies you see within the above dynamics, the more likely you have a successful team on your hands. If you observe these groupings happening across group-lines (server guys and network guys… network guys and programmers… helpdesk and programmers, whatever), that’s a pretty good indication you’ve got good geeks, and they’re gelling well . If you notice team members heading for the door the moment work is “over”, never involved with other members, probably they’re being shunned and you’ve got some decisions to make.

Minesweeper Consultant, Solitaire Expert

Doctors are a close parallel. The stakes may be higher in medicine, but the work in both fields requires a technical expertise that can’t be faked and a proficiency that can only be measured by qualified peers.

I’ve never met a knowledge worker that cares about degrees or certifications, theirs or others. Your MIS or MBA earns you no credibility. Your A+ or MCSE won’t even get you a second glance. Experience, demonstrable skill, and respect are the only currencies that matter. The worst thing a manager, co-worker, or anyone else can do is try to fake it. Fraud is always a bad decision, generally generates more work, and smacks of irresponsibility- None of those things a knowledge worker will appreciate.

It’s Lupus

While everyone would like to work for a nice person who is always right, IT pros will prefer a jerk who is always right over a nice person who is always wrong. Wrong creates unnecessary work, impossible situations and major failures.

Would you prefer a nice doctor that cuts off your left leg to cure your broken right thumb? Or someone who barely acknowledges you exist, but seemingly effortlessly diagnoses your rare condition, and sets you on the correct treatment before walking out of the room, never to see you again? If I had a nickel for every time my mother said “no one likes a know-it-all”, I would have retired at 17. Your uber knowledge workers are always right. They are. They have to be. If they weren’t, you wouldn’t keep them around. Why would you? What’s the point in an asshole who’s wrong frequently/all the time? Showing that you have confidence in their decisions, and will back them up in between the time that they say “we should do X” and X is finally proven to be “right” means the world to them, and gains you major credibility. Throwing out illogical arguments, or pedantic edge-cases that are nothing more than theoretical CYA, loses you major credibility.

It’s not about being right for the sake of being right but being right for the sake of saving a lot of time, effort, money and credibility.

Separating the Wheat from the Chafe

If someone has to constantly be taught Computers 101 every time a new problem presents itself, he can’t contribute in the most fundamental way.

Team members that aren’t competent and are unwilling to learn need to be reassigned away from knowledge workers. They’re poisonous, and over time will drag down morale and cause retention problems.

Strong IT groups view correctness as a virtue, and certitude as a delivery method. Meek IT groups, beaten down by inconsistent policies and a lack of structural support, are simply ineffective at driving change and creating efficiencies

As a manager, if you have a group that actively works to maintain the status quo; not changing or evolving with the times; not improving or increasing services – then you have a problem. The problem could be you, or it could be the group, or it could be a bit of both.

Knowledge workers strive to create efficiency and to evolve their own role. Good knowledge workers will seek out new services to offer, or new ways to do old things. Great knowledge workers will continuously automate their “old job” such that every 6…8…15 months their current job looks nothing (relatively) like their old job. If they’re not doing this (re-read Self-organization, too), it’s time to shake things up.

If the problem is you, shore up your support for them. Make a real, sustained effort to get in their corner- to show them you have their back. It’s amazing how quickly a good group can turn around with a little consistent, positive focus. An obtuse group won’t respond at all to TLC.

Some people will still think I spelled chaff wrong.

You’re Paying Money For That?

IT pros are sensitive to logic — that’s what you pay them for. When things don’t add up, they are prone to express their opinions on the matter, and the level of response will be proportional to the absurdity of the event. The more things that occur that make no sense, the more cynical IT pros will become.

Nothing sets a knowledge worker off faster, more viscerally, or with less ability to restrain, than making an illogical or over-valued acquisition. If your team says “we can do that with open source software” or “we could do that in-house”, you need to listen. What they’re really saying is “only an idiot would go and spend money on shit… Money you’re not putting into our salaries”. Knowledge workers want to provide great services. They’re not going to suggest a product that stinks, or won’t meet the needs- it would reflect poorly on their decision-making skills. The more upset they are, the more you need to listen. That nice shiny new pair of shoes with all the bells and whistles will cost you dearly in morale. Dearly.

Pointless administravia like meaningless “evaluations”, paperwork, etc. are similarly likely to receive push-back.

What executives often fail to recognize is that every decision made that impacts IT is a technical decision. Not just some of the decisions, and not just the details of the decision, but every decision, bar none… It can cost an organization literally millions of dollars.

When HR goes out and buys a new HR system that’s “easy to use” and “won’t burden your IT people” and wizzbangwow without consulting IT on several levels, this is a bad thing. Of course, when you bring in your knowledge workers, and they say “this really shouldn’t be purchased”, you need to listen to them. Again, I’ve never met a knowledge worker that gives a care about their own workload: they care about making sound decisions, and helping the organization succeed. HR may be impressed with the marketing glitz surrounding the nice shiny new pair of shoes with all the bells and whistles, but that doesn’t mean the organization should acquire it.

Management Behaving Absurdly

Good IT pros are not anti-bureaucracy, as many observers think. They are anti-stupidity.

Sun-Tzu said (out of order) “Defeated warriors go to war first and then seek to win, while victorious warriors win first and then go to war”.

Mat-Thew said “A manager takes a collection of resources and accomplishes work, while a leader fosters accomplishment by inspiring their resources toward work”.

If you feel your knowledge workers have problems with authority or don’t respect you because you’re bureaucratic, you could be right. However, Occam’s Razor tells us that if you see this problem in more of your workers than you don’t, the problem isn’t them… It’s you.

Even marginal knowledge workers acknowledge the need for, and willing to participate in, logical bureaucracy. They want to get paid, so they account for their hours and report them. They want to have a record of work, so they will use ticketing systems. They want the organization to succeed, so they will provide truthful expert knowledge when called upon (a manager would call this a “meeting”), as long as there is logical purpose for the provision.

Arbitrary or micro-management, illogical decisions, inconsistent policies, the creation of unnecessary work.

If you have personnel who need micro-management, they’re not knowledge workers – but smart (ostensibly) people there to collect a paycheck. You’ll also notice through self-organization, that they’ll be on the “outside”. Micro-managing knowledge workers is detrimental, because it steps on self-organization, shows lack of understanding, and most damningly- it causes a knowledge worker to question your motives, and increases the likelihood you’re a credit whore.

Executives expect expert advice from the top IT person, but they have no way of knowing when they aren’t getting it. Therein lies the problem.

As a manager, nothing will create an image problem for you faster than if those over you don’t feel they can get the answers they need from you. If executives are talking directly to your subordinates, you’re getting side-doored, and are becoming irrelevant. Once this hits a critical mass, expect to be relocated or removed altogether. IT managers must have awareness of what’s going on now, what technologies are being used, and where your knowledge workers are leading into tomorrow. If you don’t know those things, or can’t communicate them up the ladder, you’re obsolete.

And make sure all your managers are practicing and learning. It is very easy to slip behind the curve in those positions, but just as with doctors, the only way to be relevant is to practice and maintain an expertise. In IT, six months to a year is all that stands between respect and irrelevance.

I can’t expand much on this. If you’re managing systems people, learn systems- constantly. Programmers? Network analysts? Doesn’t matter- you need to stay current with their world. You do not need their depth of knowledge, but they need you to understand that C is more than the third letter of the alphabet, that linked-lists should never be used outside of a computer science class, that flow control is good but flow constraint is bad, that X is something more than the 24th letter of the alphabet, and you damned-well better know the non-paged memory limit of a 32-bit system architecture.

Who?

IT pros would prefer to make a good decision than to get credit for it.

I was having coffee with a colleague earlier this week (discussing management pitfalls, ironically), when a woman I took classes with many years ago walked up and said “I didn’t know you worked here!” My colleague smiled at her and said “You really just made his day by saying that”. It’s true. Knowledge workers want to be involved in sound decisions, and furthering goals, mostly eschewing credit as long as internal respect orders are maintained. Knowledge workers will chew on their tongues until they bleed to avoid  saying “I told you so” (although it’s always on the tip of their tongue, and do come out, occasionally); they’ll credit “the team” when a spotlight is on them.

If you have people that say “I” instead of “we” frequently, there are two reasons. Either the person is there for just a paycheck, or there is concern that management will take credit for their work. Don’t do that. If you look at self-organization, you should be able to spot where they stand.

Making Management Matter

The primary task of any IT group is to teach people how to work. That’s may sound authoritarian, but it’s not.

Left to their own devices, non-knowledge workers (inside, and outside of IT) will maintain the status quo. They’ll do the minimum necessary to collect their paycheck, and never work any smarter. As a manager, it’s critical that you realize the value of your knowledge workers beyond their cubicle walls. Your top people can raise the bar across the organization, provide strategic visioning, and fundamentally improve (which means “change”, a scary thing) how the organization functions.

Take an interest. IT pros work their butts off for people they respect, so you need to give them every reason to afford you some.

As said previously, knowledge workers will take bullets for those they respect. Once you’ve earned it stay engaged, take non-managerial interest in what’s going on, keep them excited about working for you even when the work they’re doing isn’t.

Favor technical competence and leadership skills.

When you notice a knowledge worker take the reins in a crisis and pull the whole group – possible even other groups – through it, that should leave an impression with you. That worker should be your right hand. If they aren’t in your team, maybe you need to figure out how to change that. When you notice a knowledge worker disavow responsibility for a failure – personal or team – or come up with lists of excuses, that too should leave an impression with you. Irresponsibility is not a trait of a knowledge worker. Not ever.

If you need someone to keep track of where projects are, file paperwork, produce reports and do customer relations, hire some assistants for a lot less money.

Delegating chores and administravia to knowledge workers is a waste of money, brainpower, and above all: Respect. Insulate your brainshare from these items and let them provide you value.

When it comes to performance checks, yearly reviews are worthless without a 360-degree assessment.

Everyone hates performance checks, whether knowledge worker, clock-fodder, runt or luddite. The key item is that knowledge workers are all for assessments if they’re purposeful and logical. Having a manager rate a subordinate (generally higher than they deserve), write some courteous crap (because their MBA professor told them that positive rewards work better than punishments) and give them goals to work on (because there’s always room for improvement), is an illogical, purposeless system that doesn’t result in the subordinate understanding any more about themselves or their performance. It’s total crap. Total. Crap.

Multi-factor assessments (such as the 360, the author mentions) are actually logical, have defined purpose, and result in a fairly consistent understanding about how their performance is perceived. MFAs involve diagnostics by superiors, subordinates, peers, and others with sufficient contact and experience: Sometimes major clients, organization psychologists, whomever. This takes a lot of work, but I’ve seen it correct some fairly chronic problems in short-order. Seeing that everyone thinks you need to shower more, generally has some impact.

Periodically, bring a few key IT brains to the boardroom to observe the problems of the organization at large, even about things outside of the IT world, if only to make use of their exquisitely refined BS detectors. A good IT pro is trained in how to accomplish work; their skills are not necessarily limited to computing. In fact, the best business decision-makers I know are IT people who aren’t even managers.

I, like a lot of knowledge workers, wear two hats: We’re an employee somewhere, and we’re consultants. As employees, often, we’re undervalued, underpaid, and our opinions don’t matter outside of niche topics. As consultants, often, we’re prized and paraded, CEOs and CIOs hang on our recommendations, technical staff cower knowing our every gesture will generate work for them, and when the sizable invoice arrives the payment clerks make sure it is turned around immediately.

It’s a surprisingly sharp dichotomy to live with, but those of us that do this understand it: As employees, we make hundreds (thousands, sometimes) of decisions a day, fifty weeks a year, the vast majority of them silently making things better from the shadows. As such, the organization doesn’t really value our individual decisions. As consultants, we make a handful of decisions, all of them pedantically documented and justified, and our clients revel in the certainty that their consultant just made some amazing decisions, and will put together working groups to implement them post-haste.

I had a consulting gig a few years ago where I quickly identified a knowledge worker who was pretty much a subject matter expert in what I was there to do. I separated him from the greater group and we had a discussion over lunch, where he eventually confided his frustration that I was there at all. I told him, essentially, what was in the previous paragraph, and said he should start making all of his decisions out loud while I was there. It would be obnoxious, abrupt, and I would end up getting sick of it – but it served the intended purpose: He knew what he was talking about, and was making congruent decisions to my own. They didn’t need to pay me $5k/day to do what their in-house SME could do for far less. That was the last time they called me in for that topic, and last I checked, the SME was a division head.

As a manager, you need to pull your knowledge workers out of their zones. You need to get their opinions on greater organizational issues. Not only will they respect you for it, but the successes they breed will be attributed to your foresight in engaging them.

A Nice Little Bow

Taking an honest interest in helping your IT group help you is probably the smartest business move an organization can make.

I sound like a broken record, but it all really comes down to this: Engage, listen, learn, support, encourage, repeat.

Unlike in many industries, the fight in most IT groups is in how to get things done, not how to avoid work. IT pros will self-organize, disrupt and subvert in the name of accomplishing work.

If you have subordinates unwilling to do new things, learn new things, or avoid work- they’re not knowledge workers, they’re there to collect a paycheck. Where you should see heated battles is over the details: Which technologies should be used to solve a problem? Which systems should be involved? How will it be pathed? Where will it have dependencies? How will it survive global thermonuclear war? Can it be prototyped? Can it be virtualized? Should it be virtualized? Synchronous or asynchronous? Threaded or processed? Static or dynamic? Database or flatfile? Which database? I feel a lot better just writing out all those things. That’s what matters.

Our most dynamic group has taken a “build it, and they will come” mentality for numerous services that eventually became focal-points for the organization. They go out of their way to do “unofficial” development work, subversively at times, knowing that in the end they were making good decisions, and providing valuable solutions. That’s what a real knowledge worker does, and that’s really all they care about.

Sep 09 2009

RIP Nortel

[UPDATE: Avaya did indeed win the auction, for nearly twice their stalking-horse price. We'll see how that plays out.]

This Friday, Nortel’s enterprise products division will be auctioned. Avaya is the stalking-horse, with numerous other companies additionally submitting bids. If history holds, Nortel Enterprise will not be Avaya’s, but belong to someone else (despite all the buzz Avaya and its partners are making). Regardless, that someone will acquire the best enterprising data switching platforms made.

Hands-down, the ERS 5000 -series has continued to be the best integrated UTP switch, since coming onto the scene many years ago. Beefy backplane, generous uplinks, flexible stacking, decent price-point. I’ve baked switches from every vendor willing to send me gear, and no one can handle the battery like an ERS 5000.

The ERS 8600 chassis-based platforms also make a mockery of the competition. Robust, redundant backplanes; extremely flexible slotting options; ridiculous feature set; decent price-point. Better marketing could have really pumped this line, as a number of the slot options were best-of-breed. Again, I’ve baked a number of chassis, and the 8600-series delights on most all of the benchmarks that matter.

There are lots of other diamonds in the enterprise space. Quite a lot. I’m not a “loyal” customer of anyone: I believe in market Darwinism. Unfortunately in this case, the best products were latched onto a fiscally irresponsible company, led by brain-dead opportunists who from the outset have undermined the legacy that is … was … Nortel.

I look forward to Friday – to see who’s getting the chance to own the best-of-breed enterprise networking products, and hope when they sit down and take a good look, they realize what an amazing set of assets they have, and Do The Right Thing with them.

Regardless, Nortel is a less-than-skeletal image of its former self, and a venerable multi-century legacy is all but a footnote.

Sep 08 2009

A Rebuttal of a Rebuttal…

Somehow, while clicking away at the Internet, I managed to get to this blog post, which is a rebuttal of this article by Cory Doctrow about cloud computing. Please keep in mind I dislike most of the things that come off of Cory’s fingers intensely, and even elements of the aforementioned article. Where I blew an artery was at the rebutter saying:

Cloud computing fosters intense competition.

This is not just wrong (people are wrong all the time, and I don’t blog about it), it’s dangerously wrong. Nowhere in technology have we had less competition potential, and more monopoly potential than in the realm of cloud computing. Why? Cloud computing is all about size and marketing. I remember back in the day when I was happily using WebCrawler to scour the Intertubes for what I wanted to find, when some jerks from Stanford started some crappy search engine called Google. Pffft. “There’s tons of competition in the search engine market”, the analysts said. I was concerned then, I’m concerned now.

Any centralized Internet service is about size and marketing. Read up on my last post if you want to know how I feel about centralized services in general. Right now there are three major players in the “cloud computing” market. That will grow, as did search engines, to probably 20 major players and everyone will think I’m nuts, and then they will be slammed against the wall by One.

One that claims not to be evil.

One that believes (truly) they’re doing good.

One that rules them all.

This is not some psychic prediction, this is the recognition of a pattern that has played out too many times in the young life of the commercial Internet.

Sep 02 2009

Distributed By Design

The Internet, by taking its name literally, means “the network between networks”. 40 years ago today, the Men and Women that first started implementing the technologies that grew up to be the Internet didn’t want to create one network- they wanted to unify distributed networks redundantly. They wanted to do it this way so that a problem on one network, had literally no impact on other networks – and especially no impact on the Greater Network.

Yet humans are congregational, fashionable beasts. Humans, in general, want to be where there are other humans- even virtually. They want to be a “part of something”. They want to be on Facebook or MySpace. They want an iPhone. They want a Gmail account. They want to be LinkedIn. They love fads and the feeling of faux exclusivity that they bring, like Twitter. They fight against distribution.

The Fallout of Centralization

When Gmail is offline for a few hours, it makes international news. Millions of people across the globe are without e-mail! Crisis! Panic! When Facebook “gets a virus”, it’s international news! Millions of people are at risk! Crisis! Panic! When Twitter gets knocked offline, it’s international news! Millions of people can’t tell each other what they’re eating! Crisis! Panic!

Why won’t they get their e-mail from a more local provider? Why won’t they get other services from a more local provider? Why do people by Macs? Why do people like malls? They fight against distributed designs, and try to pull things together. “One-stop shopping” to a network neophyte is all giggles and rainbows- To a network architect it’s “bad design”.

Internet history is littered with fads and centralized solutions that inevitably fail to retain attention after the Next Big Thing arrives: But distributed solutions tend to last much much longer. Why?

The Argument for Centralization

There’s only one reason Google loses more money than my lunch cost every time someone watches a YouTube video: Control. Companies offer centralized services so they can control their users, their content, and their market. Facebook doesn’t care about user privacy or quality services, they care about making money (or, in their case, losing less money). They want you to be on their site as often and for as long as possible, and they’ve already sold your information in order to fund more gizmos and whirligigs (”apps”, they call them) to keep you coming back and sticking around. Control.

If Facebook was distributed – anyone could put up their own Facebook-compatible site, and allow other geographically-similar people to use it – how would they exercise control? How would they censor you, collect your personal data and sell it? How would they push you the latest version of your favorite whirligig? How would they profit off of you (or prevent someone else from profiting off of you)? Control.

While Gmail is a centralized interface to a distributed system (collectively known as “e-mail”), they’d just love it if everyone used Gmail and there was no distributed e-mail system. They mine your e-mail for keywords to target ads at you, making truckloads of money at the expense of your privacy- Ads that companies pay a premium for because Google can already “guarantee” you’re interested. Control.

The Argument for Distribution

Keeping components small, simple, and “close” is a doctrine of many disciplines, and is imperative for “critical infrastructure”. I have more than one client that intentionally buy equipment from multiple vendors, to reduce risk of single-vendor product failures. Several only update a fraction of systems and application software to new versions at a time, to reduce risk of debilitating bugs (one client, literally, has some 8 year-old operating systems running unpatched for this reason). More, and more institutions have multiple WAN (”Internet”) connections, from multiple providers, to reduce risk of single-provider failures.

Distributed infrastructure is more about “us” (the global network users)  than “you” (the individual). You don’t care whether Gmail (with a few million users) is down, or whether your ISP (with a few thousand users) is down: all you care about is that you can’t get to your e-mail. Because of this, it’s hard to convince “you” that distributed is better. You, the individual, don’t care about network survivability or how many other people are impacted- you only care about yourself, and your inability to access your mail.

Would you like to travel to your State capital to go to a centralized hospital? How about to the National Library to check out a book? When it comes to travel-related logistics, most people go “duh, local hospitals and libraries and schools and whatnot make tons of sense”: But once they go online – once that facade of geography is lifted – eYouSpaceFaceGoogleTwitterBookMailBay.com really is the best thing in the world, OMG?!!

HOWTO

So what can we do? What can you do? Appreciate and utilize local, distributed infrastructure. Don’t outsource/offshore your new whizbang application when your local hosting provider can handle it. Limit your dependence on centralized services, and find distributed substitutes where needed. Pretty much every infrastructure application has a distributed counterpart, many of them with much much less evil than their centralized cousins. I’m not saying “don’t go to website X” – I enjoy Wikipedia and CNN and all sorts of centralized information sources – but I don’t depend on them for communication, and I certainly know other places to go if they’re offline.

Distributed service architectures are about freedom and survivability. Centralized service architectures are about captivity and control. Happy Birthday, Internet. I look forward to another 40 years of architectural ingenuity.

WordPress Themes