Successfully Delivering Complex Systems

Image by dvidal.lorente via Flickr

Have you ever worked on a large transactional system? The complexity of a large transaction, such as the development of insurance quote or loan origination system where there are many integration points with third party data providers, often experience navigational challenges or become too complex with the need to reconcile data. They are often too difficult for users to work.

Typically, what happens in these systems is the development team is continually confronted by end users with demands for increased flexibility. Development continually attempts to add more flexibility into the system, but the code becomes too complicated. The development team and the users become more and more frustrated as efforts bring little in the way of results to the problem.

Large transactional systems that are designed poorly usually die of their own weight. As more and more complexity is handled, the code becomes considerably more brittle. Often good coding practices are bypassed, and you just end up with a bowl of spaghetti in your implementation.

So, why is that? Intuition would lead you to think that the more you attack a problem the better the end result will be. But, it never works out that way. The more you work to include new features, shortcuts, hyperlinks, etc., the more messy the system becomes.

The problem lies in the approach to handling complex transactions. Most teams do a good job of identifying the transaction that needs to be modeled. They immediately go to work to develop a solution that almost always ends up being some sort of logically organized wizard. The application may have the ability for navigation, say with a few quick navigational features (such as tabs). Even though there is an appearance of an orderly transaction, you quickly finds that there are plenty of dependencies between the tabs. As new requests come in from users to handle alternative workflows, you quickly discover that in-line validations often fall apart as data dependencies exist between various sections of the transaction. The system quickly exposes its inflexibility. One good sign you are getting into trouble is when small requests seem inordinately long to solve.

The way to handle complex transactional systems is to stop driving all solutions and alternatives to the transactional level and begin thinking about the interactivity required by the users when they collect the data. In the insurance industry, it is not uncommon to spend 15-30 minutes on a single transaction, such as a quote or application. Begin by acknowledging that there is absolutely no way that ANY transaction will ever go from top to bottom in an orderly fashion (ever!). The interactivity your user will face with his customer will require ultimate flexibility when assembling the transaction.

So, rather than dealing with how to navigate at a transactional level, start thinking about the tools that are necessary for the user to successfully navigate his interaction with his customer. Once those tools are in place, your user will be able to quickly navigate through his interaction, and be able to deep dive into the transaction when and where it is appropriate. By not confusing the interaction with the transaction, you will be able to deliver simple, tight, and distinct elements of your transaction and (ultimately) avoid creating an unworkable mess.

So, lets see this in practice by getting back to the insurance example. When studying the interaction between user and customer, there are many things required to complete the transaction that have nothing to do with the transaction at all! For example, the customer may want to run different scenarios, or the user may want to jot down a note or a telephone number if a call gets dropped, or the user may want to explore historical information during the transaction that has nothing to do with the user's new transaction (e.g. like how much he paid the last time he renewed his policy). The team should explore this interaction and be able to distinguish what a user desktop would look like in spite of a transaction. You may want to introduce concepts like a scenarios file with a scenario comparison tool. You may need to build a "post-it" feature where the user can jot down a quick note or that phone number if the call gets dropped. Maybe the users need a cut/paste feature to a small portlet window that can hold information you will need to reference throughout the call. Maybe they need an "undo" button, or multiple retractable points stored to being again in the interaction. These items are not transactional in nature, they are desktop enablers at best and will allow you to abstract the flexibility the users will need above the transactional requirements.

The next thing to do is segregate the components of a transaction from each other so they are simple and distinct, and can be "assembled" into the entire transaction. Avoid wizards (implied, or even explicit wizards) and sequence rules. Keep interactivity and the flexibility at the higher level, and the transaction element level self-contained.

The more complex the transaction is, the less likely you will be able to stick to a transactional workflow. Model the interaction, and the transactional elements will be able to be defined quickly, efficiently and not die of their own weight because you'll avoid confusing the transaction with the interaction.

How have you dealt with segregation of interaction and transaction in your complex systems? I'd like to know.

Large Project? Invest in a Continuous Integration Environment

Image via Wikipedia

I've always admired small, highly effective teams. The reality is that it is much simpler to be effective when dealing with small teams. The smaller the more effective. If we take this to the ultimate extreme, say a team of one person, there are no arguments, work is done and understood by everyone on the team.

In reality, most projects aren't done by just one person. Generally, communication and understanding becomes exponentially diminished with greater numbers of people on a project. Imagine running a project with 50 people. Regardless of all the design documentation, meetings and communication, there are bound to be misunderstandings and an assortment of problems.

That is why its important to constantly keep the code base in check. Sure, that's what QA is there for. However, what I've found is that with very few exceptions most QA teams have a heavy emphasis on functional qa testing, and often that is performed manually over and over again.

There is undisputed value in functional testing. But what is often overlooked, is the need to capture those functional tests and automate them. If you are on a large project, and haven't implemented test automation software, you are wasting money... lots of it.

However, that alone isn't enough. The old saying regarding quality testing is that "you have to build it into your product, not just test it at the end." And so, we now face the modern-age tester: the developer, the need for test driven development, and continuous integration.

TDD has developers create tests before code is ever written. Naturally, if you run the test before anything is written, it should fail. Once the unit of code is completed, the developer can run the same test and prove that his code works.

To accomplish this, Developers, especially those on large projects where code is flying around everywhere, need to develop xUnit test (e.g. jUnit for java, etc.) that prove to them, and others, that their code is working. Once the unit of code is developed and the junit test passes, they should migrate the code to another development environment: The Continuous Integration Environment (if you don't have one of these, you need to go procure one).

The continuous integration environment(s) are used to repeatedly run the entire suite of tests developed to prove the code works. This is critical in large projects, because undoubtedly at some point, someone will change existing code and will introduce regression errors somewhere else in the application. In fact, this is generally why teams begin to develop "snowflakes" (see my blog post on snowflakes) which is even worse.

Regression errors can be so reputation damaging to your team, that you ultimately would be wise to avoid regression altogether and invest in continuous integration. No one likes to hear from their users that something that was working fine no longer works.

Developers should immediately promote their code into a continuous integration environment after completing a unit of work and making sure that the x-unit tests pass. Continuous integration should run tests repeatedly many times per day. The person in the role of continuous integration tester, should promote the code to the next scheduled run, and include the tests the developer created to verify his code. This way, the developer can be assured that the continuous integration testing resource will know if other developers break his code.

If the test run passes, all is good and the developer moves on to the next task. If not, the continuous integration tester, demotes the code and sends the developer back to the drawing board.

Now, one common problem I've seen is when testers start changing tests to accomodate sub-optimal (i.e. crappy) code. HUGE MISTAKE... DON'T DO IT! This ultimately will ruin your project. It's absolutely critical keep the code pristine. Reject the bad code, and get that integration build right back to being pristine.... Ostracize that developer like a medieval noble tossing a scarlet fevered serf over the castle walls. Out! Now! Before he infects everyone.

Obviously, you're not going to fire the guy (at least not yet), but he needs to get the message that the code has to work without breaking the pristine code already in place.

Occasionally, you will come across changes that cause breaks (which is also normal, and is part of the cost of support automation and integrated testing in place). This will require the continuous integration tester to remove tests. However, don't ever do that carte blanche. Each test needs to be tagged with the person who created it, and serious discussions need to occur if the continuous integration suite tests are going to removed. Bring in the person whose test has been violated, and get their buy in, as well as any analyst that can validate the necessary change. Never just listen to the guy who broke the test... we know what he'll tell you!

What challenges have you experienced in your code when working large projects? I'd like to know.

Related articles by Zemanta

• How Facebook Automatically Tests Code: PHPUnit & Watir (allfacebook.com)

Agile Survivor: How to Avoid Getting Voted Off the Island!

One of the most interesting things about moving to Agile is the self-governing aspect of being on a scrum team. Agile provides tremendous transparency to its participants. Everyone is working together to get a goal accomplished, and they become accountable to each other each day in the daily stand-up meeting.

If you can't cut it, the team will eventually boot you from the island, ala Survivor. The only thing missing is the formal tribal council, the tiki flame lantern and the metal do-bob to extinguish your flame.

Though eliminating individuals really shouldnt be the goal of any organization, raising the bar and expecting great performance is.. and, if after numerous attempts at counseling, etc. the individual doesn't improve, this hard reality will ultimately pay off. Its actually kind of refreshing to management to witness "self-governing" teams take the action necessary to get results. So, from this perspective, Agile can be a great way to raise the bar.

However, Agile teams can get "clickish," and they can, over time, become stagnant in productivity. Why?

The Agile methodology tends to group individuals that will work together for many sprints, often for years. Teams get into a groove, and you may find yourself creating "high performance" as well as "regular" performance teams. But the stagnancy problem can occur at the peak of the maturity of an Agile team: when the team becomes comfortable with all its members. Typically, at this time, the team has come to love agile, each other, and they feel productivity is just fine.

But what if it isn't?

Image by Getty Images via @daylife

I recently attended an Agile conference in Lost Wages (aka Las Vegas), where a few senior types (old foagies like me) started asking about how to improve overall performance of Agile teams. There weren't too many good answers in the sessions. Responses seemed a bit defensive, and delivered with a "you don't get it" sort of attitude.

Well, I got news for some of those speakers. While the Agile experience is extremely rewarding to the team, project results are non-negotiable. What I mean is "Results are defined as Scope delivered within time and cost constraints." Businesses invest in IT projects because they want results, not just a great experience for the techies... and, don't get me wrong, when the experience is good and the employees love it, then generally, you'll have more dedication to the projects, and the team will get "more into" their work.... no argument there. I'm talking about overall results.

There is no argument that Agile is a game-changer. Users are brought in early, and your team is always working on high priority items with frequent deliverables. That's all goodness. But, as a manager, you need to be aware of various items:

1. Measure Personal Productivity- OMG! How can you single people out????? Tepper, you are a sacriligeous hypocrite! Stop insulting the Agile Gods! Look, as a manager, you need to make sure your team is still playing a little "Survivor".... Chances are there may be people getting carried by others, who may be creating negative inflow or regression errors. It's human nature to take care of our own, and Agile teams can become family. The reality is they have probably stopped optimizing their productivity. Measuring personal productivity and occasionally changing team members around will keep things crisp- but shake things up too often and you'll only introduce chaos.
2. Identify and Correct Negative Inflow and Regression Problems- What problems are being created and who is creating them? Ultimately, these items turn into additional work for the team. What is breaking in production? As the manager, it is your responsibility to deliver projects with high quality. Get your teams to find these and deal with the root cause.

Over the next few blogs we'll discuss techniques to measure and prevent many of the issues that slow down Agile teams.

What are your challenges in your Agile implementation? How are you optimizing performance? I'd like to know.

Related articles by Zemanta

• Jumpstart Your Understanding of Agile (eon.businesswire.com)

Contracts

Image via Wikipedia

Why is it that some deals go well and some don't? What is the secret to striking a great consulting deal and, if you're trying to sell one, what is the secret to closing a deal? What are the fundamental pieces to forming a deal and what can you do to avoid getting the short end of a deal?

Lets talk about deals, today. How to make sure contracts you enter into a good deal, and how to avoid bad deals. Lets cover:

1. Closing Deals
2. Why "Win-Win" deals are the only ones that work best
3. Terms and Conditions

Closing Deals

For a good number of years I worked as a leader in a couple of consulting practices. My responsibilities included selling and delivering deals. The one thing I learned early on when I was selling deals is that you need three things to get a deal closed:

1. A need
2. A budget
3. A sense of urgency

It's my experience that any deal that doesn't have all three elements on this list will not get signed. If you have two of the three, your deal won't close. I mention it only because it is very common to have the first two, but the signature authority will not sign without the urgency. Many of our sales people could talk to potential clients for hours on end about a deal... no trouble getting the meeting, even outlining a project plan and budget, but they couldn't close the deal. In the business climate, if there isn't a sense of urgency, your deal will be pushed to the back.

In other cases, there was a sense of urgency.. Lots of good discussion, but the sales people never checked to see if the people they were talking to actually had a budget. Lots of enthusiasm and need, but they were talking to the wrong people.

So, its important to realize you need all three.

Why "Win-Win" Deals are the Only Ones that Work Best.

It is all too common that the sharper negotiator gets the upper hand in a consulting deal, whether its the client or the contractor. However, when your procuring intellect, as you do in consulting, its important to recognize that you can damage yourself by breaking the other party. When you strike a "win-lose" the likelihood of any project success is severely diminished. These hardly ever work.

Lets say you're the consulting firm and end up on the LOSE side of a negotiation... The other people you will need to please on this contract are your partners/owners, and you will be forced to do everything in your power to make the deal work for them, and that includes subcontracting the work just so you can retain a profit. I find it amazing how often folks will revel in the fact that they just drove a supplier into the ground, only to be presented with the provider's "B" team. As a customer, your goal is to drive a great bargain, but to also recognize when things just look too good. When you push it that far, you're pretty much looking at a disaster in the making.

The opposite is true as well. If you are the consulting agency and you just got the sweet deal everyone has always dreamed about, be ready. It's only a matter of time before the client figures it out and starts working you out, quite possibly forever. Abusive deals may bring you great short term revenue, but can also backfire on you when someone finally figures out what is going on. Your reputation will be marred, and someone (you, or both you and the guy at the client site who signed the deal) will be shown the door. Greed may be good on Wall Street, but in the business of collaboration to successfully deliver a project, its not.

While its important to drive a good bargain, you will reach a point of diminishing returns. One way to avoid this is to figure out what is good for both sides. Contracts need to be established in a "win-win" style, especially where services are involved, or you will have a catastrophe looming in the wings. I have found in my role as a"client" who understands how consulting practices operate, that my most trusted partners will share their goals with me. Those include:

1. Profit they are trying to achieve
2. Contingency they are building into the project
3. Marketing / Reputation goals.

I, on the other hand, will share my goals with them as it relates to:

1. Assuring I get qualified and dedicated staff
2. Knowledge transition
3. Warranties

Time and Materials Considerations

I also like Time and Materials contracts with lots of signoffs. When I was in the consulting side, I would look for frequent signoffs from clients on deliverables throughout the entire project. I used to tell everyone in the practice "Your always good until the very last signature." What that meant was that if I had a client signature on a document that articulated the progress and the particular deliverable of the project we were in agreement at that time. I would make a point of saying to my clients that we would have many of these. I'd also pointed out that if they were unhappy with our services they always had the option to let us go, but I structured my frequent signoffs so we couldn't dispute any deliverable up to the point of the signoff.

I found that this no-nonsense style worked for both of us (clients and consultants) because it kept us focused on working on the project. In a fixed price contract, as a provider you are focused on meeting the absolute minimum required to maximize your profit. As a client you are focused on getting as much as humanly possible from your provider for the same amount of money.

Guess what. That doesn't work. That is founded on grandstanding and embedded beauracracy. There is no efficiency and streamling... no economy to the solution. A fixed price contract always has a winner and a loser, and often two losers. Do time and materials and you get quicker results, more elegant solutions and longer value from what will become a trusted relationship.

The Terms and Conditions

The terms and conditions is a critical piece of any contract. For anyone who's done contracts, you know that you will spend a good deal amount of time arguing IP. Keep in mind that the tools used to create your solution are rarely IP your provider can sell you. The techniques are also items consultants need to be able to redeploy. However, your specific algorithms and methods which your company owns are yours, and as a client you should protect your company from potential misuse with other clients of your consultants.

One thing I insist on doing in any terms and conditions negotiation is to make sure that whatever you discuss works both ways. Always ask yourself "Is this term fair to me and is it fair to you? What if we worded this exactly the opposite way.... does it still work?" I find that good terms and conditions are equilateral not unilateral in their structure. If you're not sure exactly whether the term you're negotiating is critical, ask for exactly the opposite, if the other side finds it untolerable, then there is an advantage that is probably inequitable. Avoid agreeing to those points.

Also, remember, that a skilled negotiator may have to give in on several points... Get a feel for what it is you are willing to trade for. Its not too uncommon to say "I'll give you this, if you give me that." In the end, the Terms and Conditions are just a structure for the delivery of the work. Often, the Ts and Cs get into a tug of war between each party's legal departments. So, it is important to be flexible, but both sides should strive to be fair.

UAT - The Hidden Time Bomb

Image by Getty Images via @daylife

So many software developers are familiar with the four phases of software development. No, I don't mean the SDLC that spells out Inception, Elaboration, Construction and Transition.... More like the SDLC that starts with the "Honeymoon" phase, the "disillusionment", the "trenches" and "catching up to expectations."

We've all seen it... Everyone is genuinely excited at the beginning of a project.. Enthusiasm is high as requirements are written, the users seem engaged... and even though the requirements are a bit difficult to read or maybe slightly unclear, there is a feeling from the users that the team gets it, so we move along. Somewhere at about the 50% mark it settles in. The users become a little more involved and the team realizes that the requirements didn't quite cover all of the features that are going to be needed. There is tremendous intensity in this phase (the trenches). Finally, you think you are there. You're coming to the end of your budget, the schedule dictates that you should be wrapping it up and do final user acceptance tests (UAT). The software will be deployed in about a month and you are ready to deploy.

So, here we are. UAT. All of a sudden people you've never met show up at your door and politely introduce themselves. You smile, turn to your team and ask... "who is that, again?" - They don't know either. The first few tests work just fine. The first day was a success, but there are a few more days to get through. On the second day, the users come up with a series of tests your team is unaware of and magically, like a surreal nightmare, your development train runs over that penny on the rail, and the cars begin to fall off the track, toppling all over the place. Your project is WAY off the mark.... It doesn't do this, and it doesn't do that....

You have lost control. That wonderful project of yours that you've reported as "Green" in your status reports for the past 6 months is now "RED" ... Blood red. You are caught off guard... you have lost your balance... you are in a free fall and you can't even see the floor. Sure, if we were in outer space there would be complete silence, but we're not... so management hears you screaming "Noooooooo!" as you fall, down... down... down... no end in sight... just a faint echo as they look into your development well and you fade from view. (dramatic, huh?).

Well, unfortunately, this is not all that uncommon. This happens a lot. UAT is the gotcha for the development team. There is nothing more in this world that raises the hair on my neck than a UAT surprise. As a project manager or development lead, UAT is the validation that you got it right, and for some reason, everyone gets so preoccupied with the development of the project that they miss the most important part of any software development... the acceptance of the code.

I can tell you, I have seen the best projects fail in UAT. Perfectly delivered, meticulous projects, that when reviewed were absolutely delivered to the contract. The development team did it, but the client wouldn't accept it. The worst case that I can remember ultimately took a small consulting practice out of business as UAT went on for months beyond the fixed price contract. The practice prevailed in arbitration, but the costs were overwhelming and the office was shut down. Perfect software development proven in an arbitration resulted in a failed consulting practice. Unfair. Oh, and in addition, the client who decided to go to arbitration lost a lot of money as well chasing the consulting company.

So, how can one avoid these kind of problems? Even if you deliver your project perfectly, as my friends in Dallas did. The first thing you need to come to terms with is that your contract should go well beyond the development of the project. Its easy to say, "we did everything right," they just don't want it, or its not my problem they don't get it. I guarantee that at a minimum you will pay, your reputation will suffer, your business will suffer.

The UAT phase is the most critical phase of your project. Does that sound ridiculous? Think about it. If you can't control the UAT, you are entering the "financial loss" phase of the project. And, let me be clear, when I say control, I don't mean "win the test of wills." I mean understand the amount of time required, the potential to make critical fixes, and assure the software will be used and accepted by going through ALL of the pertinent test cases required to accept it.

The biggest flaw I see in software development is the poor preparation teams make towards UAT. The best way to avoid UAT time bombs is to get your users to pony up and agree to all of the test cases they will run. Immediately challenge them when they identify them. I've listed questions below that you need to find answers to, and formally agree to, immediately after requirements and no later than the beginning of development (months ahead of UAT). Additionally, you need to revisit this list multiple times throughout development to make sure there will be no surprises in UAT. If you find that the answers are changing or growing, you need to plan and budget accordingly:

1. Who will be performing the UAT?
2. Do those identified represent 100% of the users?
3. What specific cases will you run? (get a list and full understanding of conditions that will be checked)
4. Are we certain that those cases will cover 100% of the acceptance ?
5. Which exception cases do we need to run in UAT? (outside of normal flow)
6. Are there any more?
7. When I document these, will you be able to provide an authoritive signature that states that successful completion of these tests constitutes acceptance?
8. If not, then lets keep working this document until you can.
9. Who will provide the final signature on acceptance?

The development of a well written and understood UAT plan is validation for your requirements. If the UAT goes beyond your requirements documents, your team missed the objective, and the software will not be accepted.

Furthermore, assign a lead who is skilled in change management to be the responsible person on the dev team for a successful UAT. This person should be meeting regularly with the client's organization to show them work in progress (what the system looks like and how it will be used) months in advance of the software's completion. They should understand concerns and help promote the use of the software. The feedback they receive should be shared with the development team and taken very seriously. When replacing an existing system, they should also take the time to understand the older system so that important features the users want/need aren't overlooked.

What UAT surprises have you experienced in your career? How could those have been mitigated? I'd like to know.

SOA - Maximizing your investment

Image via Wikipedia

The move to Service Oriented Architecture (SOA) is more than 10 years old now. The technology makes sense. It is rooted in the plug-and-play sort of mind set that is essential to businesses that want options with their platform solutions. No longer is integration direct. The integration strategy of SOA is based on web-services calls between systems. The promise of SOA is that with integration at the services layer, you can replace systems without having to stress out over where in the code all the integration points might be. Additonally, with a clear integration layer, organizations avoid the pitfalls of direct database access across systems. Services are distinct, clear and easily found and maintained.

All good. By all accounts everyone pretty much has arrived. Note: If your company isn't there yet, "plan to party like it's 1999," because you pretty much aren't living in this millenium.

There are a few things that are often overlooked in SOA implementations. I'll list them in summary here, and we can delve into them below:

1. Take the opportunity to architect in a "Vendor Broker" solution for third party providers.
2. Make sure accountability for SOA maintenance is well defined.
3. Avoid synchronous traps.

Here we go!

Vendor Broker Component

So, what is this Vendor Broker Component? Very simple. Its a tailorable component that governs how much business you give to third party providers. This is nothing more than a traffic cop directing traffic that you can instruct. A vendor broker is a key component to keeping your business operational costs down.

Why? Well, if you want to keep your costs down, and continually drive them down, it's a good idea to treat your suppliers as comodities providers. Lets say you get data from "Data Provider A." You have a great relationship with them and have done business with them for years. But now they inform you that they are going to raise their rates. You're stuck. It might take a few months to integrate to another provider, assuming you can get through legal in a short period of time. You don't have the leverage you need to keep them constrained.

A services Broker component puts the leverage squarely in your hands. The broker directs requests to different providers. You configure it so that you can route business to the provider you want to get the majority of the business, and if you change your mind, you can change it immediately.

Now, lets replay that meeting with Data Provider A. He wants to increase rates. You respond, with, "Wow.. I don't know, I was hoping you were coming in to lower rates." (I love it when this happens!)

They hold their ground. They know it will take you months to locate and implement a replacement provider. They're banking on the fact that this exercise would not be strategic for you. You probably don't have the resources available at this time to work on the new migration. They're banking on the fact that you have more important things to do than migrate away from them. They believe you are joined at the hip with them and they can get away with the rate increase. It's a small increase, anyway...you love their service, and you'll soon forget.

After the meeting, you immediately call Scotty in the engine room and tell him "Set the broker to only deliver 20% of the business to Data Provider A and give the rest to Data provider B." Minutes later, you're on the path to showing Data Provider A that raising your rates could really hurt their business.

Next Meeting:

Data Provider A: "Hey, we've been noticing your orders are down considerably this month."

You: "You guys are a bit pricey. We want to do business with you, and we'll continue to give you business, but you're going to have to do something about those rates."

They understand what's happening.

Tada! You get lower rates. You tell Scotty to turn up the thrusters for Data Provider A. Give it a few months, and the guy from Data Provider B will call next. Eventually, you get to replay the whole meeting with a new set of players (Data provider B).

Guess what, you're being fiscally responsibile to YOUR company by driving costs down. By the way, as a business person, its critical to not blow away any given provider because, if you do this right, you can leverage multiple providers over time and get unbelievably cheap rates. But to do that, there has to be competition for them. Take the time to do this right.

Accountability for SOA

If you are in a highly dynamic shop, where your employees are constantly changing maintaining and enhancing their platforms, its imperative that services ownership lives with the platform that built them. I know that sounds just too logical. However, I've seen (all too often) that the services build out is a "once and done" sort of event, and platform integration deteriorates as systems evolve. The key to SOA accountability is:

1. Assign an owner for the services in the platform.
2. Have them broadcast changes in the platform to all services consumers (other systems owners that use that platform's services) and
3. Make sure services testing is in place each time a release goes out.

Seems simple enough, but often overlooked.

A Final Word: Avoid Synchronous Traps.

There are two ways to communicate with services: Synchronously and Asynchronously. While SOA is a great way to architect solutions, be careful to not become overly dependent on synchronous communication. For those who may not recall what these two terms mean, synchronous requires a response before moving forward... Kind of like your 3 year old asking you a question. They follow you around and don't let up until you give them what they want.

Asynchronous communications allow you to submit a request, and move on... The response will eventually come. Its critical in SOA to try to push asynchronous interfaces, or errors will start piling up when your integrated partners experience their troubles.

To do this effectively, build a timing feature into your Broker component, and tie that to a service level contract with your providers.

That way, when you need something that feels "synchronous" (i.e real time quick response) you don't tie up resources if a provider is "down" and can reroute an order to a provider that is "up." A common problem is that IT people tend to think of asynchronous communication for lengthy transactions. That's a common misconception: Aynchronous doesn't mean that you should have to wait a long time before a response. You can build asynchronous integration that "feels" synchronous (i.e. expecting a quick response), but you should design your system to effectively handle outtages that aren't yours, and allow for a rewarding application user experience. If you build in too many synchronous integration points, your users will feel pain when any of those have slow performance or are down and they get stuck in your app.

What are some of the strategies you've implemented and challenges with SOA in your organization? I'd like to know.

Snowflakes

Image via Wikipedia

Have you ever been on a walk in a ski resort town right smack in the middle of winter. I used to go to Mammoth Mountain, California quite a bit to ski when I was younger. I used to love taking a shuttle into the heart of the town and going for a brisk walk at night. Everything was covered in snow. I can still hear the sound of the snow "crunching" as I walked across it on my way to a local restaurant or in and out of shops. The best part was when it would just start to snow and you'd see the snowflakes fall. I'd reach out and catch one and look at it. A smile would come across my face as I made out all the little branches of it. I'd think, "Wow! they say there are no two that are alike." So nice.

Reality check...(insert sound of scratchy record).... Back to work.

Well, snowflakes may be beautiful in Mammoth, but when you find one in your code, its a friggin' nightmare.

So, what is a snowflake? Well, once again I have to give credit to my good friend Igor who came up with this term. (the guy really is a genius).

A snowflake happens when people who either don't understand or care about the underlying software architecture begin taking short cuts they shouldn't be taking. They compromise the code-base by creating little branches of code that hang off the core to do a specific thing, a kind of

"one-off sort-of thing that's like that other thing in there, but I don't want to figure that out, so I'll just put this little thing in here so I can be done with this and move on to the next thing."

Rather than take the time to engineer the solution in the correct place, they take the path of least resistance and decide to invent a solution with disregard for enhancing or fixing the code where it belongs. Through ignorance or negligence, they have little regard for the impact of their actions.

Now, doing that once doesn't really constitute a snowflake. ... even doing it many times doesn't... That just makes the code sloppy. The snowflake occurs when you start compounding the branch, by adding more and more dependent code branches that hang off the branch, and the underlying circumstances or assumptions within the code are just plain wrong.

If an organization allows this to happen for too long, the problem becomes insurmountable. The bad assumptions are compounded over and over again and your team will become paralyzed. The snowflake becomes so large, that when you ask your staff to fix it, they'll tell you it can't be done or they don't know how. Here's another hint, when you ask for something you think is a reasonable request, they tell you it will take millions of dollars. It goes something like this:

You: "Hey, how long do you think it would take us to change this address validation?"

Developer: "Here, or everywhere in the code?"

(Not a good sign)

Developer: Probably 2 months and $100k.

You: "Uhuh. $#^&! We have a friggin' snowflake on our hands."

That's why it's a good practice to refactor. Let me be clear about this... Don't refactor every once in awhile.... (make a big smile) ... Like having a "special refactoring sprint." (hold that smile)

(stop smiling... look very serious)... Nope!....(still serious).

ALWAYS refactor.

Refactor like you are a Banzai tree grower shaping a tree or pulling the weeds out of your rice paddy. Keep the code pristine. There is NO room for snowflakes in software.

ALWAYS refactor.

When you get to the point that your software has become a snowflake, refactoring becomes impossible. It's too complicated and costly. No one will understand what to change or how to do it, and they become fearful they will break the code. You might as well start thinking about sunsetting this project immediately and begin writing the "Son of (insert your current system name here)" proposal.

Refactoring is a difficult concept to sell to the business. It's a lot like replacing the roof on your house. It doesn't really add any value to the house, but it keeps the water out. So, don't try to sell it to the business, just do it. Build it into your estimation model for any additional work you are about to undertake. You should be able to continuously refactor for about 10% of your project cost. If you stay on top of your code by building it into your culture, you'll be able to do it for less.

How have you dealt with fragile code in your company? Do you talk about refactoring? What are your thoughts? I'd like to hear them.

Gus

Raising the Bar - Measuring the Right Thing

Image via Wikipedia

Are you a gamer? You know, the guy who secretly sneaks away after dinner and turns on your X-Box or PS3. The alter ego sets in and off you go to kill Nazis, Zombies or turn a lap faster than Mario Andretti (I imagine many young techies have no idea who Mario Andretti is). Maybe you are the greatest mock-rock-star since Slash and can wail out solos on your little plastic guitar in Guitar Hero or Rock Band. Or maybe your one of those Facebook virtual farmers growing virtual crops.

I used to be a gamer, but to be perfectly honest, I don't think I've been excited about gaming for about 20 years now. My son loves to game, and he tries to get me to do it. I visited him last year up at Berkeley, and he got me to play Guitar Hero. He was amazing... I sucked. At one point, I turned to him and said "Look, you're so good at this you ought to learn how to play the guitar for real!" He wasn't interested in guitar playing, he just wanted to win the game.

The interesting thing about people is regardless of how good or bad they are at gaming, once you understand the objective of a game and how it is played, you can always get a little better. It doesn't matter what the game is. As long as it provides a score, you can improve it with a little effort. And certainly, such is the case with IT workers. They are master gamers!

We recently switched one of our projects to the Agile methodology. Agile has its roots in Lean and Xtreme programming principles. Its a great methodology for getting people to creatively collaborate and get work done more effectively. I decided to take our organization into Agile for two purposes:

1. Better Quality
2. Greater Throughput

I'll talk more about Agile in future blogs, but for now, lets just accept it as a really cool methodology that will delight programmers once they get the hang of it and get comfortable with the exposure it brings.

One of the fundamental practices in Agile is to estimate the effort of the tasks the team is about to undertake in a group setting. Mike Cohn, has a great youtube video that will teach you how to estimate by playing "Planning Poker." The purpose of planning poker is to estimate tasks as a relative measure to other tasks. The theory is that its pretty darn close to impossible to guess the absolute measure of something, but it is far easier to estimate things when using relative measures.

For example, ask your team "How tall is the Empire State Building?" The answers will be all over the place, from a few hundred feet to maybe even a mile high. Now, show your team a picture of the New York Skyline featuring the Empire state Building, and ask them "How many of those buildings would you have to stack up to be as tall as the Empire State building?" You will get very accurate and consistant answers.

That's a fundamental concept of Agile. People understand relative difficulty of tasks with greater accuracy than the specific effort required to solve any one particular task. In an Agile environment, you don't really care too much at first how accurate the estimation is... you let that develop. You measure that with each sprint (a time-boxed length of time your team will work). The team will complete as many tasks possible in your backlog of items. And, once the sprint is complete, you measure their velocity (throughput).

So, there we were at the team's first Agile Planning Poker event (okay, it was a meeting). Everyone was given a deck of cards. Each deck contained the following cards: 1, 2, 3, 5, 8, 13, 20, 40, and a 100. Items from a prioritized backlog were brought up one at a time. The team discussed each item for just a few minutes. Then everyone was asked to go to their deck and pull out a card that represented the difficulty of the task. Note, you have to pick a card or stay completely out of the voting (people who had no idea how to solve a particular problem were encouraged not to vote). No numbers can be selected that are in-between card values.. like 7's. Its either a 5 or an 8! Pick either the 5 or the 8. Remember, this is all relative anyway.

10 cards were pulled. I saw a 5, a bunch of 8's and two 13's. In Agile, you ask the outliers why they voted their number. Their input is considered, then you revote. Eventually, the team came to agreement on the score of each task. It was recorded and the next item was scored.

By the end of the meeting the team completed scoring the relative difficulties of each of the items. We knew the priority, and our team was ready to start their first sprint. The team was asked to select as many items they believed they could deliver in a 4 week timeframe. The goal was to have working, deployable code by the end of the 4 weeks. The team picked up 40 points.

Well, we discovered that in our first sprint (the 4 week period), the team completed 45! What an accomplishment! Everyone loved the process, they felt energized about how fresh it was and became very eager to take on the next sprint. Another planning meeting and the team picked 45 points. Guess what? They delivered an amazing 59 points that month! This Agile methodology is AMAZING ! ! !

As you can imagine, I was absolutely thrilled with the results. In fact, one team reached a velocity of 99 points one month. They were so proud. So was I.... right up until we learned about the regression problems we were introducing in the code.

Even though the teams were super-enthusaistic about the new way we were doing things, they got a little sloppy in their ability to catch regression problems. Lets just say that on that part of the game, well, they kind of bit the dust.

We were measuring points for each sprint... an indication of busy-ness and effort. However, we weren't measuring the additional problems we were introducing, because, quite frankly, no one knew we were introducing problems at first.

Igor Mandrosov, my good Russian friend of 15 years now, who also happens to be our Agile coach, said to me "Your problem isn't velocity, your problem is Negative Inflow."

"What's that?" I said.

"When you run high velocity your regression error rate goes through the roof." He actually told me the ratio of velocity to negative inflow points... not a good ratio, but we are improving.

"Okay, lets start measuring it and share it with the developers. Things will get better." I said.

Gamers!

What are some of the ways that you address quality issues in your company?

Gus

P.S. Negative Inflow is a term Igor Mandrosov coined. To date, I haven't seen any tools, outside of the ones we've developed under Igor's direction, in the Agile marketplace that addresses regression errors. As far as I'm concerned, Igor is advancing Agile, and he should trademark "Negative In-Flow"

Cultural Diversity and Communication

Image via Wikipedia

I just finished reading "Outliers: The Story of Success" by Malcolm Gladwell. It is an outstanding book, and I highly recommend it to anyone who is interested in success and how to achieve it. While the book focuses primarily on populations of individuals, demographics and the dedication of successful individuals, there is one chapter in particular that focuses on communication, and how the lack of it can lead to failure.

Chapter seven: The Ethnic Theory of Plane Crashes.

The chapter centers around a few airline crashes and the communications between pilots, their first officers and air traffic controllers. In particular, it focuses on a Korean Air Lines crash (KAL 801) and the Colombian airliner Avianca 052. Both were absolutely devestating crashes. Both were caused primarily due to poor communications in the cockpit.

The NTSA investigated both (though one was abroad), and in both cases came to several conclusions.

1. No fewer than 7 mistakes were made before each of the crashes. No single mistake would have been fatal, but the the compounding of them (i.e. one mistake was the basis for decisions that led to other mistakes, etc), eventually led to the passenger's and crew's demise.
2. Cultural communication styles based on cultural norms, as it relates to position and respect for authority, led to "Mitigated Speech" between those in the know and those who were not aware of the criticality of the situation until each of the crashes occurred.

Lesson #1: Get the Facts Before Making Conclusions

I began to think about how that relates to IT projects. How many times do we think we know what is going on, but actually have bad information. I came out of a meeting a week ago where there was considerable concern about one of my project's adoption rates (in this case, an internet application). We have been running stagnant for about 4 weeks. Adoption rates are flat. Of particular concern was a large client of ours, and the fact that they did not seem to be using the platform hardly at all.

Many in the meeting jumped to conclusions about the quality of the code, the stability of the platform, features that were not yet released that were holding a considerably sized client from moving forward etc. I mentioned that while we had problems, things seemed stable for sometime now, performance was good. We certainly have our share of defects, but all seemed relatively stable, but even with that, I still had my concerns and doubts as to why this client had not jumped on board.

The project leader and I huddled after the meeting and we began playing through the numerous scenarios as to why this large client's adoption rate was so low. We determined we needed to speak to them, and come to agreement on a prioritized set of features that would allow them to go for a full push.

When we called the client, we were relieved to find out that the reason the client hadn't adopted the product was because they were busy with an internal rollout of another initiative and made the decision to postpone their launch until this Friday.. (i.e. tomorrow). They would ALL be on tomorrow, all 600 of them.

Now, while that was somewhat refreshing to hear, I look back at the meeting where we were guessing (without accurate knowledge) and compounded a scenario based on a building set of assumptions that could have led to a "fatal" situation (i.e. I get fired, my project leader gets fired).

Lesson #2: Understand Cultural Diversity and how it affects communication

The second finding of the NTSA in Outliers, was that Mitigated Speech was a significant factor that led to the crashes. I contend that Mitigated Speech is a HUGE problem in IT shops, (though, not as fatal as in airliners). The author points to a hierarchy of mitigated speech. In summary it goes like this:

1. Command
2. Obligation
3. Suggestion
4. Query
5. Preference
6. Hint

In each of the airline crashes, the pilots, copilots and the tower communicated in some form of mitigated speech that did not escalate into a situation that was salvageable. Lets see how these forms work:

1. Command - Turn 30 degrees to the right
2. Obligation - I think we should be turning right about now
3. Suggestion - Lets go around the weather
4. Query - Which direction would you like to deviate?
5. Preference - I think it would be wise to turn
6. Hint - That weather up ahead looks mean!

The conclusion for the airlines is "there is no room for anything in the cockpit but the command form in a known emergency situation." However, due to cultural norms and customs, the first officers were talking to their bosses, and downplayed the delivery of the information... usually to the Hint level in most cases!

We did an informal study of our IT department last year and discovered that there were no fewer than 22 distinct and different native languages spoken by our employees. That's a lot of cultural diversity. But, in today's IT world, that's not that all uncommon. Each of these speakers come from different backgrounds and cultures. Communication styles vary tremendously.

In Outliers, the author speaks of a "Power Distance Index" and the attitude towards hierarchy. In particular, it deals with how much a culture respects and and values authority. In low PDI countries (like Austria, Netherlands) authorititative figures downplay their position by renouncing formal symbols and formalities. In high PDI countries (e.g. Belgium, France), not so. In the case of the Avianca Airliner that crashed near JFK in New York, the Colombian first officer did not challenge the authority of the tower, and ultimately followed a command that he should have contested. The conclusion of the reviewers was "there must be significantly reduced PDI in the cockpit."

In IT, we also experience PDI. I can walk into a project meeting with 30 people, and you can see the pockets of people who do not speak until they are asked to, and those who are chomping at the bit to challenge, etc. The important thing for managers is to recognize that there is PDI, and you may be getting mitigated speech. The simple query "do you think it would be best to back up the database before we proceed?" may be a subtle warning that you are seconds away from making the mistake of your life.

In summary, take the time to learn about the culture of your employees. Teach them what you have learned about mitigated speech, solicit clear and direct communication, and begin to recognize what they are really telling you. The IT mistakes we make may not be fatal, but they are at least costly in most cases, so it is in your best interest to make sure you are getting the straight scoop!

What are the communication challenges in your organization?

Implementing a Matrix Org

Image by ThomasThomas via Flickr

So, you've just gotten word that your company will be implementing a matrix organization in IT. What is that? Why now? What does it mean to me? A matrix organization can be a game changer for your company. Throughput can easily triple for IT. It can feel a lot like the difference between driving in 1st and 4th gear in your 911. This article addresses the background behind the move to matrix organizations, their origins, and some of the pitfalls to watch out for so that your implementation can bring the benefits you are seeking to gain.

The origins of the matrix organization started with consulting companies dozens of years ago. They are structured so that individuals belong to a professional discipline (e.g. development, architecture, analysis, quality assurance, etc), and are assigned to one or more clients for different projects. The matrix is defined by two axis: Technical Discipline and Project(s). In consulting practices, consultants are assigned by a resource manager over the technical discipline to an account manager on a project. For the purposes of this discussion, we'll call the professional discipline the "horizontal" and the projects the "vertical" axis.

Corporations have traditionally supported their business function's IT needs with silo'd support departments. There are benefits: Business domain knowledge of the IT staff tends to be high. But, from a technical practitioner perspective, the IT staff tends to be less specialized as they try to support ALL of the IT needs of the business function. It's not unlikely to hear employees in silo'd organizations to say things like "we had our big development push last year... this year we're in maintenance mode."

Corporations take on many key initiatives just to remain competitive today. Business is fierce, and one would be hard pressed to find a business that is betting on just one initiative to succeed. The costs of supporting a traditional departmentally silo'd model is overwhelming if you want to stay in business. The corporate demand to initiate and complete IT projects is many times what it was say 20 years ago.

The need for highly efficient expertise in horizontal disciplines (IT specialization) has made the traditional consultant matrix organizational structure attractive to corporations. No longer is it common to be the IT guy that just works for marketing, or sales, or distribution. Departmental IT silos are minimally staffed, if at all, and usually just have one or two maintenance personnel or "super-users" assigned for key systems. Very capable, intelligent specialists are often assigned to more than one "key" initiative in an organization, because of the need for their specific skill level.

The matrix organization has arrived in corporations, and while this can be very exciting for type "A" consultants, it has the potential to make a good number of your IT employees uneasy. The transition introduces stress and uncertainty. The sense of security your employee had (by being and belonging to the same team year after year) is eliminated. They are now evaluated on their personal merits as a technical practitioner. They need to stay skilled, motivated, and efficient because assignments will frequently change as initiatives complete and the landscape will repeatedly be renewed. This is the organizational paradigm shift (to borrow an old term), and to get the results you desire you would be wise to be careful in your implementation so that it will succeed.

To successfully implement a matrix organization in a traditional corporate structure you will need to put many things in place. Among them are:

1. Mentorship. The employee must feel he has someone he can talk to and that's not just the resource manager. The mentorship model requires senior people who can professionally advise the employee on growth areas. The mentor is someone who poses no threats or demands, someone who can listen to the individual's concerns and champion changes to the management of the company. Having a good mentorship model is a great way to improve your organization into reaching the levels of a learning organization. Mentors should meet with mentees at least once or twice per month.

2. Professional development. Implementation of a matrix organization speaks volumes to your employees about your beliefs in optimzation and high efficiency. Let them believe you by investing in them. Time should be set aside each year for employees to improve their skills, whether that's letting them attend conferences, seminars, or training classes and achieving certification levels. Other areas for professional development may include budget for books, or creating a library available for research.

3. Build a sense of community. Meet regularly within each horizontal discipline. That translates into scheduling time away from initiatives for a few hours each month to meet and discuss experiences on their initiatives, approaches to solving problems, and developing best practices for your organization. Enable your employees by creating collaborative work areas, wikis (knowledge bases), and provide communication methods, such as internal on-line chat, so that they can communicate more effectively within the discipline. Your employees will feel lost and overworked without a sense of community. In a matrix organization, employees have to recognize that there is something in it for their own personal growth. There is no better feeling than that of collegiality. Contributions will begin to flow in. Recognize contributions that can be leveraged to other team members and initiatives and celebrate the achievements of individuals to the professional discipline.

What are some of your thoughts ,challenges and experiences in implementing a matrix organization in your company?

IT Solutions Think-Tank Introduction

Image by Emilio Segre Visual Archives via Flickr

Welcome to the IT Solutions Think Tank. The purpose of this blog is to collaborate on solutions to various problems, as it relates to Information Technology and IT Organizations. I will post topic areas, and we can all provide commentary in on how to best address the issue. If you have some areas that you would like addressed, place a response to this post, and I will open up blog entries that we can work.

So, you may be asking, who is this blogger? I'm a 30 year veteran of IT, practically all of it in applications development. I am currently a vice president of enterprise applications for a large insurance company, and have worked in both industry and consulting at different times in my career. From practice leading to successive industry management positions. From waterfall, to iterative, RUP, Agile and all the hybrids (water-scrum-fall???). From onshore to offshore and near-shore, from centralized to distributed, small scale to huge, I've pretty much played an integral part in all of it at sometime in my career.

Most importantly though is the feedback and collaboration we will develop on this blog. This is a place where IT professionals can come to discuss topics, opinions and approaches. The IT Solutions Think Tank hopefully will be a rewarding sounding board for most of you and a place to visit to gain additional insights into a complicated, yet highly rewarding field.

Best wishes,
Gus Tepper

P.S. If you would like to email me directly, send your thoughts to gustepper@cox.net.