Successfully Delivering Complex Systems

Image by dvidal.lorente via Flickr

Have you ever worked on a large transactional system? The complexity of a large transaction, such as the development of insurance quote or loan origination system where there are many integration points with third party data providers, often experience navigational challenges or become too complex with the need to reconcile data. They are often too difficult for users to work.

Typically, what happens in these systems is the development team is continually confronted by end users with demands for increased flexibility. Development continually attempts to add more flexibility into the system, but the code becomes too complicated. The development team and the users become more and more frustrated as efforts bring little in the way of results to the problem.

Large transactional systems that are designed poorly usually die of their own weight. As more and more complexity is handled, the code becomes considerably more brittle. Often good coding practices are bypassed, and you just end up with a bowl of spaghetti in your implementation.

So, why is that? Intuition would lead you to think that the more you attack a problem the better the end result will be. But, it never works out that way. The more you work to include new features, shortcuts, hyperlinks, etc., the more messy the system becomes.

The problem lies in the approach to handling complex transactions. Most teams do a good job of identifying the transaction that needs to be modeled. They immediately go to work to develop a solution that almost always ends up being some sort of logically organized wizard. The application may have the ability for navigation, say with a few quick navigational features (such as tabs). Even though there is an appearance of an orderly transaction, you quickly finds that there are plenty of dependencies between the tabs. As new requests come in from users to handle alternative workflows, you quickly discover that in-line validations often fall apart as data dependencies exist between various sections of the transaction. The system quickly exposes its inflexibility. One good sign you are getting into trouble is when small requests seem inordinately long to solve.

The way to handle complex transactional systems is to stop driving all solutions and alternatives to the transactional level and begin thinking about the interactivity required by the users when they collect the data. In the insurance industry, it is not uncommon to spend 15-30 minutes on a single transaction, such as a quote or application. Begin by acknowledging that there is absolutely no way that ANY transaction will ever go from top to bottom in an orderly fashion (ever!). The interactivity your user will face with his customer will require ultimate flexibility when assembling the transaction.

So, rather than dealing with how to navigate at a transactional level, start thinking about the tools that are necessary for the user to successfully navigate his interaction with his customer. Once those tools are in place, your user will be able to quickly navigate through his interaction, and be able to deep dive into the transaction when and where it is appropriate. By not confusing the interaction with the transaction, you will be able to deliver simple, tight, and distinct elements of your transaction and (ultimately) avoid creating an unworkable mess.

So, lets see this in practice by getting back to the insurance example. When studying the interaction between user and customer, there are many things required to complete the transaction that have nothing to do with the transaction at all! For example, the customer may want to run different scenarios, or the user may want to jot down a note or a telephone number if a call gets dropped, or the user may want to explore historical information during the transaction that has nothing to do with the user's new transaction (e.g. like how much he paid the last time he renewed his policy). The team should explore this interaction and be able to distinguish what a user desktop would look like in spite of a transaction. You may want to introduce concepts like a scenarios file with a scenario comparison tool. You may need to build a "post-it" feature where the user can jot down a quick note or that phone number if the call gets dropped. Maybe the users need a cut/paste feature to a small portlet window that can hold information you will need to reference throughout the call. Maybe they need an "undo" button, or multiple retractable points stored to being again in the interaction. These items are not transactional in nature, they are desktop enablers at best and will allow you to abstract the flexibility the users will need above the transactional requirements.

The next thing to do is segregate the components of a transaction from each other so they are simple and distinct, and can be "assembled" into the entire transaction. Avoid wizards (implied, or even explicit wizards) and sequence rules. Keep interactivity and the flexibility at the higher level, and the transaction element level self-contained.

The more complex the transaction is, the less likely you will be able to stick to a transactional workflow. Model the interaction, and the transactional elements will be able to be defined quickly, efficiently and not die of their own weight because you'll avoid confusing the transaction with the interaction.

How have you dealt with segregation of interaction and transaction in your complex systems? I'd like to know.

Large Project? Invest in a Continuous Integration Environment

Image via Wikipedia

I've always admired small, highly effective teams. The reality is that it is much simpler to be effective when dealing with small teams. The smaller the more effective. If we take this to the ultimate extreme, say a team of one person, there are no arguments, work is done and understood by everyone on the team.

In reality, most projects aren't done by just one person. Generally, communication and understanding becomes exponentially diminished with greater numbers of people on a project. Imagine running a project with 50 people. Regardless of all the design documentation, meetings and communication, there are bound to be misunderstandings and an assortment of problems.

That is why its important to constantly keep the code base in check. Sure, that's what QA is there for. However, what I've found is that with very few exceptions most QA teams have a heavy emphasis on functional qa testing, and often that is performed manually over and over again.

There is undisputed value in functional testing. But what is often overlooked, is the need to capture those functional tests and automate them. If you are on a large project, and haven't implemented test automation software, you are wasting money... lots of it.

However, that alone isn't enough. The old saying regarding quality testing is that "you have to build it into your product, not just test it at the end." And so, we now face the modern-age tester: the developer, the need for test driven development, and continuous integration.

TDD has developers create tests before code is ever written. Naturally, if you run the test before anything is written, it should fail. Once the unit of code is completed, the developer can run the same test and prove that his code works.

To accomplish this, Developers, especially those on large projects where code is flying around everywhere, need to develop xUnit test (e.g. jUnit for java, etc.) that prove to them, and others, that their code is working. Once the unit of code is developed and the junit test passes, they should migrate the code to another development environment: The Continuous Integration Environment (if you don't have one of these, you need to go procure one).

The continuous integration environment(s) are used to repeatedly run the entire suite of tests developed to prove the code works. This is critical in large projects, because undoubtedly at some point, someone will change existing code and will introduce regression errors somewhere else in the application. In fact, this is generally why teams begin to develop "snowflakes" (see my blog post on snowflakes) which is even worse.

Regression errors can be so reputation damaging to your team, that you ultimately would be wise to avoid regression altogether and invest in continuous integration. No one likes to hear from their users that something that was working fine no longer works.

Developers should immediately promote their code into a continuous integration environment after completing a unit of work and making sure that the x-unit tests pass. Continuous integration should run tests repeatedly many times per day. The person in the role of continuous integration tester, should promote the code to the next scheduled run, and include the tests the developer created to verify his code. This way, the developer can be assured that the continuous integration testing resource will know if other developers break his code.

If the test run passes, all is good and the developer moves on to the next task. If not, the continuous integration tester, demotes the code and sends the developer back to the drawing board.

Now, one common problem I've seen is when testers start changing tests to accomodate sub-optimal (i.e. crappy) code. HUGE MISTAKE... DON'T DO IT! This ultimately will ruin your project. It's absolutely critical keep the code pristine. Reject the bad code, and get that integration build right back to being pristine.... Ostracize that developer like a medieval noble tossing a scarlet fevered serf over the castle walls. Out! Now! Before he infects everyone.

Obviously, you're not going to fire the guy (at least not yet), but he needs to get the message that the code has to work without breaking the pristine code already in place.

Occasionally, you will come across changes that cause breaks (which is also normal, and is part of the cost of support automation and integrated testing in place). This will require the continuous integration tester to remove tests. However, don't ever do that carte blanche. Each test needs to be tagged with the person who created it, and serious discussions need to occur if the continuous integration suite tests are going to removed. Bring in the person whose test has been violated, and get their buy in, as well as any analyst that can validate the necessary change. Never just listen to the guy who broke the test... we know what he'll tell you!

What challenges have you experienced in your code when working large projects? I'd like to know.

Related articles by Zemanta

• How Facebook Automatically Tests Code: PHPUnit & Watir (allfacebook.com)

Agile Survivor: How to Avoid Getting Voted Off the Island!

One of the most interesting things about moving to Agile is the self-governing aspect of being on a scrum team. Agile provides tremendous transparency to its participants. Everyone is working together to get a goal accomplished, and they become accountable to each other each day in the daily stand-up meeting.

If you can't cut it, the team will eventually boot you from the island, ala Survivor. The only thing missing is the formal tribal council, the tiki flame lantern and the metal do-bob to extinguish your flame.

Though eliminating individuals really shouldnt be the goal of any organization, raising the bar and expecting great performance is.. and, if after numerous attempts at counseling, etc. the individual doesn't improve, this hard reality will ultimately pay off. Its actually kind of refreshing to management to witness "self-governing" teams take the action necessary to get results. So, from this perspective, Agile can be a great way to raise the bar.

However, Agile teams can get "clickish," and they can, over time, become stagnant in productivity. Why?

The Agile methodology tends to group individuals that will work together for many sprints, often for years. Teams get into a groove, and you may find yourself creating "high performance" as well as "regular" performance teams. But the stagnancy problem can occur at the peak of the maturity of an Agile team: when the team becomes comfortable with all its members. Typically, at this time, the team has come to love agile, each other, and they feel productivity is just fine.

But what if it isn't?

Image by Getty Images via @daylife

I recently attended an Agile conference in Lost Wages (aka Las Vegas), where a few senior types (old foagies like me) started asking about how to improve overall performance of Agile teams. There weren't too many good answers in the sessions. Responses seemed a bit defensive, and delivered with a "you don't get it" sort of attitude.

Well, I got news for some of those speakers. While the Agile experience is extremely rewarding to the team, project results are non-negotiable. What I mean is "Results are defined as Scope delivered within time and cost constraints." Businesses invest in IT projects because they want results, not just a great experience for the techies... and, don't get me wrong, when the experience is good and the employees love it, then generally, you'll have more dedication to the projects, and the team will get "more into" their work.... no argument there. I'm talking about overall results.

There is no argument that Agile is a game-changer. Users are brought in early, and your team is always working on high priority items with frequent deliverables. That's all goodness. But, as a manager, you need to be aware of various items:

1. Measure Personal Productivity- OMG! How can you single people out????? Tepper, you are a sacriligeous hypocrite! Stop insulting the Agile Gods! Look, as a manager, you need to make sure your team is still playing a little "Survivor".... Chances are there may be people getting carried by others, who may be creating negative inflow or regression errors. It's human nature to take care of our own, and Agile teams can become family. The reality is they have probably stopped optimizing their productivity. Measuring personal productivity and occasionally changing team members around will keep things crisp- but shake things up too often and you'll only introduce chaos.
2. Identify and Correct Negative Inflow and Regression Problems- What problems are being created and who is creating them? Ultimately, these items turn into additional work for the team. What is breaking in production? As the manager, it is your responsibility to deliver projects with high quality. Get your teams to find these and deal with the root cause.

Over the next few blogs we'll discuss techniques to measure and prevent many of the issues that slow down Agile teams.

What are your challenges in your Agile implementation? How are you optimizing performance? I'd like to know.

Related articles by Zemanta

• Jumpstart Your Understanding of Agile (eon.businesswire.com)

Contracts

Image via Wikipedia

Why is it that some deals go well and some don't? What is the secret to striking a great consulting deal and, if you're trying to sell one, what is the secret to closing a deal? What are the fundamental pieces to forming a deal and what can you do to avoid getting the short end of a deal?

Lets talk about deals, today. How to make sure contracts you enter into a good deal, and how to avoid bad deals. Lets cover:

1. Closing Deals
2. Why "Win-Win" deals are the only ones that work best
3. Terms and Conditions

Closing Deals

For a good number of years I worked as a leader in a couple of consulting practices. My responsibilities included selling and delivering deals. The one thing I learned early on when I was selling deals is that you need three things to get a deal closed:

1. A need
2. A budget
3. A sense of urgency

It's my experience that any deal that doesn't have all three elements on this list will not get signed. If you have two of the three, your deal won't close. I mention it only because it is very common to have the first two, but the signature authority will not sign without the urgency. Many of our sales people could talk to potential clients for hours on end about a deal... no trouble getting the meeting, even outlining a project plan and budget, but they couldn't close the deal. In the business climate, if there isn't a sense of urgency, your deal will be pushed to the back.

In other cases, there was a sense of urgency.. Lots of good discussion, but the sales people never checked to see if the people they were talking to actually had a budget. Lots of enthusiasm and need, but they were talking to the wrong people.

So, its important to realize you need all three.

Why "Win-Win" Deals are the Only Ones that Work Best.

It is all too common that the sharper negotiator gets the upper hand in a consulting deal, whether its the client or the contractor. However, when your procuring intellect, as you do in consulting, its important to recognize that you can damage yourself by breaking the other party. When you strike a "win-lose" the likelihood of any project success is severely diminished. These hardly ever work.

Lets say you're the consulting firm and end up on the LOSE side of a negotiation... The other people you will need to please on this contract are your partners/owners, and you will be forced to do everything in your power to make the deal work for them, and that includes subcontracting the work just so you can retain a profit. I find it amazing how often folks will revel in the fact that they just drove a supplier into the ground, only to be presented with the provider's "B" team. As a customer, your goal is to drive a great bargain, but to also recognize when things just look too good. When you push it that far, you're pretty much looking at a disaster in the making.

The opposite is true as well. If you are the consulting agency and you just got the sweet deal everyone has always dreamed about, be ready. It's only a matter of time before the client figures it out and starts working you out, quite possibly forever. Abusive deals may bring you great short term revenue, but can also backfire on you when someone finally figures out what is going on. Your reputation will be marred, and someone (you, or both you and the guy at the client site who signed the deal) will be shown the door. Greed may be good on Wall Street, but in the business of collaboration to successfully deliver a project, its not.

While its important to drive a good bargain, you will reach a point of diminishing returns. One way to avoid this is to figure out what is good for both sides. Contracts need to be established in a "win-win" style, especially where services are involved, or you will have a catastrophe looming in the wings. I have found in my role as a"client" who understands how consulting practices operate, that my most trusted partners will share their goals with me. Those include:

1. Profit they are trying to achieve
2. Contingency they are building into the project
3. Marketing / Reputation goals.

I, on the other hand, will share my goals with them as it relates to:

1. Assuring I get qualified and dedicated staff
2. Knowledge transition
3. Warranties

Time and Materials Considerations

I also like Time and Materials contracts with lots of signoffs. When I was in the consulting side, I would look for frequent signoffs from clients on deliverables throughout the entire project. I used to tell everyone in the practice "Your always good until the very last signature." What that meant was that if I had a client signature on a document that articulated the progress and the particular deliverable of the project we were in agreement at that time. I would make a point of saying to my clients that we would have many of these. I'd also pointed out that if they were unhappy with our services they always had the option to let us go, but I structured my frequent signoffs so we couldn't dispute any deliverable up to the point of the signoff.

I found that this no-nonsense style worked for both of us (clients and consultants) because it kept us focused on working on the project. In a fixed price contract, as a provider you are focused on meeting the absolute minimum required to maximize your profit. As a client you are focused on getting as much as humanly possible from your provider for the same amount of money.

Guess what. That doesn't work. That is founded on grandstanding and embedded beauracracy. There is no efficiency and streamling... no economy to the solution. A fixed price contract always has a winner and a loser, and often two losers. Do time and materials and you get quicker results, more elegant solutions and longer value from what will become a trusted relationship.

The Terms and Conditions

The terms and conditions is a critical piece of any contract. For anyone who's done contracts, you know that you will spend a good deal amount of time arguing IP. Keep in mind that the tools used to create your solution are rarely IP your provider can sell you. The techniques are also items consultants need to be able to redeploy. However, your specific algorithms and methods which your company owns are yours, and as a client you should protect your company from potential misuse with other clients of your consultants.

One thing I insist on doing in any terms and conditions negotiation is to make sure that whatever you discuss works both ways. Always ask yourself "Is this term fair to me and is it fair to you? What if we worded this exactly the opposite way.... does it still work?" I find that good terms and conditions are equilateral not unilateral in their structure. If you're not sure exactly whether the term you're negotiating is critical, ask for exactly the opposite, if the other side finds it untolerable, then there is an advantage that is probably inequitable. Avoid agreeing to those points.

Also, remember, that a skilled negotiator may have to give in on several points... Get a feel for what it is you are willing to trade for. Its not too uncommon to say "I'll give you this, if you give me that." In the end, the Terms and Conditions are just a structure for the delivery of the work. Often, the Ts and Cs get into a tug of war between each party's legal departments. So, it is important to be flexible, but both sides should strive to be fair.

UAT - The Hidden Time Bomb

Image by Getty Images via @daylife

So many software developers are familiar with the four phases of software development. No, I don't mean the SDLC that spells out Inception, Elaboration, Construction and Transition.... More like the SDLC that starts with the "Honeymoon" phase, the "disillusionment", the "trenches" and "catching up to expectations."

We've all seen it... Everyone is genuinely excited at the beginning of a project.. Enthusiasm is high as requirements are written, the users seem engaged... and even though the requirements are a bit difficult to read or maybe slightly unclear, there is a feeling from the users that the team gets it, so we move along. Somewhere at about the 50% mark it settles in. The users become a little more involved and the team realizes that the requirements didn't quite cover all of the features that are going to be needed. There is tremendous intensity in this phase (the trenches). Finally, you think you are there. You're coming to the end of your budget, the schedule dictates that you should be wrapping it up and do final user acceptance tests (UAT). The software will be deployed in about a month and you are ready to deploy.

So, here we are. UAT. All of a sudden people you've never met show up at your door and politely introduce themselves. You smile, turn to your team and ask... "who is that, again?" - They don't know either. The first few tests work just fine. The first day was a success, but there are a few more days to get through. On the second day, the users come up with a series of tests your team is unaware of and magically, like a surreal nightmare, your development train runs over that penny on the rail, and the cars begin to fall off the track, toppling all over the place. Your project is WAY off the mark.... It doesn't do this, and it doesn't do that....

You have lost control. That wonderful project of yours that you've reported as "Green" in your status reports for the past 6 months is now "RED" ... Blood red. You are caught off guard... you have lost your balance... you are in a free fall and you can't even see the floor. Sure, if we were in outer space there would be complete silence, but we're not... so management hears you screaming "Noooooooo!" as you fall, down... down... down... no end in sight... just a faint echo as they look into your development well and you fade from view. (dramatic, huh?).

Well, unfortunately, this is not all that uncommon. This happens a lot. UAT is the gotcha for the development team. There is nothing more in this world that raises the hair on my neck than a UAT surprise. As a project manager or development lead, UAT is the validation that you got it right, and for some reason, everyone gets so preoccupied with the development of the project that they miss the most important part of any software development... the acceptance of the code.

I can tell you, I have seen the best projects fail in UAT. Perfectly delivered, meticulous projects, that when reviewed were absolutely delivered to the contract. The development team did it, but the client wouldn't accept it. The worst case that I can remember ultimately took a small consulting practice out of business as UAT went on for months beyond the fixed price contract. The practice prevailed in arbitration, but the costs were overwhelming and the office was shut down. Perfect software development proven in an arbitration resulted in a failed consulting practice. Unfair. Oh, and in addition, the client who decided to go to arbitration lost a lot of money as well chasing the consulting company.

So, how can one avoid these kind of problems? Even if you deliver your project perfectly, as my friends in Dallas did. The first thing you need to come to terms with is that your contract should go well beyond the development of the project. Its easy to say, "we did everything right," they just don't want it, or its not my problem they don't get it. I guarantee that at a minimum you will pay, your reputation will suffer, your business will suffer.

The UAT phase is the most critical phase of your project. Does that sound ridiculous? Think about it. If you can't control the UAT, you are entering the "financial loss" phase of the project. And, let me be clear, when I say control, I don't mean "win the test of wills." I mean understand the amount of time required, the potential to make critical fixes, and assure the software will be used and accepted by going through ALL of the pertinent test cases required to accept it.

The biggest flaw I see in software development is the poor preparation teams make towards UAT. The best way to avoid UAT time bombs is to get your users to pony up and agree to all of the test cases they will run. Immediately challenge them when they identify them. I've listed questions below that you need to find answers to, and formally agree to, immediately after requirements and no later than the beginning of development (months ahead of UAT). Additionally, you need to revisit this list multiple times throughout development to make sure there will be no surprises in UAT. If you find that the answers are changing or growing, you need to plan and budget accordingly:

1. Who will be performing the UAT?
2. Do those identified represent 100% of the users?
3. What specific cases will you run? (get a list and full understanding of conditions that will be checked)
4. Are we certain that those cases will cover 100% of the acceptance ?
5. Which exception cases do we need to run in UAT? (outside of normal flow)
6. Are there any more?
7. When I document these, will you be able to provide an authoritive signature that states that successful completion of these tests constitutes acceptance?
8. If not, then lets keep working this document until you can.
9. Who will provide the final signature on acceptance?

The development of a well written and understood UAT plan is validation for your requirements. If the UAT goes beyond your requirements documents, your team missed the objective, and the software will not be accepted.

Furthermore, assign a lead who is skilled in change management to be the responsible person on the dev team for a successful UAT. This person should be meeting regularly with the client's organization to show them work in progress (what the system looks like and how it will be used) months in advance of the software's completion. They should understand concerns and help promote the use of the software. The feedback they receive should be shared with the development team and taken very seriously. When replacing an existing system, they should also take the time to understand the older system so that important features the users want/need aren't overlooked.

What UAT surprises have you experienced in your career? How could those have been mitigated? I'd like to know.

SOA - Maximizing your investment

Image via Wikipedia

The move to Service Oriented Architecture (SOA) is more than 10 years old now. The technology makes sense. It is rooted in the plug-and-play sort of mind set that is essential to businesses that want options with their platform solutions. No longer is integration direct. The integration strategy of SOA is based on web-services calls between systems. The promise of SOA is that with integration at the services layer, you can replace systems without having to stress out over where in the code all the integration points might be. Additonally, with a clear integration layer, organizations avoid the pitfalls of direct database access across systems. Services are distinct, clear and easily found and maintained.

All good. By all accounts everyone pretty much has arrived. Note: If your company isn't there yet, "plan to party like it's 1999," because you pretty much aren't living in this millenium.

There are a few things that are often overlooked in SOA implementations. I'll list them in summary here, and we can delve into them below:

1. Take the opportunity to architect in a "Vendor Broker" solution for third party providers.
2. Make sure accountability for SOA maintenance is well defined.
3. Avoid synchronous traps.

Here we go!

Vendor Broker Component

So, what is this Vendor Broker Component? Very simple. Its a tailorable component that governs how much business you give to third party providers. This is nothing more than a traffic cop directing traffic that you can instruct. A vendor broker is a key component to keeping your business operational costs down.

Why? Well, if you want to keep your costs down, and continually drive them down, it's a good idea to treat your suppliers as comodities providers. Lets say you get data from "Data Provider A." You have a great relationship with them and have done business with them for years. But now they inform you that they are going to raise their rates. You're stuck. It might take a few months to integrate to another provider, assuming you can get through legal in a short period of time. You don't have the leverage you need to keep them constrained.

A services Broker component puts the leverage squarely in your hands. The broker directs requests to different providers. You configure it so that you can route business to the provider you want to get the majority of the business, and if you change your mind, you can change it immediately.

Now, lets replay that meeting with Data Provider A. He wants to increase rates. You respond, with, "Wow.. I don't know, I was hoping you were coming in to lower rates." (I love it when this happens!)

They hold their ground. They know it will take you months to locate and implement a replacement provider. They're banking on the fact that this exercise would not be strategic for you. You probably don't have the resources available at this time to work on the new migration. They're banking on the fact that you have more important things to do than migrate away from them. They believe you are joined at the hip with them and they can get away with the rate increase. It's a small increase, anyway...you love their service, and you'll soon forget.

After the meeting, you immediately call Scotty in the engine room and tell him "Set the broker to only deliver 20% of the business to Data Provider A and give the rest to Data provider B." Minutes later, you're on the path to showing Data Provider A that raising your rates could really hurt their business.

Next Meeting:

Data Provider A: "Hey, we've been noticing your orders are down considerably this month."

You: "You guys are a bit pricey. We want to do business with you, and we'll continue to give you business, but you're going to have to do something about those rates."

They understand what's happening.

Tada! You get lower rates. You tell Scotty to turn up the thrusters for Data Provider A. Give it a few months, and the guy from Data Provider B will call next. Eventually, you get to replay the whole meeting with a new set of players (Data provider B).

Guess what, you're being fiscally responsibile to YOUR company by driving costs down. By the way, as a business person, its critical to not blow away any given provider because, if you do this right, you can leverage multiple providers over time and get unbelievably cheap rates. But to do that, there has to be competition for them. Take the time to do this right.

Accountability for SOA

If you are in a highly dynamic shop, where your employees are constantly changing maintaining and enhancing their platforms, its imperative that services ownership lives with the platform that built them. I know that sounds just too logical. However, I've seen (all too often) that the services build out is a "once and done" sort of event, and platform integration deteriorates as systems evolve. The key to SOA accountability is:

1. Assign an owner for the services in the platform.
2. Have them broadcast changes in the platform to all services consumers (other systems owners that use that platform's services) and
3. Make sure services testing is in place each time a release goes out.

Seems simple enough, but often overlooked.

A Final Word: Avoid Synchronous Traps.

There are two ways to communicate with services: Synchronously and Asynchronously. While SOA is a great way to architect solutions, be careful to not become overly dependent on synchronous communication. For those who may not recall what these two terms mean, synchronous requires a response before moving forward... Kind of like your 3 year old asking you a question. They follow you around and don't let up until you give them what they want.

Asynchronous communications allow you to submit a request, and move on... The response will eventually come. Its critical in SOA to try to push asynchronous interfaces, or errors will start piling up when your integrated partners experience their troubles.

To do this effectively, build a timing feature into your Broker component, and tie that to a service level contract with your providers.

That way, when you need something that feels "synchronous" (i.e real time quick response) you don't tie up resources if a provider is "down" and can reroute an order to a provider that is "up." A common problem is that IT people tend to think of asynchronous communication for lengthy transactions. That's a common misconception: Aynchronous doesn't mean that you should have to wait a long time before a response. You can build asynchronous integration that "feels" synchronous (i.e. expecting a quick response), but you should design your system to effectively handle outtages that aren't yours, and allow for a rewarding application user experience. If you build in too many synchronous integration points, your users will feel pain when any of those have slow performance or are down and they get stuck in your app.

What are some of the strategies you've implemented and challenges with SOA in your organization? I'd like to know.

Snowflakes

Image via Wikipedia

Have you ever been on a walk in a ski resort town right smack in the middle of winter. I used to go to Mammoth Mountain, California quite a bit to ski when I was younger. I used to love taking a shuttle into the heart of the town and going for a brisk walk at night. Everything was covered in snow. I can still hear the sound of the snow "crunching" as I walked across it on my way to a local restaurant or in and out of shops. The best part was when it would just start to snow and you'd see the snowflakes fall. I'd reach out and catch one and look at it. A smile would come across my face as I made out all the little branches of it. I'd think, "Wow! they say there are no two that are alike." So nice.

Reality check...(insert sound of scratchy record).... Back to work.

Well, snowflakes may be beautiful in Mammoth, but when you find one in your code, its a friggin' nightmare.

So, what is a snowflake? Well, once again I have to give credit to my good friend Igor who came up with this term. (the guy really is a genius).

A snowflake happens when people who either don't understand or care about the underlying software architecture begin taking short cuts they shouldn't be taking. They compromise the code-base by creating little branches of code that hang off the core to do a specific thing, a kind of

"one-off sort-of thing that's like that other thing in there, but I don't want to figure that out, so I'll just put this little thing in here so I can be done with this and move on to the next thing."

Rather than take the time to engineer the solution in the correct place, they take the path of least resistance and decide to invent a solution with disregard for enhancing or fixing the code where it belongs. Through ignorance or negligence, they have little regard for the impact of their actions.

Now, doing that once doesn't really constitute a snowflake. ... even doing it many times doesn't... That just makes the code sloppy. The snowflake occurs when you start compounding the branch, by adding more and more dependent code branches that hang off the branch, and the underlying circumstances or assumptions within the code are just plain wrong.

If an organization allows this to happen for too long, the problem becomes insurmountable. The bad assumptions are compounded over and over again and your team will become paralyzed. The snowflake becomes so large, that when you ask your staff to fix it, they'll tell you it can't be done or they don't know how. Here's another hint, when you ask for something you think is a reasonable request, they tell you it will take millions of dollars. It goes something like this:

You: "Hey, how long do you think it would take us to change this address validation?"

Developer: "Here, or everywhere in the code?"

(Not a good sign)

Developer: Probably 2 months and $100k.

You: "Uhuh. $#^&! We have a friggin' snowflake on our hands."

That's why it's a good practice to refactor. Let me be clear about this... Don't refactor every once in awhile.... (make a big smile) ... Like having a "special refactoring sprint." (hold that smile)

(stop smiling... look very serious)... Nope!....(still serious).

ALWAYS refactor.

Refactor like you are a Banzai tree grower shaping a tree or pulling the weeds out of your rice paddy. Keep the code pristine. There is NO room for snowflakes in software.

ALWAYS refactor.

When you get to the point that your software has become a snowflake, refactoring becomes impossible. It's too complicated and costly. No one will understand what to change or how to do it, and they become fearful they will break the code. You might as well start thinking about sunsetting this project immediately and begin writing the "Son of (insert your current system name here)" proposal.

Refactoring is a difficult concept to sell to the business. It's a lot like replacing the roof on your house. It doesn't really add any value to the house, but it keeps the water out. So, don't try to sell it to the business, just do it. Build it into your estimation model for any additional work you are about to undertake. You should be able to continuously refactor for about 10% of your project cost. If you stay on top of your code by building it into your culture, you'll be able to do it for less.

How have you dealt with fragile code in your company? Do you talk about refactoring? What are your thoughts? I'd like to hear them.

Gus