Successfully Delivering Complex Systems

Image by dvidal.lorente via Flickr

Have you ever worked on a large transactional system? The complexity of a large transaction, such as the development of insurance quote or loan origination system where there are many integration points with third party data providers, often experience navigational challenges or become too complex with the need to reconcile data. They are often too difficult for users to work.

Typically, what happens in these systems is the development team is continually confronted by end users with demands for increased flexibility. Development continually attempts to add more flexibility into the system, but the code becomes too complicated. The development team and the users become more and more frustrated as efforts bring little in the way of results to the problem.

Large transactional systems that are designed poorly usually die of their own weight. As more and more complexity is handled, the code becomes considerably more brittle. Often good coding practices are bypassed, and you just end up with a bowl of spaghetti in your implementation.

So, why is that? Intuition would lead you to think that the more you attack a problem the better the end result will be. But, it never works out that way. The more you work to include new features, shortcuts, hyperlinks, etc., the more messy the system becomes.

The problem lies in the approach to handling complex transactions. Most teams do a good job of identifying the transaction that needs to be modeled. They immediately go to work to develop a solution that almost always ends up being some sort of logically organized wizard. The application may have the ability for navigation, say with a few quick navigational features (such as tabs). Even though there is an appearance of an orderly transaction, you quickly finds that there are plenty of dependencies between the tabs. As new requests come in from users to handle alternative workflows, you quickly discover that in-line validations often fall apart as data dependencies exist between various sections of the transaction. The system quickly exposes its inflexibility. One good sign you are getting into trouble is when small requests seem inordinately long to solve.

The way to handle complex transactional systems is to stop driving all solutions and alternatives to the transactional level and begin thinking about the interactivity required by the users when they collect the data. In the insurance industry, it is not uncommon to spend 15-30 minutes on a single transaction, such as a quote or application. Begin by acknowledging that there is absolutely no way that ANY transaction will ever go from top to bottom in an orderly fashion (ever!). The interactivity your user will face with his customer will require ultimate flexibility when assembling the transaction.

So, rather than dealing with how to navigate at a transactional level, start thinking about the tools that are necessary for the user to successfully navigate his interaction with his customer. Once those tools are in place, your user will be able to quickly navigate through his interaction, and be able to deep dive into the transaction when and where it is appropriate. By not confusing the interaction with the transaction, you will be able to deliver simple, tight, and distinct elements of your transaction and (ultimately) avoid creating an unworkable mess.

So, lets see this in practice by getting back to the insurance example. When studying the interaction between user and customer, there are many things required to complete the transaction that have nothing to do with the transaction at all! For example, the customer may want to run different scenarios, or the user may want to jot down a note or a telephone number if a call gets dropped, or the user may want to explore historical information during the transaction that has nothing to do with the user's new transaction (e.g. like how much he paid the last time he renewed his policy). The team should explore this interaction and be able to distinguish what a user desktop would look like in spite of a transaction. You may want to introduce concepts like a scenarios file with a scenario comparison tool. You may need to build a "post-it" feature where the user can jot down a quick note or that phone number if the call gets dropped. Maybe the users need a cut/paste feature to a small portlet window that can hold information you will need to reference throughout the call. Maybe they need an "undo" button, or multiple retractable points stored to being again in the interaction. These items are not transactional in nature, they are desktop enablers at best and will allow you to abstract the flexibility the users will need above the transactional requirements.

The next thing to do is segregate the components of a transaction from each other so they are simple and distinct, and can be "assembled" into the entire transaction. Avoid wizards (implied, or even explicit wizards) and sequence rules. Keep interactivity and the flexibility at the higher level, and the transaction element level self-contained.

The more complex the transaction is, the less likely you will be able to stick to a transactional workflow. Model the interaction, and the transactional elements will be able to be defined quickly, efficiently and not die of their own weight because you'll avoid confusing the transaction with the interaction.

How have you dealt with segregation of interaction and transaction in your complex systems? I'd like to know.