Monday, February 2, 2009

No Wonder Enterprise Software Sucks^H^H^H^H^H Is Low Quality

I am currently developing software for an IT group. This software is for the company's customers to manage their purchases, essentially. The software originally came into being in the early Nineties (using *shudder* PowerBuilder) and has been developed continually for 15 years. It is a big piece of software.

Like many enterprise applications, this one consists mostly of pages of data (lots and lots of grids) that can be interacted with in various ways, possibly leading to new screens with new grids, and so on. Given the current PowerBuilder interface is a desktop application (accessed via Citrix), the expectation is a rich, desktop-like experience.

So the company decides to bring the application into the 21st century. It is being re-written using web technologies using Java and Flex. Of course, these are the umbrella technologies; sitting underneath those umbrellas are Spring, iBatis, JUnit, JMS, LiveCycle Data Services, Anvil, and, of course, hundreds of database tables and possibly thousands of stored procedures.

These technologies lead to all kinds of architectural goodness: strong model-view-controller architecture, loose coupling between classes, separation of SQL from application logic, N-tier deployment, and abstraction at every turn. Architecturally, it is a marvel!

But for all the marvelous architecture, what does it take to actually understand a component of the system. From top-to-bottom:
  1. Find the Flex file that defines the component. Note that this may be either an MXML file or an ActionScript file. To get to the actual component you want, you may have to iterate multiple times here. Grok the UI code.
  2. Identify the ActionScript service, data objects, and parameter classes being used. Identify the server destination for the ActionScript service.
  3. Find the destination in the remoting XML configuration file (comparing string IDs). From that, I can find the name of my Spring bean.
  4. Find the Spring bean in my Spring XML configuration file (again, comparing string IDs). From this, I can find my Java service. Heaven help me if there is AOP involved that affects my code...
  5. Open the Java service. Now I have my business logic, but I may still need to find my persistence code, so now I can open my DAO.
  6. For any given method, find the iBatis IDs for the actual query being run. Luckily, our convention is the DAO and the iBatis XML file sit next to the Java service, or it would be a grep to find the actual file.
  7. Find the iBatis node in my iBatis XML file (again, comparing string IDs). Hope I'm not looking at a stored procedure...
  8. If I'm looking at a stored procedure, go to the RDBMS, open the development tool for it, and look at the procedure.
Once I understand the structure, I can limit this to only four places: MXML/ActionScript, Java, iBatis, and RDBMS. Of course, at runtime, I have to understand how data and flow control is being transferred between these.

It is too complex. I'm not a great programmer, but I am good at understanding context; yet I have a difficult time following this context. For the programmers who are on the bottom half of the bell curve, it must be impossible.

My point is not to rail to Flex and Java. Honestly, it wouldn't look a whole lot different using AJAX and Python (assuming use of an ORM and framework like Django). My point is that the architectural complexity of these applications inhibit a person's understanding of the code. Reducing the architectural complexity increases complexity in other areas (such as increasing the number of responsibilities a piece of code has), so I don't believe it is just a case of over-architecting.

When I compare the "fun" level in this, versus the fun level of the applications I was coding 15 years ago, I understand why my job isn't as enjoyable. The real danger, as I see it, is that (especially in many IT shops where the average developer is...well...average) we've overspent our complexity budget. Without actually doing anything, applications are becoming too complex to understand, build, and maintain.

My experience is that most developers give up trying to understand and just do things by convention. Look to a working sample; see how that was done; copy, paste, and change the body of some method. There may be one or two people who really grok the whole project, who are always called in on the challenging problems. The remainder of the work is done by faith alone.

As a result, is it any wonder that enterprise software sucks is low quality? More importantly, is there a better way?


Dmitriy Kopylenko said...

I would have to disagree. Software solving complex problems cannot be simplistic. And having a sound layered architecture is very helpful in the long run for relative ease of maintenance/enhancements. I'm speaking from experience of building layered Spring-based systems for 6+ years now.

And I believe that "below average" programmers who always complain that they don't understand something without trying to understand, are in the wrong business. It certainly helps if the organization has passionate senior developers who do help junior programmers to become better developers and make them appreciate sound software architectures of today's complex world.

As a side note, take a serious look at Grails which does a good job of reducing some of the complexity in the enterprise Java land.

HighTechSamurai said...

I would have to disagree with the other commenter. From the post, I think that main problem is the use of Flex :-) but I know that is just me.
You can't talk about who is and isn't in the right business - you can't ask people to leave and it's not normally your job to hire/fire peers.
Finally, having also developed applications for six years; my latest being Spring/Hibernate/Struts, as soon as software approaches a certain level of complexity it should either be more than one piece of software or redesigned.
The original post is right on.

Dmitriy Kopylenko said...

Struts - makes me puke.

Try Grails FTW!

Anonymous said...

I agree with Travis, but I would also like to point out that in addition to creating unnecessary complexity, many "architect wannabees" grab the latest cool framework and try to jam that into the project so that their resume looks great when they inevitably, usually sooner rather than later, move on to their next job. It is like a sales guy who can't sell anything, except an "architect" who doesn't know how to write code leaves a wake of garbage in his path, whereas a bad sales guy just never sells anything.

Java has become the refuge of the erector set brigade. If the erector set doesn't have the part that you want, too freaking bad. Even more startling, that type of approach has not only become acceptable, it has become desirable. That is also why enterprise software SUCKS!

Bruce Conrad said...

Great post, Travis. It can be very trying to find the exact spot where a defect occurs. It's also worth pointing out that well over half of all developers are "average or below". There's got to be a simpler way, or better tools, or something to address this issue. said...

The idea of the "below average" programmers bothers me here because it pollutes this debate.

It is very easy to point fingers to the developer, when the topic is on the complexity of the software.

I think the initial poster is right in raising the point on the complexity of code and the difficulty to support it.
And this is true irrespective of the level of the programmer.

Sitka said...

The new mantra should be "all programmers are below average", reverse grade inflation.

The inherent problem is the iterative layering of abstractions. Every organism has junk DNA. Every added abstraction adds to the junk DNA for the system being described. Abstractions are used for us humans to get a grasp at the problem. Anytime we can't figure out the system, we will add an abstraction. If we could remember 1000 things with same clarity as 3 our code would look much different. I think it would look like noise, channel 3.

When I look at old code bases and what Travis describes isn't just an issue for enterprise software but all software that has grown large is that domain logic gets diluted across so many lines of code and so many classes that it is incomprehensible to anyone, not just below average programmers. Business logic spread like a thin pate so as to be tasteless.

Here is a simple scenario, how about logic that says, "if someone has more the $100 dollars worth of physical goods in their cart we charge them 80% of actual cost to ship those goods via the carrier of their choice." Now how many lines of code does that take to express in your system. And how many places does it have to be checked/called/double checked, etc?

I think the use of DSLs (domain specific languages) is a huge step in the right direction. See the podcasts on

Funny thing, the top link on se-radio is about MPS a domain specific language generator from Jet Brains!

Thanks for the post, I think the message needs to get out and the software world is coming around slowly. It will ebb back towards suckage, but I would like to have lesscode to deal with. Languages on the JVM (clojure, javascript) and the new module system in java 7 are very exciting. As a developer I would love to think in ideas and not shovel klocs around.

Sorokan said...

Very good post. Maybe I'm not a good programmer too, but I'm earning money this way for 14 years now, and I know that some colleques share your impression. I know the ugly copy&paste procedural hell as well as the typical modern framework stack hell you describe. I realy like the wonderful ideas in all those frameworks, seperation of concerns in aop and declarative programming but... every additional framework, technology and concept has a cost:

1. the cost for writing code for every layer and finding the route. Simple things just need too much time because of this constant cost.

2. the learning courve is multiplied.

3. the cost for everything that doesn't fit in the frameworks oppinion about the world.

4. from a customers perspective - will it be easier to find experts in 20 years knowing all 10 Languages and 12 Frameworks in your currently redesigned state-of-the-art application than finding a cobol expert today?

For DSLs: I tried it for a long time and had to give up finaly. In theory it serves a wonderful clean and compact description of my application without any noise. But the DSL and the generator isn't for free - in fact all of the dirt is just sweeped under the carpet, and because I have to deal with a more general approach I have to solve much harder problems. And what if I want to combine it with some other domain? Maybe DSL are just suitable for some niches.

In my experience handling complexity means:

- not doing something
- using just one or two simple and right abstractions
- being brave enough to accept dirt
- reimplement it from time to time
- knowing some nice tools

- throwing away every tool / server / framework / layer that isn't essential

Gwen said...

Sounds like you have not
done any maintenance for a project
done by a group of dudes without much of a well known layers/framework. Maintenance for a project like that sucks much more and once you go through that experience you will change your point of view. I would say when
original developers are out of reach in the project like that then the project is pretty much dead.

Personally I hate these frameworks(seems like every month someone comes up with a new one) because of the learning curve, for example the task
of getting a return value from a stored procedure in IBATIS hooked up to .NET/SQL Server is still a mystery to me after spending 2 hours trying to do that. (while it would take 1 min
directly using .NET)

However I have to say, in most cases I would prefer to dig through implementation of well known framework and less through your custom logic.