Does Spring scale in large projects?

The short answer is that it does not, from a development point of view. Actually it kind of scales better the less you use it. That sounds like something that would come from someone who hates Spring, but I don’t. The last three iterations of my project I actually created a lot of Spring xml-files, while supporting other developers when they removed Spring xml-files from other places in the codebase.

So what’s the story here? Simple answer again: You should have a good reason for doing the stuff you do. This reminds my of a simple strategy I had when Spring was up and coming: Spring is good, we’ll use it any way we can. I suspect this strategy for being wide spread, but without any good reason, except lack of alternatives. The motivation often being the need for an IoC container to support testability and transaction management.

So what happens if you apply this simple strategy to a large project? Being in the retrospective mode I will try to sum up some of the anti-patterns that emerged while applying “the standard approach” with Spring in a large project.

One class, one bean definition

Let use start with a classic example. You have hibernate in your stack and your application needs functionality for managing users. You map your table in the database to a user-entity-class, add a repository and a service to have your layering in place, and your application have what it needs to manage the data. The applicationContext could look like this:

<bean id=”userRepository”>
<property name=”sessionFactory” ref=”sessionFactory”/>
</bean>

<bean id=”userService”>
<property name=”userRepository” ref=”userRepository”/>
</bean>

Does this approach scale? Let’s say the codebase contains thousands of classes. I’m sure that Spring gladly will parse those thousands of bean definitions, and proxy them all if there are any advices around. Given the cpu, memory and time, the applicationContext will scale from a technical point of view. But that’s not the point, from a developers point of view this is a nightmare, because loading the applicationContext takes a serious amount of time. If daily development involves the applicationContext you could quickly have your own version of the #1 Programmer excuse for legitimately slacking off:

“My code is loading the spring-appctx”

The cure

So how do you remove this pain? Simple answer: do it like you did before, with code. If you have a good reason for a bean to be spring managed, keep it in there for now. If you argue that you will lose benefits of spring by taking beans out of the container I would recommend that you evaluate the cost it has on your test-code-test development cycle. When in need to speed up a test-code-test cycle, or a build for that matter, reducing dependencies to the applicationContext is often low hanging fruit. The new code could look like this:

public class UserRepository {
private SessionFactory sessionFactory;
public UserRepository(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}

public class UserService {
private UserRepository userRepository;
public UserService() {
}

public UserService(SessionFactory sessionFactory) {
this.userRepository = new UserRepository(sessionFactory);
}
.. .
public void setUserRepository(UserRepository userRepository) {
this.userRepository = userRepository;
}

This simple approach reduces the applicationContext to the following

<bean id=”userService”>
<constructor-arg ref=”sessionFactory”/>
</bean>

I have kept the default constructor and setter for UserRepository on UserService, to keep the class testable.

One module, one spring-configuration

One characteristic of a large project is that it often consists of several modules, where some modules are applications and others are reusable service-modules. Often there is no need for remoting so the service module is simply added as a dependency to the applications that needs it.

Again we start with a classic scenario and look at a standard approach to integrate the services into one of the applications:

app-1-applicationContext.xml:
<beans>
<import resource=”service-module-a-applicationContext.xml”/>

So what could go wrong here? From the example you can see that the application has a explicit dependency to service-module-a. But since we a working on a large project we could have the case where service-module-a also has a dependency to service-module-b, since some of the services in module-a uses stuff from module-b:

service-module-a-applicationContext.xml:
<beans>
<import resource=”service-module-b-applicationContext.xml”/>

If you add the complexity of having multiple development teams spread out on the different modules you will experience that nested imports in Spring is a recipe for disaster, it scales one level at most. There’s no better way to loose control with your application than being downstream of spring-configuration-imports. Small changes in a upstream spring-config have the power to short circuit downstream clients. If the change happens in a module that is transitive to your application the job of finding the problem could be time consuming and frustrating at best.

If you have ever experienced how a circular dependency between bean definitions in a upstream module have randomly knocked out beans in the downstream module, you know what I’m talking about.

The cure

Simple ansvar: Don’t go there. Don’t import spring-configuration from upstream modules, especially those that don’t belong to your bounded context. As a client of a service you’re not interested in the details of what it takes to build a service. Neither are you interested in services that your application don’t use, there is no need to be exposed by the configuration needs of those services. The important thing a client should provide is the actual dependencies that is required to instantiate the services they need, such as a datasource or a prepared sessionfactory.

This means that the only modules that are allowed to have spring-configuration are applications, which actually matches well with the term applicationContext.xml.

The new code could look like this:

app-1-applicationContext.xml:
<beans>
<import resource=”app-1-app-code.xml”/>
<import resource=”app-1-module-dependencies.xml”/>

app-1-module-dependencies.xml:
<beans>
<bean id=”userService”>
<constructor-arg ref=”sessionFactory”/>
</bean>

By putting application code and module-dependencies in separate configuration files you have a separation between the two that lets you easily write tests that mock out all the dependencies.

Basically you have control, as opposed to import a world of configuration files under active development, never know what to expect.

Transaction management at the class-level

One of the selling points of Spring is that you have declarative transaction management, the freedom to start a transaction anywhere you want. The pain you can experience from this is that transaction boundaries makes it hard to reuse a service, or declarativ transactions in the code are just wrong. For some reason it actually happens that propagation level ‘never’ is used on a service, I suspect not from the right reasons. But this is a true partykiller if you want to reuse the service in a transaction. Anyway, how often do you need this freedom?

The cure

Simple answer: Less freedom, and no freedom at all at the service-layer, the service should expect an active transaction. Give the responsibility to the application, and define simple rules for when a transaction begins, and when it commits or roll back. If you are developing an application like spring batch this matches perfectly, since an essential part of that framework is tuning how much you do in a transaction. In a webapp you could go with a dead simple approach: delegate the responsibility to a servlet-filter, which means that the developers in most cases don’t have to think about it at all. Same with a webservice application, go with the filter. If simple is enough, go with the simple stuff.

Summary

It might be “old school” to use xml for spring configuration, now that you have annotations and component scanning, but we startet out with xml and stuck with it. I think the problems will remain the same, whether you use annotations or XML.

So does spring scale? I asked @miss_haugen at javabin today, she replied that there is nothing wrong with the framework, it’s often how it’s used that causes the problem. Good answer.

If you have any comments feel free to drop a line, and I will try to answer and possibly give more context, which is an essential part of any experience with any tool or framework.

Posted in Spring | Comments Off

Technical debt – How much can be justified?

That’s the ambitious title of my presentation that I held at Statens Pensjonskasse on their agile day. Technical debt is a wide topic, so having an understanding of what it means is important before the discussion on how much can be started. I think the TechnicalDebtQuadrant is useful for introducing non-technical people to the different aspects of technical debt.

To justify taking up technical debt is easy, it could be what the project needs in order to be a success. Not solving all problems at once and focus on delivering customer value is a good agile tradition, and here is where technical debt in the form of shortcuts come in handy. The risk is that the shortcuts are too many, and decreases the quality so much over time, that the project could be held to a standstill.

So again, how much can be justified? Of course that answer can’t be given in numbers. I gave my audience two examples to consider. The first example contained a design shortcut where technical debt could be said to represent 50% of the initial budget. The other example represented a microscopic debt in comparison. So what was the conclusion about these two scenarios? It is not how much, but how it’s done, that is important.

In general I would suggest the following rules when considering technical debt:

  1. Don’t take up a lot of small debt, the interest is high
  2. Be careful, not every shortcut is a smart choice
  3. Be methodical, debt should be paid
  4. Be open about it, give the customer the right expectations about future maintenance and development
Posted in Agile development | Comments Off

BDoc, so far

I have been working with BDoc for almost two years now. The characteristics of the work is that of R&D, small steps, all the time gaining more insight. The first public appearance of BDoc was on The Server Side 2008, in the article BDoc – Supporting Behavior-Driven Development. Since then I have developed BDoc further, and also getting valuable help from the community.

Micael Vesterlund joined the project after a while and has been the driving force behind the scenario support in BDoc. Espen Dalløkken had a dive in the codebase and changed the navigation principals from something that worked on paper to something that would work on the web.

After answering on a CfP at this years JavaZone I had a go with presentation driven development, promising a demonstration of BDoc with an application that should calculate pension for Norwegian citizens.

To view your work as a product that should be sold to an audience changes your mindset. Gone are the love of technical challenges buried in the code, features are the new vine, and I wanted them ready for the presentation. It kind of reminded me of a project several years back, where the customer announced that the software would be presented for its users, three months from the current date. They would be flown in from all over the country, and the airport hotel was already booked, the deadline was set.

Just like my older project I had to take up some deliberate technical debt to meet the deadline, prioritizing only functionality part of the demo. The debt is now paid and BDoc is again ready for new features, and also ready with it’s second demo, the game of yatzy. A snapshot is given below:

yatzy-bdd

Just like Micael and Espen have joined the project, with comments and code, I would like to hear your opinion about BDoc. Code are always welcome, but comments are also something I value.

BDoc is installed as a maven plugin, if you are new to maven, all you have to do is to download it from http://maven.apache.org, and install it. To make it real easy to test BDoc I have made a zip out of the yatzy example. Download it, write mvn install in the project directory, and you will have the bdoc report generated in the directory target/site/bdoc.

There are some hidden links in the bdoc report, left overs from the presentation. Find them and you will see the code behind the documentation, just like in the snapshot.

My spam-filter is currently not working on my blog, but I appreciate any comments sent to perottobc@gmail.com.

Posted in Behaviour-Driven Development | Comments Off

Efficient testing – My JavaZone presentation

Efficient testing – My JavaZone presentation

 

Testing is more than verifying a set of requirements, in my view it is also a tool for having the code ready for change and new requirements. But it is also possible to end up down the wrong path, where tests works against change, by breaking all the time. Choosing the right tools and using them correctly is key when you want to embrace change.

 

At this years JavaZone I held a presentation about efficient testing. If you didn’t get to see me live you could check it out here (in norwegian).

Posted in Agile development, Behaviour-Driven Development, Functional testing | Leave a comment

Just BlazeDS

That’s the name of the next open source project I’m getting into. It was started out by Espen Dalløkken in February 2008, as an investigation on part of the Live Cycle (LC) suite from Adobe Labs, called BlazeDS. It is open source and makes it real easy to integrate the client flex application with the server.

You can find the source code at http://just-blaze-ds.googlecode.com

Technology stack:

  • Flex / Air
  • Blaze DS
  • Spring
  • Hibernate
  • HSQLDB
  • Jetty

Of course this is a good opportunity to promote BDoc, so the application will be documented with user stories generated from the test code. Since functional testing of the GUI is a nice thing to support, we will add that as well. Hopefully with some help from the FunFX guys.  The GUI will also get an overhaul.

Posted in Flex | Leave a comment

BDoc on TheServerSide.com

Read about BDoc on TheServerSide.com, and what other people have to say about it. Please join the discussion as well.

                                           
Posted in Behaviour-Driven Development | Leave a comment

GUI Test Patterns

I had an interesting discussion about how to test applications through the user interface on JavaZone. I promised to sum up my experiences, so here it is:

Setup Page

Scenario: You have an application with a lot of pages and want to test something a page twelve.

First try: Make your GUI test tool fill out eleven pages and finally do your stuff at page twelve.

Consequence:

  1. The test takes a long time to run, because of the eleven pages that needs to be filled out.
  2. The test is fragile, because an error on any of the eleven pages will break the test
  3. The test is a maintenance nightmare, changes in the chain of pages will require the test to be updated. A lot of tests like this will require a lot of unnecessary work.

Solution

Make a setup page in your application that can be used to generate test data. Rewrite the test so that it first uses the setup page to generate test data, and then takes the shortest route to page number twelve. The Mock Client pattern is a good starting point for implementing the setup page. By simulating a client directly against the service layer you also have a pretty nice integration test. It’s important that the mock clients run in Continuous Integration, if they break you want to know it as soon as possible. Remember to secure the setup page, so that it is unreachable in production.

Object per page

Scenario: You are starting out with automatic GUI-testing and are writing tests with the low level API that comes with the tool.

Problem: The the name of a button has changed and a lot of tests are breaking. Because the low level API has been used in different tests, we need to change the name of the button several places.

Solution: Make your own high level API of your application, on object per page. The page object could have methods like:

  • page.go_to
  • page.add_row( “col value 1”, “col value 2” )
  • page.save

Result:

  • The tests are easier to read
  • The tests are faster to write
  • A change in a page will only result in a change in the page object

Integration page

Problem: It’s difficult to run tests because other systems needs to be up and running and test data must be available and synchronised.

Solution: Make an integration page where you could change live integration points with mocks. When the tests run in Continuous Integration use mock integration, but keep the test data as close to live testing as possible. If key test data is equal in both live and mock testing, the tests can be reused for live testing. At the end of the iteration, turn to live integration, and run the tests to verify expected behaviour. Remember to secure the integration page, so that it is unreachable in production.

Words of advice

  • Define a maximum test time for your suite, minutes are acceptable, hours are not
  • The number of good tests are limited, to many tests will burden the project
  • Keep track of tests that fails when the application isn’t broken, fragile tests belong in the garbage bin
  • Migrate your tests into integration tests if you can, sounds difficult, but challenge yourself and the team. GUI testing is more expensive than integration testing.
Posted in Functional testing | Leave a comment

You code, BDoc documents

Meet me at Smidig2008, 9-10 October.
This is the introduction to my lightning talk:

Documentation will often be outdated the moment it has been written. Documenting is often considered to be a boring task as well. BDoc is a tool that produces documentation of the system from unit tests. When the a test change, the documentation will change. BDoc also supports Behaviour-driven development (BDD) and could be used to associate tests with user stories.

This talk is for you who have a need for updated documentation of a system. You could be a developer, tester or a responsible for the system being documented. BDoc could also be used to document behaviour that is unpractical to implement as functional acceptance tests.

BDoc is written for Maven2 and supports tests written in JUnit3, Junit4 and TestNG. Current version is 0.7.8 and is mature enough to use. Read more at http://bdoc.googlecode.com.

Read more about my norwegian talk here

Posted in Behaviour-Driven Development | Leave a comment

BDoc

BDoc is the name my open source project that extracts documentation from unit tests. The ‘B’ stands for behaviour and reveals that BDoc also is a tool for Behaviour-Driven Development (BDD). My goal is to promote Test First and let BDoc be the catalyst for test names that describes behaviour. BDoc itself doesn’t solve anything, it simply transforms the test code into documentation. How good this documentation is depends on what tests exists and how they are named.

Read more at http://bdoc.googlecode.com.

Posted in Behaviour-Driven Development | Leave a comment