Easier web app testing by mapping entities to UI objects

Automated, browser-based testing is a key element of web application development, benefiting both simple and complex applications. Writing effective tests for browser-based apps can be a complex, tedious and often repetitive task. In this post, I will be discussing a general approach to write meaningful, loosely-coupled UI tests for web applications by going beyond the Page Object Design Pattern into a more fine-grained approach I call ‘Logical entities to UI object mapping‘. I will show code in written Java 8 leveraging the Selenium and Selenide frameworks to show examples of the method described.

Layers of web app testing responsibility
Layers of web app testing responsibility

In web development, a common component used to perform browser-based testing is Selenium, which is a suite of frameworks and tools to automate browsers. All code in the Selenium projects is licensed under Apache 2.0. Selenium WebDriver is the most important component, which exposes an API to control several web browser and browser engines, and includes several language bindings: Java, C#, Python, Ruby, Perl, PHP and Javascript.

Page Object test design pattern

Selenium and the WebDriver API are very flexible and relatively easy to use. Make sure to check out the Test Design Consideration page of on the Selenium documentation for tips on how to structure your code, specially the Page Object Design Pattern section. There is a great post by Martin Fowler on the subject, plus lots of resources and blog posts on the web.

The following example is from the Selenium documentation, showing how coding a test using the straightforward WebDriver API can result in complex code that mixes many concerns:

/***
 * Tests login feature
 */
public class Login {

        public void testLogin() {
                selenium.type("inputBox", "testUser");
                selenium.type("password", "my supersecret password");
                selenium.click("sign-in");
                selenium.waitForPageToLoad("PageWaitPeriod");
                Assert.assertTrue(selenium.isElementPresent("compose button"),
                                "Login was unsuccessful");
        }
}

This kind of code, specially when adding lots of selectors and adding more test logic, can quickly transform into hard to maintain, spaguetti code.

When using the Page Object pattern, the original test code can be refactored into a more OOP class that has more semantics:

/**
 * Page Object encapsulates the Sign-in page.
 */
public class SignInPage {

        private Selenium selenium;

        public SignInPage(Selenium selenium) {
                this.selenium = selenium;
                if(!selenium.getTitle().equals("Sign in page")) {
                        throw new IllegalStateException("This is not sign in page, current page is: "
                                        +selenium.getLocation());
                }
        }

        /**
         * Login as valid user
         *
         * @param userName
         * @param password
         * @return HomePage object
         */
        public HomePage loginValidUser(String userName, String password) {
                selenium.type("usernamefield", userName);
                selenium.type("passwordfield", password);
                selenium.click("sign-in");
                selenium.waitForPageToLoad("waitPeriod");

                return new HomePage(selenium);
        }
}

There are also fair warnings on the tradeoffs involved in applying the pattern, as described in this blog post.

In this case the warning is focused on the risk of grouping unrelated semantic and intent in aggregation ‘page’ classes.

In my case, I also tend to stay away from ‘page’ classes and write UI abstractions around the application’s logical entities. Logical entities are mapped to UI objects and then accessed naturally through their relationships so user actions and navigation are kept intentional. To also keep code within the domain of UI testing, most UI class methods are explicitly called after common user browser actions like clicking, moving the mouse or entering text into forms. In practice I have found that this combination is a ‘sweet spot’ combination of useful abstraction, loose coupling, testability and practicality.

The following diagram illustrates the concept for a typical content management system three level master-detail:

Grouping logical entities in semantic web UI testing, showcasing method names
Grouping logical entities in semantic web UI testing, showcasing method names

I will be showing next some specific examples of applying the ‘Logical entities to UI object mapping’ pattern in Java 8, using Selenide on top of Selenium to automate the browser.

Introducing Selenide

Surprisingly less well-known than it should be, Selenide is a great Selenium WebDriver abstraction layer written in Java that makes it easier to manipulate browsers and write tests than just by using Selenium. Selenide simplifies writing tests based on Selenium with a surprisingly intuitive interface. To demonstrate the simplicity, here’s an example from the getting started:

@Test
public void userCanLoginByUsername() {
  open("/login");
  $(By.name("user.name")).setValue("johny");
  $("#submit").click();
  $(".loading_progress").should(disappear); // Waits until element disappears
  $("#username").shouldHave(text("Hello, Johny!")); // Waits until element gets text
}

The interface offered is easy to use and you usually don’t even need to look at the documentation. Waiting for elements to appear is implicit in many parts of the code and Selenide generally works as you would expect. I encourage you to try it, write a few tests and you will be surprised at the simplicity.

Regardless of the added convenience and abstraction of the Selenide library, we can still end with spaguetti code if we are not careful. For instance, if we use CSS selectors to access one UI element with $(org.openqa.selenium.By seleniumSelector) and want to change related classes in the client-side, we need to go everywhere in our testing code and change the CSS selector references.

Separating UI handling from tests

In a OSS project I’ve been recently working on (Morfeu), I am doing extensive testing of the web UI and I’m using the Logical entities to UI object mapping approach all over the place. For instance, in the project there is a list of catalogues (like a master-detail listing) as a logical application entity. Each catalogue has a list of different catalogue entries, which in turn contain documents (not shown in the example), completing a simple three layer of logical app entities. The code to operate and access the master catalogue list is as follows:

public class UICatalogues {

private static final String CATALOGUE_LIST = "#catalogue-list";
private static final String CATALOGUE_LIST_ENTRY = ".catalogue-list-entry";

public static UICatalogues openCatalogues() {
	return new UICatalogues();
}

public UICatalogues shouldAppear() {
	$(CATALOGUE_LIST).should(appear);
	return this;
}

public UICatalogues shouldBeVisible() {
	$(CATALOGUE_LIST).shouldBe(visible);
	return this;
}

public List allCatalogueEntries() {
	return $$(CATALOGUE_LIST_ENTRY).stream().map(e -> new UICatalogueEntry(e)).collect(Collectors.toList());
}

public UICatalogue clickOn(int i) {
	List catalogueEntries = this.allCatalogueEntries();
	int count = catalogueEntries.size();
	if (i>=count) { 
		throw new IndexOutOfBoundsException("Could not click on catalogue entry "+i+" as there are only "+count);
	}
	catalogueEntries.get(i).click();
	return UICatalogue.openCatalogue();
}

}

To start with, public static UICatalogues openCatalogues() is used to open the catalogues list, it’s a static method as one of the constraints is that there is only one catalogue list and that it appears as soon as the user loads the application. A static method is a convenient way to access the ‘catalogues’ object instance as opposed to a public constructor, which maps to the application semantics. If there was a need to change the behaviour into requiring user action to load the catalogues (such as a mouse click) the implementation could be changed without changing the caller code. If the change were to be quite radical and significantly change the logical entity relationships, like allowing multiple catalogues, the API and caller code could (and should) be refactored to reflect the big change in application behaviour.

The methods UICatalogues shouldAppear() and shouldBeVisible() are pretty self explanatory, and can be used to ensure the catalogue list loads and displays properly.

The methods are quite simple themselves and the usage of Selenide is pretty much self-explanatory.

The next method List allCatalogueEntries() is used to obtain the list of catalogues, and takes advantage of Java 8 streams, mapping the list of found elements into new UICatalogueEntry instances:

public List allCatalogueEntries() {
	return $$(CATALOGUE_LIST_ENTRY).stream().map(e -> new UICatalogueEntry(e)).collect(Collectors.toList());
}

A stream of low-level catalogue entry elements found by Selenide is mapped to new catalogue instances and then collected into a using Java 8 list collector.

UICatalogueEntry‘s constructor accepts a SelenideElement to provide each catalogue entry instance with its local context. SelenideElement instances are a wrapper of Selenium elements and are the basic building blocks of testing using Selenide.

The last method UICatalogue clickOn(int i) is a convenience method to click on a specific catalogue entry and is also quite straightforward:

public UICatalogue clickOn(int i) {
	List catalogueEntries = this.allCatalogueEntries();
	int count = catalogueEntries.size();
	if (i>=count) { 
		throw new IndexOutOfBoundsException("Could not click on catalogue entry "+i+" as there are only "+count);
	}
	catalogueEntries.get(i).click();
	return UICatalogue.openCatalogue();
}

UICatalogue is also a semantic web UI class that includes the click() method that loads a catalogue, which is returned (in this case also without any parameters in the current implementation).

It should be noted that there are other valid ways to design this. The clickOn method only uses public methods so it could rightly be perceived as ‘client’ code, but as clicking on a catalogue is done very often, this is offered as a way to avoid repetition. It’s important to spend some time thinking about this design, giving consideration to style, operation frequency, potential for repetition, amount of convenience methods and so forth.

Also a bit of a personal code style choice, I am staying away from get/set semantics to better distinguish UI test logic from typical application code (which commonly employs get/set prefixes). A backend code getter could potentially perform a complex operation while the UI code just selects a CSS class and reads a value displayed on the page, hence the shorter method name. Following this logic, user action methods like clickOnXXX will be the ones performing complex operations like navigating to a different page and so forth, so they have a more explicit verb prefix. Of course getter/setters can be used if that suits more your style.

Using the abstraction UI classes

Usage is also straightforward, like this method:

@Test
public void catalogueListTest() throws Exception {
	open(appBaseURL);
	List catalogueEntries = UICatalogues.openCatalogues()
													  		.shouldAppear()
													  		.getAllCatalogueEntries();
	assertEquals(EXPECTED_CATALOGUES_COUNT, catalogueEntries.size());
	assertEquals("Wrong catalogue content", "Catalogue 1", catalogueEntries.get(0).name());
	assertEquals("Wrong catalogue content", "Catalogue 2", catalogueEntries.get(1).name());
	assertEquals("Wrong catalogue content", "Catalogue 1 yaml", catalogueEntries.get(2).name());
	UIProblem.shouldNotBeVisible();
}

This code is easy to read, and shows exactly what kind of behaviour the is being tested. In this case, it’s testing that once the application is loaded, the catalogues should appear, there should be EXPECTED_CATALOGUES_COUNT entries, and also the particular catalogue order and names. Finally, no error should be shown. Also note the readability of catalogueEntries.get(0).name() vs a more verbose catalogueEntries.get(0).getName().

More complex behaviour is also modelled quite well:

UICellEditor stuffEditor = stuff.edit().shouldAppear();
assertNotNull(stuffEditor);
Optional value = stuffEditor.getValue();
assertTrue(value.isPresent());
assertEquals("Stuff content", value.get());
assertFalse("Should not be able to create a value for this cell",  stuffEditor.isCreateValueVisible());
assertTrue("Should be able to remove value for this cell",  stuffEditor.isRemoveValueVisible());

stuffEditor.clickRemoveValue();
value = stuffEditor.getValue();
assertFalse(value.isPresent());

In this case the code is testing a form that contains a value and can be removed entirely (as opposed to cleared) by clicking on a specific widget or button. Once the widget is activated, the form value disappears and is not there anymore.

More examples like these can be found in the Morfeu project tests.

Conclusions

It is definitely useful to abstract web UI testing using techniques like ‘Logical entities to UI object mapping’ described in this post or the Page Object mapping pattern. If the right abstraction level is applied correctly, test are more meaningful, their intent is more obvious and the actual web app implementation can be changed more easily. Techniques applied with tools like Selenide make writing the semantic UI code even easier and combined with Java 8 stream support, testing code ends up being super-fun to write.

Annex

For more examples please take a look at examples from the Selenide documentation on the direct UI and test mix approach:

  @Test
  public void search_selenide_in_google() {
    open("https://google.com/ncr");
    $(By.name("q")).val("selenide").pressEnter();
    $$("#ires .g").shouldHave(sizeGreaterThan(1));
    $("#ires .g").shouldBe(visible).shouldHave(
        text("Selenide: concise UI tests in Java"),
        text("selenide.org"));
  }

(Note that the example is intended for simplicity and separation of concerns is not the aim of the code). The code is pretty readable and concise, but could be improved by applying the Page Object test design pattern or the one described in this blog post. Alternatively, for smaller tests having the classes in constants or in a test configuration could also be practical.

The Selenide project also has some examples of the more semantic approach using Page Objects here.

Building an FTP server Maven plugin from scratch

In this post we design and code a new Maven plugin that fires up a FTP server to aid in integration testing of FTP-bound processes, thus demonstrating the flexibility and power of Maven plugins.
Continue reading “Building an FTP server Maven plugin from scratch”

Templating the OSGi way with Freemarker

After some well-deserved rest the OSGi components on the server series is back with a vengeance and a somewhat independent post. For some background please check the other posts in the series.

A common feature of Web applications is the need to output structured content, be it XML, XHTML markup, JSON or many others. A number of technologies is used to do that but few seem to be that dynamic, usually only reloading templates when they change or loading them from URLs. Surely we can leverage OSGi to make things more interesting…

Therefore we should be following the OSGi dynamic philosophy as much as possible, exploiting the features made available by the framework such as dynamic discovery, services, bundle headers and the lot.

In the case of our cache app we are violating quite a few basic design principles by having the format embedded in the java code. So we should as well use a template separate from the code and if possible reuse some existing well-known templating engine from somewhere.

Given these basic design principles let’s get started.

Firstly, we need a robust and flexible templating engine. We select the mature Freemarker engine which is reasonably speedy and has the power and flexibility we want. Make sure you check out the license, of course.

We could stop at putting the library JAR in a bundle and package it so it can be used by any other bundle and that is what we do to be able to use it in OSGi. That however doesn’t exploit many of the nicer OSGi capabilities so we will create another bundle called ‘com.calidos.dani.osgi.freemarker-loader’.

What we want is to allow bundles to carry templates inside them and have our freemarker-loader templating bundle discover them automagically. This is the same technique that the Spring dynamic modules use and you can find more info here. The mechanism is described in this diagram:

OSGi freemarker templating diagram

That is easy enough with a BundleTracker and a BundleTrackerCustomizer implementation. The BundleTracker class tracks bundles being added to the environment like this:

tracker = new BundleTracker(context, Bundle.RESOLVED, templateTracker);
tracker.open();

With this snippet the tracker instance will look for bundles in the RESOLVED state (which lets us track fragments). The ‘templateTracker’ object is an instance of BundleTrackerCustomizer and will receive callbacks whenever bundles are added to the environment.

For instance, when a bundle is added we check for a special header in the bundle which tells us what is the relative path of available templates in the bundle being resolved:

public Object addingBundle(Bundle bundle, BundleEvent event) {
		
// we look for the header and act accordingly		
String templatesLocation = (String) bundle.getHeaders().get(TEMPLATE_HEADER);
if (templatesLocation!=null) {
			 
	Enumeration bundleTemplates = bundle.findEntries(templatesLocation, "*.ftl", true);
	HashSet templatesFromAddedBundle = new HashSet();
	while (bundleTemplates.hasMoreElements()) {
			
		URL templateURL = bundleTemplates.nextElement();
		addTemplate(bundle, templateURL,templatesLocation);
		templatesFromAddedBundle.add(templateURL);
				
	}
	
	templatesOfEachBundle.put(bundle.getBundleId(), templatesFromAddedBundle);
			
}
return null;
		
}	// addingBundle

An interesting method being used here is ‘findEntries’ which loads all the entries in the provided templates folder and lets us add them to our holding structure. We also take care to implement the methods to remove the templates and update them accordingly whenever bundles are updated or unloaded from the environment.

Having TEMPLATE_HEADER with a value of ‘Freemarker-Templates’ means that bundles having a header such as Freemarker-Templates: /templates will have any templates within that folder (please note that the ‘/templates’ bit is not added to template URLs!).

The next thing we need to do is make the loaded templates available to the environment. To do that we make a freemarker Configuration object accessible as an OSGi service object. That Configuration instance is the main object Freemarker to load and use templates and has an interesting mechanism to override its template loading mechanism we use to make available our OSGi environment templates.

freemarkerConfig.setTemplateLoader( new URLTemplateLoader() {
			
@Override
protected URL getURL(String url) {
Stack templateStack = templates.get(url);
if (templateStack!=null) {
	TemplateEntry templateStackTop = templateStack.peek();
	if (templateStackTop!=null) {
		return templateStackTop.getTemplateURL();
	}
	return null;
}
return null;
}

});

The service Configuration object is set with a template loader inner class that uses our template holding objects to retrieve templates stored in our OSGi context. Cool.

This also allows us to effectively disable the template refreshing cycles that Freemarker does by default (allegedly making it slightly more efficient). Now we only need to refresh a bundle containing the templates to get the new version. This can be modified by using the appropriate methods on the Configuration service of course. (There is another method explained later).

An interesting feature we can add to exploit the dynamic nature of OSGi is to make templates available in a stack. This means different bundles can dynamically overwrite templates by the same name. Moreover, once a template is removed the previous version becomes available. This can be used to make temporary changes to templates to add or remove diagnostics information, test new templates temporarily, etc.

We do that using a Stack of TemplateEntry objects, TemplateEntry being a helper class to store template entries.

This is all very nice but we have a problem when having multiple versions of the same bundle that hold multiple copies of the same template, this means they will stack and we have no way to access a particular version of a template. To solve this problem we store each template in three different stacks by three different URLs:

  • ‘path/file.ftl’
  • ‘bundle://bundlename/path/file.ftl’
  • ‘bundle://bundlename:version/path/file.ftl’

In this manner we can use the more generic URL in most cases but still can access specific versions when needed. It is important to think about the dynamic nature of OSGi as well as the possibility of several versions of the same bundle coexisting peacefully in the same environment.

From the perspective of any bundle using the service in the simplest case it only needs to look for a service named ‘freemarker.template.Configuration’. For convenience, the property ‘dynamicConfiguration’ is set to ‘true’ to coexist peacefully with other Configuration services (maybe coming from an official Freemarker bundle). For instance, if we know for sure our dynamic Configuration service is the only one present we can do:

context.getServiceReference(Configuration.class.getName());

That will give us the highest ranking Configuration service. If there are several such services we can use a call like this to get the one that has the dynamic OSGi loader:

context.getServiceReferences(Configuration.class.getName(), "dynamicConfiguration=true");

There is one last feature which lets bundle users feed an already configured template configuration object to the bundle by publishing a Configuration service with property ‘preparedConfiguration’ set to ‘true’. This will get picked up by the bundle and its template loader put in sequence with the dynamic OSGi template loader. This means that any original Configuration settings are maintained (For further information on service filtering based on properties please check the BundleContext javadoc.).

Best thing to do is to go and try it by downloading the bundles here. Source is also available.

Components on the server (6): adding Integration Testing

In this installment of the server-side OSGi series, we add integration testing capabilities to our project. Integration testing goes beyond plain unit testing and checks the interactions between real components. This is in contrast with unit testing, which generally uses mockups to represent components outside the one being tested. Please take a look at previous installments, as usual.

In the case of integration testing, it is manly used in a pre-production environment, with a valid build that has all unit tests passed. It can even be used in production to just after a deployment is made, taking care not to have destructive checks or massive load tests in the integration test code. YMMV.

To achieve integration testing we need to check the various OSGi components deployed interact in the way that is expected of them. Therefore we need to test the components in a group and not in isolation. To do that in the OSGi world means we need to have access to the OSGi context from within the tests to access services, call them and check their responses, etc.

To allow for this kind of integration testing within the OSGi environment, we make a slight modification to the excellent test.extender we have already patched in the previous installment.

Basically, the basic test.extender seeks out any JUnit test classes within the fragment bundle, creates an instance using an empty constructor and then fires up the tests. This is activated either by default when the fragment is loaded or by using ‘test ‘ in the console. For further information please see the previous post about this subject.

For our integration testing, we add an extra command to test.extender:

public Object _integrationTest(CommandInterpreter intp) {
        String nextArgument = intp.nextArgument();
    	testExtender.integrationTest(Long.parseLong(nextArgument));
    	return null;
}

And we refactor the TestExtender to add the integrationTest method which reuses some of the code to instantiate test cases using a constructor that accepts the OSGi context as a parameter.

Constructor[] constructors = clazz.getConstructors();
boolean foundConstructor = false;
for (int i = 0; i < constructors.length && !foundConstructor; i++) {
	Constructor constructor = constructors[i];
	Class[] types = constructor.getParameterTypes();
	if (types.length==1 && types[0].isInstance(context)) {
		foundConstructor = true;
		EClassUtils.testClass(inspectClass, constructor.newInstance(context));
	}
} // for

The OSGi context is passed onto the constructor and then the test class is run. It is obviously up to the test class to use the context appropriately for its integration testing.

In our cache project setup, we can do some useful integration testing on the cache.controller component, basically checking if the interaction with the provider components is behaving as we expect it. The integration testing is also added to a fragment that can be deployed optionally, of course.

We start by creating the fragment and adding a testing class like this:

Adding test class

Next, we add the constructor that accepts an OSGi context, which is very simple:

public CacheIntegrationTest(BundleContext ctx) {
	super();
	this.context = ctx;
}

In the setup and teardown methods we get and unget the cache service to perform the testing:


public void setUp() throws Exception {
	serviceReference = context.getServiceReference(Cache.class.getName());
	controller = (CacheControllerCore) context.getService(serviceReference);

}

public void tearDown() throws Exception {		
	context.ungetService(serviceReference);
	controller = null;		
}

In this case we get the controller cache service and store it in an instance used to perform the tests. This is quite simple and fulfills our intended purpose but we still have the flexibility to make more complex integration testing if needed.

Next we create as many test cases as needed:

public void testGet() {
	try {
		controller.init();
		double v = Math.random();
		String k = "/k"+v;
		controller.set(k, v);
		assertEquals(v, controller.get(k));
	} catch (CacheProviderException e) {
		e.printStackTrace();
		fail(e.getMessage());
	}

}

It should be noted that while the code looks like regular testing code, it is actually using real services from the OSGi environment as opposed to mockups. This means we are testing the real integration between components as well as the individual controller component code. The disadvantage here is that if there is an error in the controller we might mistake the problem with an issue with the services used. In conclusion, having integration code doesn’t negate the need to have unit tests.

Once we load the fragment onto the environment, first we need to obtain the bundle id of the integration fragment and then launch the integration testing in this manner:


osgi> integrate 125
Bundle : [125] : com.calidos.dani.osgi.cache.controller.integration
_
CLASS : [com.calidos.dani.osgi.cache.controller.CacheIntegrationTest]
___________________________________________________________________________
Method : [ testInit ] PASS
Method : [ testInitInt ] PASS
Method : [ testSize ] PASS
14:21:43,077 WARN CacheControllerCore Couldn't clear some of the provider caches as operation is unsupported
14:21:43,077 WARN CacheControllerCore Couldn't clear some of the provider caches as operation is unsupported
Method : [ testClear ] PASS
Method : [ testSet ] PASS
Method : [ testGet ] PASS
Method : [ testGetStatus ] PASS
___________________________________________________________________________

The results tell us that all operations are OK but we need to bear in mind that the clear operation is not supported in some backend caches. If this is what is expected by the operator then all is fine.

We take advantage of the new integration testing functionality to make some extensive changes to logging and exception handling of the controller code. By running the integration tests we make sure all seems to work fine (even though we still need some proper unit testing of the controller). Modifications are made quite quickly thanks to the integration tests.

To recap, we’ve added integration testing support to the existing ‘test.extender’ bundle and created integration testing code for the cache controller component. This has allowed us to make code changes quickly with less risk of mistakes.

Here you can find a patch for the test extender project as well as the patched testing bundle already compiled. Enjoy!

Components on the server (5): better Unit Testing

In this installment of the OSGi series, we add more complete Unit Testing support in the project. We also establish that some behaviour of the Servlet Bridge may not be what we want and then provide a way to customize it.

Continue reading “Components on the server (5): better Unit Testing”

Components on the server (4): adding Tomcat support

In this post, we examine what is needed to deploy OSGi in a regular Servlet Container using the Equinox Servlet Bridge. We also use the Servlet Bridge to deploy our OSGi cache using Tomcat and wrap it up all together in a standalone WAR archive.

Please read the previous installments to get up to speed and see the source used in this post.

Firstly, we need to setup some kind of project to hold all that is going in the WAR archive. This can be done using the WTP (Web Tools Project) or just as a regular project.

In this case we do it using a plain-vanilla resource project but feel free to use WTP as the basics are the same.

So firstly we create a resources project to hold the stuff, named ‘com.calidos.dani.osgi.cache.package’ for instance.
Secondly, we create the folder structure we need to hold all the files that need to reside in the final WAR webapp, so we create a WEB-INF folder to hold all the classes, libraries metadata and configuration.

We also know we need a web.xml file to define all the servlets this webapp declares. This is where the Equinox Servlet Bridge kicks in. It provides a Servlet that loads up the environment as well as forwards any HTTP requests onto our OSGi-managed Servlet instances.

We check the Equinox in a servlet container article and learn what the bridge Servlet is called, what parameters it can take and any other web.xml details we need.

For instance, we declare that the webapp service class is our BridgeServlet class:

<servlet-class>org.eclipse.equinox.servletbridge.BridgeServlet</servlet-class>
<init-param>
<param-name>commandline</param-name>
<param-value>-console</param-value>
</init-param>
<init-param>
<param-name>enableFrameworkControls</param-name>
<param-value>true</param-value>
</init-param>

We also add two parameters (more information and additional options can be found on the FrameworkLauncher servlet bridge class and the Equinox documentation).

Next, we need this special servlet itself. I prefer to check out the servlet from CVS and compile it myself but you can also find it here (from the the Latest Release pick up the org.eclipse.equinox.servletbridge jar). In the case of using the source, the CVS connection data is the following:


Method: pserver
User: anonymous
Host: dev.eclipse.org
Repository path: /cvsroot/rt
CVS Module: org.eclipse.equinox/server-side/bundles/org.eclipse.equinox.servletbridge

Once we import the project into Eclipse it looks like this:

Servlet bridge project

As we can see, there are just three classes which compose the servlet, a special classloader and finally a class that launches OSGi.

If we import the project into our workspace, we have compiled classes in the bin/ folder of the project but we really want them neatly packaged as a jar file. Therefore, we right-click on the project to export as a JAR java package:

Servlet bridge JAR export

We take this JAR archive and save it in our package project WEB-INF/lib folder. We also check the composition of the file to make sure it includes all the classes we need.

This takes care of the plain webapp side of things so we move onto actually loading OSGi and what configuration it needs.

As mandated by Equinox we add a launch.ini file to clear and possibly override System properties (please see the source for details on that).

We then create an eclipse folder, which the servlet bridge expects to find the OSGi platform jar and any bundles (including ours).

We select all our project bundles, right-click and select the ‘Export…:Deployable plug-ins and fragments’ option.

This leaves us with a list of bundles that compose our custom code:

com.calidos.dani.osgi.cache.controller_1.0.0.beta.jar
com.calidos.dani.osgi.cache.frontend.http_1.0.0.beta.jar
com.calidos.dani.osgi.cache.log4jconfig_1.0.0.beta.jar
com.calidos.dani.osgi.cache.provider.memcached_1.0.0.beta.jar
com.calidos.dani.osgi.cache.provider.memory_0.0.1.dev.jar

Good, as we know, there are some standard and some special bundle dependencies we need as well. Let’s go through them by groups. We get a bunch of basic OSGi bundles we should include in most projects:

org.eclipse.equinox.registry_3.4.100.v20090520-1800.jar
org.eclipse.osgi.services_3.2.0.v20090520-1800.jar
org.eclipse.osgi.util_3.2.0.v20090520-1800.jar
org.eclipse.osgi_3.5.1.R35x_v20090827.jar

There are several options to obtain these bundles:

For expediency, we can go to our Eclipse installation folder and look for them in the ‘plugins’ subfolder. We then copy them into our package ‘plugins’ folder and there you go.

The second option is to get them from the Equinox distribution.

Another option is to the prepackaged servlet bridge feature which can be picked from CVS as well:

Method: pserver
User: anonymous
Host: dev.eclipse.org
Repository path: /cvsroot/rt
CVS Module: org.eclipse.equinox/server-side/features/org.eclipse.equinox.servletbridge.feature

Once that is loaded onto our workspace, we open the feature.xml and select the ‘Export Wizard’ form the ‘Exporting’ section. This lets us export these minimum bundles and generate both the deployment ‘feature.xml’ file as well as the bundle listing (even though the wizard actually takes them from our Eclipse platform).

We also have some more dependencies related to doing HTTP and the logging platform:

org.apache.log4j_1.2.13.v200903072027.jar
org.eclipse.equinox.common_3.5.1.R35x_v20090807-1100.jar
org.eclipse.equinox.http.registry_1.0.200.v20090520-1800.jar
org.eclipse.equinox.http.servlet_1.0.200.v20090520-1800.jar
org.eclipse.equinox.registry_3.4.100.v20090520-1800.jar

These can also be obtained from the Eclipse plugins folder or Equinox distribution folder. One should note that modifying the feature.xml taken from the Eclipse CVS repository, adding any required plugins there and then running the Export Wizard.

Finally, we have two dependencies we need to pay special attention to:

org.eclipse.equinox.http.servletbridge_1.0.200.200911180023.jar
javax.servlet_2.4.0.200911240836.jar

In the first case, it’s a special minimal bundle that hooks the plain webapp servlet bridge with the OSGi ‘org.eclipse.equinox.http.servlet’ standard component which publishes a servlet service onto the OSGi platform.

This are the details to get it from CVS:

Method: pserver
User: anonymous
Host: dev.eclipse.org
Repository path: /cvsroot/rt
CVS Module: org.eclipse.equinox/server-side/bundles/org.eclipse.equinox.http.servletbridge

We export it as a deployable plug-in and add it to the mix. We can also download it from the Equinox distribution site as well.

In the second case, this bundle contains the basic servlet classes and interfaces and I have found problems of class signatures when using 2.5 along with the servlet bridge so unless all is compiled against 2.5 the safest bet is to go with 2.4.

(Check http://www.eclipse.org/equinox/documents/quickstart.php for more info).

Next, Equinox requires an XML file to define some metadata about the loaded bundles and make our configuration easier. The file sits in a folder called ‘features’ plus a subfolder with a reverse DNS name and is called feature.xml.

The structure is quite simple (though it can be created using the Eclipse Feature Export Wizard) and mainly holds a list of plugins, their version data and some more fields:


We complete the list with all our bundles, fully knowing that we can extend our environment on the fly once it’s loaded if we need it.

Next, we create the config.ini file which really tells equinox which of the bundles stated in the feature file to start, at which start level and lets us add even more bundles (though we need to specify where the actual file is located and the full filename in that case). It also lets us configure the environment pretty thoroughly.

The resulting config.ini file looks like this:


#Eclipse Runtime Configuration File
osgi.bundles=org.eclipse.equinox.common@1:start, org.apache.log4j@start, org.eclipse.osgi.util@start, org.eclipse.osgi.services@start, org.eclipse.equinox.http.servlet@start, \
org.eclipse.equinox.servletbridge.extensionbundle, \
org.eclipse.equinox.http.servletbridge@start, \
javax.servlet@start, org.eclipse.equinox.registry@start, org.eclipse.equinox.http.registry@start, org.eclipse.equinox.servletbridge.extensionbundle \
org.eclipse.equinox.http.servlet@start, org.eclipse.equinox.common@start, com.calidos.dani.osgi.cache.log4jconfig, \
com.calidos.dani.osgi.cache.provider.memcached@3:start, com.calidos.dani.osgi.cache.provider.memory, com.calidos.dani.osgi.cache.controller@4:start, \
com.calidos.dani.osgi.cache.frontend.http@5:start

osgi.bundles.defaultStartLevel=4

We decide not to start the in memory bundle as it only works reliably in a single instance deployment scenario. Otherwise, in a multiple server setup the in-memory cache data would become inconsistent.

Also, be sure to check the Equinox quickstart guide and documentation for more information.

Once all is is done, the project looks like this:

Finished package project

Time to right-click on the project and select ‘Export:Archive file’ to save it in zip format and rename it to .war. Ready to deploy! Remember to access it using the URL //cache/. In web.xml the ‘-console 6666’ parameter means that doing a telnet on port 6666 of the machine initiates a session within the OSGi console. WARNING: there is absolutely NO SECURITY so disable, firewall or ACL that port NOW.

As usual, here you can find the source and the completed WAR though they are one and the same.

Components on the server (3): adding a HTTP frontend

Welcome to the third instalment of our OSGi ABC tutorial. Please make sure you check both the 1st installment and the 2nd.

In this post, we will add another cache provider implementation to the mix as well as provide an HTTP front-end so the whole application can be tested.

First of all, let’s present a conceptual diagram of all the bundles and fragments involved so far.

Components diagram

Read on for more…

Continue reading “Components on the server (3): adding a HTTP frontend”

Components on the server (2): creating the first bundles

Hopefully you enjoyed the OSGi journey in its first installment.

Though simple and easy to understand, the first example does nothing out of the ordinary. It is far more interesting to start exploiting some of the basic features OSGi gets us “for free”.

For instance, we could begin by moving our first basic implementation into a separate “model” bundle and enhancing the interface so it can throw exceptions. For instance, an exception can be thrown when no implementations are available or cannot be contacted/operated.

Read the rest of the post for the implementation details…

Continue reading “Components on the server (2): creating the first bundles”

Components on the server: an OSGi mini-project

The best way to start with OSGi on the server side is with a mini-project to help clarify concepts and workout the mechanics.

I have chosen a simple idea: build a volatile cache that stores objects. The interface with it is simple HTTP calls to perform usual operations: get, set, clear and get info. Additionally, Unit Testing will of course be used (but not Acceptance Testing in this case).

We start with building a component that serves as the controller for the system. It encapsulates any implementation details and answers to the system requests.

Read on for a short tutorial on how to setup the a first bundle, add an interface and some logging and fire up the appropriate OSGi environment.

Continue reading “Components on the server: an OSGi mini-project”

Experiencing server-side OSGi with Equinox

There is currently a lot of buzz about the OSGi java component technology, also server-side. I have been playing -and working- with this interesting technology recently, mainly the Equinox server-side bundles that allow deployment of an OSGi environment in a java servlet environment.

A fundamental problem that this technology solves is being able to load an OSGi environment on a servlet server. That is a relatively costly operation that needs to be done only once and be persistent on the server in between client requests. After that point, state is maintained and bundles can be deployed and managed as needed. Creating, loading and destroying the environment for each request would just be unacceptable. One interesting functionality is then being able to serve client requests, specially HTTP requests, uploads, provide REST interfaces, etc.

Eclipse provides a number of projects to load the environment, register servlets, handle http requests, etc. Of particular interest is the ‘org.eclipse.equinox.servletbridge’ project, which initally starts the framework, loads the appropriate framework bundles, etc.

Looking at the code, you can start by checking out the main class, which has a number of interesting responsabilities:

/**
 * The BridgeServlet provides a means to bridge the servlet and
OSGi runtimes. This class has 3 main responsibilities:
 * 1) Control the lifecycle of the associated FrameworkLauncher
in line with its own lifecycle
 * 2) Provide a servlet "hook" that allows all servlet requests
to be delegated to the registered servlet
 * 3) Provide means to manually control the framework lifecycle
 */

I personally like classes that have a small set of responsabilities, but one can argue that the three are so related that they are expressions of the same one. The first responsability described is actually managing the lifecycle of the OSGi framework FrameworkLauncher class.

Basically, the Servlet Container will load the bridge web application, create an instance of ServletBridge and call its init method (defined by the HttpServlet interface which it implements). In that method the OSGi environment will be created and loaded by the following code:

framework.init(getServletConfig());
framework.deploy();
framework.start();
frameworkStarted = true;

The attribute ‘framework’ is an instance of the ‘FrameworkLauncher’ class, which encapsulates the OSGi environment management logic. The ‘init’ method pulls information from the servlet configuration (such as the name of the ‘WEB-INF’ folder) and calls an overloaded empty init method that could be exploited by specialised subclasses (more on this later).

After that, the framework is “deployed”. The method in question declares the following contract:

/**
* deploy is used to move the OSGi framework libraries into a location
   suitable for execution.
* The default behavior is to copy the contents of the webapp's
   WEB-INF/eclipse directory to the webapp's temp directory.
*/

And that’s right, the code is quite straightforward and goes along the lines of:

File servletTemp = (File) context.getAttribute("javax.servlet.context.tempdir");
platformDirectory = new File(servletTemp, "eclipse");
if (!platformDirectory.exists()) {
	platformDirectory.mkdirs();
}
File plugins = new File(platformDirectory, "plugins");
copyResource(resourceBase + "plugins/", plugins);

Which copies the bundles you want to deploy onto the OSGi environment onto the Servlet Container temporary folder (the ‘work’ folder in the case of Apache Tomcat).

After that the framework is started, reading .ini configuration options, command line switches, bundle list, run levels and so forth. We have not been able to get bundles started automatically unless we specify the bundle filename, complete with ‘.jar’ extension and all. One interesting option is the ability to fire up a standard OSGi console that uses the STDIN/OUT of the Servlet Container process.

Using the very same servlet bridge, there are a number of URLs that can be hit do control the framework, start, stop, undeploy, etc.

After that, one can register servlets as extension points or listen for services named “org.osgi.service.http.HttpService” and adding servlets with code going along these lines:

public Object addingService(ServiceReference reference) {
	HttpService httpService = (HttpService) context.getService(reference);
	httpService.registerServlet(url, servlet, null, null);
}

All is well and good. However, in working with the bridge code we have found a glitch in the startup code. Whenever the bridge app is started anew, it does the framework deployment and startup as expected but if there are any bundles that have any changes they will not be read by the environment unless the redeploy url is hit. This is fine in development environments but not on production, where you want to keep things  as smooth as possible.

Behold the power of OSS, we took a look at the code and submitted a bug entry complete with a working patch that fixes the problem. It is not pretty but it works (it has some code duplication).

Server-side OSGi is proving to be the place to be, or at least it is quite promising. Give it a try.