Easier web app testing by mapping entities to UI objects

Automated, browser-based testing is a key element of web application development, benefiting both simple and complex applications. Writing effective tests for browser-based apps can be a complex, tedious and often repetitive task. In this post, I will be discussing a general approach to write meaningful, loosely-coupled UI tests for web applications by going beyond the Page Object Design Pattern into a more fine-grained approach I call ‘Logical entities to UI object mapping‘. I will show code in written Java 8 leveraging the Selenium and Selenide frameworks to show examples of the method described.

Layers of web app testing responsibility
Layers of web app testing responsibility

In web development, a common component used to perform browser-based testing is Selenium, which is a suite of frameworks and tools to automate browsers. All code in the Selenium projects is licensed under Apache 2.0. Selenium WebDriver is the most important component, which exposes an API to control several web browser and browser engines, and includes several language bindings: Java, C#, Python, Ruby, Perl, PHP and Javascript.

Page Object test design pattern

Selenium and the WebDriver API are very flexible and relatively easy to use. Make sure to check out the Test Design Consideration page of on the Selenium documentation for tips on how to structure your code, specially the Page Object Design Pattern section. There is a great post by Martin Fowler on the subject, plus lots of resources and blog posts on the web.

The following example is from the Selenium documentation, showing how coding a test using the straightforward WebDriver API can result in complex code that mixes many concerns:

/***
 * Tests login feature
 */
public class Login {

        public void testLogin() {
                selenium.type("inputBox", "testUser");
                selenium.type("password", "my supersecret password");
                selenium.click("sign-in");
                selenium.waitForPageToLoad("PageWaitPeriod");
                Assert.assertTrue(selenium.isElementPresent("compose button"),
                                "Login was unsuccessful");
        }
}

This kind of code, specially when adding lots of selectors and adding more test logic, can quickly transform into hard to maintain, spaguetti code.

When using the Page Object pattern, the original test code can be refactored into a more OOP class that has more semantics:

/**
 * Page Object encapsulates the Sign-in page.
 */
public class SignInPage {

        private Selenium selenium;

        public SignInPage(Selenium selenium) {
                this.selenium = selenium;
                if(!selenium.getTitle().equals("Sign in page")) {
                        throw new IllegalStateException("This is not sign in page, current page is: "
                                        +selenium.getLocation());
                }
        }

        /**
         * Login as valid user
         *
         * @param userName
         * @param password
         * @return HomePage object
         */
        public HomePage loginValidUser(String userName, String password) {
                selenium.type("usernamefield", userName);
                selenium.type("passwordfield", password);
                selenium.click("sign-in");
                selenium.waitForPageToLoad("waitPeriod");

                return new HomePage(selenium);
        }
}

There are also fair warnings on the tradeoffs involved in applying the pattern, as described in this blog post.

In this case the warning is focused on the risk of grouping unrelated semantic and intent in aggregation ‘page’ classes.

In my case, I also tend to stay away from ‘page’ classes and write UI abstractions around the application’s logical entities. Logical entities are mapped to UI objects and then accessed naturally through their relationships so user actions and navigation are kept intentional. To also keep code within the domain of UI testing, most UI class methods are explicitly called after common user browser actions like clicking, moving the mouse or entering text into forms. In practice I have found that this combination is a ‘sweet spot’ combination of useful abstraction, loose coupling, testability and practicality.

The following diagram illustrates the concept for a typical content management system three level master-detail:

Grouping logical entities in semantic web UI testing, showcasing method names
Grouping logical entities in semantic web UI testing, showcasing method names

I will be showing next some specific examples of applying the ‘Logical entities to UI object mapping’ pattern in Java 8, using Selenide on top of Selenium to automate the browser.

Introducing Selenide

Surprisingly less well-known than it should be, Selenide is a great Selenium WebDriver abstraction layer written in Java that makes it easier to manipulate browsers and write tests than just by using Selenium. Selenide simplifies writing tests based on Selenium with a surprisingly intuitive interface. To demonstrate the simplicity, here’s an example from the getting started:

@Test
public void userCanLoginByUsername() {
  open("/login");
  $(By.name("user.name")).setValue("johny");
  $("#submit").click();
  $(".loading_progress").should(disappear); // Waits until element disappears
  $("#username").shouldHave(text("Hello, Johny!")); // Waits until element gets text
}

The interface offered is easy to use and you usually don’t even need to look at the documentation. Waiting for elements to appear is implicit in many parts of the code and Selenide generally works as you would expect. I encourage you to try it, write a few tests and you will be surprised at the simplicity.

Regardless of the added convenience and abstraction of the Selenide library, we can still end with spaguetti code if we are not careful. For instance, if we use CSS selectors to access one UI element with $(org.openqa.selenium.By seleniumSelector) and want to change related classes in the client-side, we need to go everywhere in our testing code and change the CSS selector references.

Separating UI handling from tests

In a OSS project I’ve been recently working on (Morfeu), I am doing extensive testing of the web UI and I’m using the Logical entities to UI object mapping approach all over the place. For instance, in the project there is a list of catalogues (like a master-detail listing) as a logical application entity. Each catalogue has a list of different catalogue entries, which in turn contain documents (not shown in the example), completing a simple three layer of logical app entities. The code to operate and access the master catalogue list is as follows:

public class UICatalogues {

private static final String CATALOGUE_LIST = "#catalogue-list";
private static final String CATALOGUE_LIST_ENTRY = ".catalogue-list-entry";

public static UICatalogues openCatalogues() {
	return new UICatalogues();
}

public UICatalogues shouldAppear() {
	$(CATALOGUE_LIST).should(appear);
	return this;
}

public UICatalogues shouldBeVisible() {
	$(CATALOGUE_LIST).shouldBe(visible);
	return this;
}

public List allCatalogueEntries() {
	return $$(CATALOGUE_LIST_ENTRY).stream().map(e -> new UICatalogueEntry(e)).collect(Collectors.toList());
}

public UICatalogue clickOn(int i) {
	List catalogueEntries = this.allCatalogueEntries();
	int count = catalogueEntries.size();
	if (i>=count) { 
		throw new IndexOutOfBoundsException("Could not click on catalogue entry "+i+" as there are only "+count);
	}
	catalogueEntries.get(i).click();
	return UICatalogue.openCatalogue();
}

}

To start with, public static UICatalogues openCatalogues() is used to open the catalogues list, it’s a static method as one of the constraints is that there is only one catalogue list and that it appears as soon as the user loads the application. A static method is a convenient way to access the ‘catalogues’ object instance as opposed to a public constructor, which maps to the application semantics. If there was a need to change the behaviour into requiring user action to load the catalogues (such as a mouse click) the implementation could be changed without changing the caller code. If the change were to be quite radical and significantly change the logical entity relationships, like allowing multiple catalogues, the API and caller code could (and should) be refactored to reflect the big change in application behaviour.

The methods UICatalogues shouldAppear() and shouldBeVisible() are pretty self explanatory, and can be used to ensure the catalogue list loads and displays properly.

The methods are quite simple themselves and the usage of Selenide is pretty much self-explanatory.

The next method List allCatalogueEntries() is used to obtain the list of catalogues, and takes advantage of Java 8 streams, mapping the list of found elements into new UICatalogueEntry instances:

public List allCatalogueEntries() {
	return $$(CATALOGUE_LIST_ENTRY).stream().map(e -> new UICatalogueEntry(e)).collect(Collectors.toList());
}

A stream of low-level catalogue entry elements found by Selenide is mapped to new catalogue instances and then collected into a using Java 8 list collector.

UICatalogueEntry‘s constructor accepts a SelenideElement to provide each catalogue entry instance with its local context. SelenideElement instances are a wrapper of Selenium elements and are the basic building blocks of testing using Selenide.

The last method UICatalogue clickOn(int i) is a convenience method to click on a specific catalogue entry and is also quite straightforward:

public UICatalogue clickOn(int i) {
	List catalogueEntries = this.allCatalogueEntries();
	int count = catalogueEntries.size();
	if (i>=count) { 
		throw new IndexOutOfBoundsException("Could not click on catalogue entry "+i+" as there are only "+count);
	}
	catalogueEntries.get(i).click();
	return UICatalogue.openCatalogue();
}

UICatalogue is also a semantic web UI class that includes the click() method that loads a catalogue, which is returned (in this case also without any parameters in the current implementation).

It should be noted that there are other valid ways to design this. The clickOn method only uses public methods so it could rightly be perceived as ‘client’ code, but as clicking on a catalogue is done very often, this is offered as a way to avoid repetition. It’s important to spend some time thinking about this design, giving consideration to style, operation frequency, potential for repetition, amount of convenience methods and so forth.

Also a bit of a personal code style choice, I am staying away from get/set semantics to better distinguish UI test logic from typical application code (which commonly employs get/set prefixes). A backend code getter could potentially perform a complex operation while the UI code just selects a CSS class and reads a value displayed on the page, hence the shorter method name. Following this logic, user action methods like clickOnXXX will be the ones performing complex operations like navigating to a different page and so forth, so they have a more explicit verb prefix. Of course getter/setters can be used if that suits more your style.

Using the abstraction UI classes

Usage is also straightforward, like this method:

@Test
public void catalogueListTest() throws Exception {
	open(appBaseURL);
	List catalogueEntries = UICatalogues.openCatalogues()
													  		.shouldAppear()
													  		.getAllCatalogueEntries();
	assertEquals(EXPECTED_CATALOGUES_COUNT, catalogueEntries.size());
	assertEquals("Wrong catalogue content", "Catalogue 1", catalogueEntries.get(0).name());
	assertEquals("Wrong catalogue content", "Catalogue 2", catalogueEntries.get(1).name());
	assertEquals("Wrong catalogue content", "Catalogue 1 yaml", catalogueEntries.get(2).name());
	UIProblem.shouldNotBeVisible();
}

This code is easy to read, and shows exactly what kind of behaviour the is being tested. In this case, it’s testing that once the application is loaded, the catalogues should appear, there should be EXPECTED_CATALOGUES_COUNT entries, and also the particular catalogue order and names. Finally, no error should be shown. Also note the readability of catalogueEntries.get(0).name() vs a more verbose catalogueEntries.get(0).getName().

More complex behaviour is also modelled quite well:

UICellEditor stuffEditor = stuff.edit().shouldAppear();
assertNotNull(stuffEditor);
Optional value = stuffEditor.getValue();
assertTrue(value.isPresent());
assertEquals("Stuff content", value.get());
assertFalse("Should not be able to create a value for this cell",  stuffEditor.isCreateValueVisible());
assertTrue("Should be able to remove value for this cell",  stuffEditor.isRemoveValueVisible());

stuffEditor.clickRemoveValue();
value = stuffEditor.getValue();
assertFalse(value.isPresent());

In this case the code is testing a form that contains a value and can be removed entirely (as opposed to cleared) by clicking on a specific widget or button. Once the widget is activated, the form value disappears and is not there anymore.

More examples like these can be found in the Morfeu project tests.

Conclusions

It is definitely useful to abstract web UI testing using techniques like ‘Logical entities to UI object mapping’ described in this post or the Page Object mapping pattern. If the right abstraction level is applied correctly, test are more meaningful, their intent is more obvious and the actual web app implementation can be changed more easily. Techniques applied with tools like Selenide make writing the semantic UI code even easier and combined with Java 8 stream support, testing code ends up being super-fun to write.

Annex

For more examples please take a look at examples from the Selenide documentation on the direct UI and test mix approach:

  @Test
  public void search_selenide_in_google() {
    open("https://google.com/ncr");
    $(By.name("q")).val("selenide").pressEnter();
    $$("#ires .g").shouldHave(sizeGreaterThan(1));
    $("#ires .g").shouldBe(visible).shouldHave(
        text("Selenide: concise UI tests in Java"),
        text("selenide.org"));
  }

(Note that the example is intended for simplicity and separation of concerns is not the aim of the code). The code is pretty readable and concise, but could be improved by applying the Page Object test design pattern or the one described in this blog post. Alternatively, for smaller tests having the classes in constants or in a test configuration could also be practical.

The Selenide project also has some examples of the more semantic approach using Page Objects here.

provashell – testing shell scripts the easy way

In this post I will describe the provashell project, an Apache 2.0 licensed bash and shell Unit Testing library using annotations. The library is a self-contained script file with no dependencies that can be embedded or used easily in your project. Extra care has been taken to use POSIX shell features and the code has been tested in the popular shells bash, dash and zsh (latest versions the time of writing this article). I will add some detail on what drove me to do it, what it does, how it works and some examples.

Unit tests should be everywhere there is code. Tests materialise our expectations of the expected behaviour, prevent obscure bugs, generally induce more elegant designs and also serve as very effective up-to-date documentation. The oft-touted benefits are well worth it.

An area commonly overlooked in unit testing is when the code is “just a script”. This is unfortunately a common misconception, code is just code and should therefore be treated as such, with solid engineering principles and rigorous testing. Here’s a thought experiment: try to tell yourself what the differences between a ‘script’ and ‘proper code’ really are. Is it length? There might be a short piece of code that configures all your company’s firewall rules or backups and that is quite an important piece of logic, isn’t it? Is it criticality? Working non-critical code tends to end up included in critical systems and by extension becomes ‘mission-critical’ as well. Is it the language it is coded in? Most languages are pretty much logically equivalent (and Turing complete) so if something coded in Python is translated into Java to do the same thing its very essential nature has not changed at all. Is it necessity? We could go on. Code is just code, and it can and should be tested.

Unit testing for bash and shell scripts

Testing bash and shell scripts is unfortunately not that common. As discussed, shell scripts should still be tested thoroughly. There are plenty of shell testing libraries out there, usually bash-specific implementations, with different licenses. They are mature and well tested implementations. However, I was on the lookout for an Apache 2.0 licensed one that was simple, with no dependencies (such as the latest C++ compiler!) and I could not find one that suited my taste. One never knows anywhere near all there is to know about shell programming (trust me, you do not, specially when taking into account different implementations and so on) so I set about writing one myself that had the outlined characteristics which would also help me to learn more about shell coding.

Main features

The specific characteristics of provashell are as follows:

* Be unit-tested itself – Using plain shell test structures to do the assertion tests themselves. One of course can test the basic assert function and then leverage that tested function to check the other assertions, but I wanted to avoid false positives and keep concerns separate. Using provashell’s own test assert functions to test itself results in more elegant code but is potentially confusing when failures occur due to cascading effects. 

* Be as POSIX-compliant as possible – To that effect, the library has been tested in the latest (as of writing) versions of bash, zsh and dash, the latter being quite POSIX-strict. While bash-isms can be very practical when coding and in interactive shell sessions, cross-shell testing is a good code quality exercise which forces engineers to double check lots of assumptions and shortcuts, generally leading to better scripts. Even though I much prefer to use zsh for interactive sessions (specially when paired with toolchains such as the genial oh my zsh), once you have the mindset of shell implementations being real programming languages, it is fairly easy to mentally switch to bash or -even better-, POSIX shell ‘mode’ when writing persistent scripts. Such mental gymnastics will greatly help if found working on an old or unfamiliar system, with only ksh installed or something like that.

* Run no user-supplied code – This is an important security characteristic. The very first version of provashell used eval to run assertion checks in a flexible way, this resulted in elegant code but it also meant that test data could include shell code that could be run by the test framework. This is insecure and should be avoided if there are other solutions at hand, even if they are a bit more complex or less elegant. The latest version does not use eval anywhere in the code so it will not execute any user o lib user code, except for running the configured setup and teardown functions, of course. Whatever happens in those functions is up to the test developer. In any case, automated unit tests should never include user-supplied or variable data, to significantly low the risk of attacks using Continuous Integration systems or any automated test routines. Following that strategy, provashell does not run any user-supplied code other than the configured setup, teardown functions and the declared tests, and the assertion functions do not execute any external code to the best of my knowledge (you should check the code yourself anyway, grep -R eval src is your friend here).  It goes without saying that test shells should pretty much never be run as root, unless there is a very good reason (which there isn’t).

* Do not do (too much) funky stuff – Try to be as simple as possible and reuse code wherever feasible, so there is little repetition in the test library. It should also be easy to read and understand by any reasonably experienced shell coder. It is worth stressing again that shell scripts are real code and should be treated as such at all times.

* Use annotations to identify tests – Tests can be named any way the developer wants. I like annotations for tests because even though appending ‘Test’ to tests is a really simple convention, sometimes it adds extra cruft to test names in a context where it is not actually needed. For instance, the test called ‘errorIsReturnedOnNetworkTimeout’ is quite self-explanatory and easily understood in a test class context. Having ‘errorIsReturnedOnNetworkTimeoutTest’ does not add much to the definition and extends the name needlessly. It is of course a matter of style and it could argued that adding annotations to tests adds the same cruft, just in a different place. In any case, provashell uses annotations to identify tests and related functions which work well and are simple to use. Here’s a summary diagram of all the supported annotations (they need to go in a bash comment line):

provashell annotations
Function annotations defined by provashell

Yeah! Give me some shell test examples

Usage is pretty straightforward and can be demonstrated with an example. Imagine we have a function we want to test that counts the number of a’s in a text passed as a parameter. It is probably not a very good implementation but is a good enough example:

countAs() {
	c=$(printf %s "$1" | sed 's/[^a]*//g' | tr -d '\n' | wc -m)
	return "$c"
}

We then have two tests like this:

#@Test
countAsNormally() {

	countAs 'a'
	assertEq "Does not count one 'a' in a string with only one 'a'" 1 $?

	countAs 'b'
	assertEq "Does not count zero a's in a string with no a's" 0 $?

	countAs 'aaa'
	assertEq "Does not count three straight a's when they are there" 3 $?

	countAs 'abab'
	assertEq "Does not count two a's in a string when they are there" 2 $?

}

#@Test
countAsEdgeCases() {
	
	countAs ''
	assertEq 'Does not count empty string correctly' 0 $?
	
}

Once we have defined the tests we need to source the library like this (using whatever path we have for the provashell file):

. provashell

This will run the provashell code within the current context, causing the tests to be executed as expected, including the annotated pre and post functions.

Complete documentation

Extensive docs and examples can be found at the GitHub provashell project page. In any case, the source is quite short and readable.

Contributing

The provashell library uses the Apache 2.0 license. Pull requests are welcome so fork away!

Easy deployment of Zookeeper and Storm in RPM packages

In this post we will package Storm and its dependencies to achieve seamless deployment of a realtime big data processing system. Following up on the first Meteorit project article, we will be adding the minimal supervisor system mon, Zookeeper, zeromq and finally Storm itself. Packaging will enable fast deployment of the whole processing system using RPM packages.
Continue reading “Easy deployment of Zookeeper and Storm in RPM packages”

Crayon: codi font en colors en WordPress amb cara i ulls

Feia temps que buscava una bona solució per pintar codi font una mica decent. Fins ara tenia el Syntax Highlighter Evolved, basat en el Syntax Highlighter d’Alex Gorbatchev, però no n’estava gaire convençut.
Navegant per altres blocs he vist el resultat del plugin de Crayon per a WordPress, desenvolupat per Aram Kocharyan. El codi font es pot trobar en la seva pàgina de GitHub llest per fer-hi contribucions, amb llicència GPLv2.
Continue reading “Crayon: codi font en colors en WordPress amb cara i ulls”

Handling real-time events with Nginx, Lua and Redis

In this post, we will explore options to handle lots of HTTP events asynchronously using Nginx, Lua for the frontend and Redis as the backend. Although there are plenty of options out there to deal with this, we will check these technologies out in a bit more detail and leaving lots of options open for customisation and tuning, while using tried and true building blocks.
Continue reading “Handling real-time events with Nginx, Lua and Redis”

Building an FTP server Maven plugin from scratch

In this post we design and code a new Maven plugin that fires up a FTP server to aid in integration testing of FTP-bound processes, thus demonstrating the flexibility and power of Maven plugins.
Continue reading “Building an FTP server Maven plugin from scratch”

Nokia, Google and Microsoft, without any options in the mobile arena

Google, Microsoft and Nokia have something in common: they had little choice in some of their latest strategic moves in the mobile arena, maybe their biggest moves in a long, long time. Unusual for such giants, eh?

In this post I argue that Google, Microsoft and Nokia have something in common: they had little choice in some of their latest strategic moves in the mobile arena, maybe their biggest moves in a long, long time. Unusual for such giants, eh?

For the sake of drama and structure I will go from the most obvious to the slightly less evident and finish with the most subtle of the three.

Microsoft, Google, Nokia

Firstly, there is Microsoft.

It is common knowledge that the old boys from Redmond are in deep trouble in the mobile arena (and still are). After being a major player in the mobile smartphone arena for quite a long time it has gradually become a so-so player, being met by some skepticism by users and critics alike.

Even though Windows Phone 7 received decent reviews it feels like too little and definitely too late.

The Windows Phone 7 rollout still feels too corporate to me and still part of a lukewarm strategy.

Yeah but Microsoft has conquered or at least made a good dent on other markets before, even if just by brute force, yes? (read XBOX).

So why would a juggernaut like Microsoft be constrained in its long-term strategy? Shouldn’t be.

Well, it’s the systems, stupid.
If anything, Apple has taught the industry one thing with the iPod, iPhone and lately with the iPad. Consumers love SYSTEMS.

Consumers don’t like loosely coupled devices. Consumers don’t like replacing or upgrading components. Consumers don’t like lengthy boot sequences. Consumers don’t like invasive firmware installations. Consumers don’t like the software provider plus OEM combination. Maybe they did in the beginning of the 90’s. We ain’t in the 90’s anymore. XBOX, though successful, has been targeted at hardcore gamers rather than casual ones up until the Kinect.

On top of that Microsoft has learned something on its own, it doesn’t know how to build consumer SYSTEMS at all.

Therefore, to get itself out of the hole, Microsoft considered its options.

Acquisition of a successful company just to go on and destroy it in the process was ruled out, lest it turn into another disaster.

Lesson learned and any acquisitions ruled out, Microsoft then took a look at the OEM market… Err, in the case of MS, it meant HTC, builder of the 80% of cell phones bearing Microsoft’s mobile OS (at least prior the 7th version). Out of 50 partners!

Well, a simple Google search for Android and HTC is quite revealing. HTC didn’t jump ship, not exactly. But it started to be really busy building handsets with that OS.

Well, considering Android had been in the market for a while, a mature product, consumers liking it, with a plethora of apps and -most importantly- being FREE, what do you expect? Considering that MS has been known to thoroughly screw its partners from time to time., who can blame HTC?

Surely, HTC has hedged its bets and is only too happy to license to MS and adopt a wait-and-see attitude to it to see if it sticks and to exploit slow IT corporate policies that still mandate Windows-Everywhere(™). The good old leverage from the fading age of Windows was gone though and Microsoft would feel like second best here.

Choice taken away: traditional OEMs were out.

To add icing on the cake, it would seem that Microsoft had little choice but to license the ARM architecture, surely watched closely by its mobile hardware OEMs who have perceived it as yet another sword hanging over their little necks. A sword that would materialise as Microsoft mobile hardware. Uh-oh. Again, Microsoft has had to take some eggs from the software-only basket and put them on the systems one though only as a last resort (see Kin fiasco).

No pure-OEM model then. No acquisitions. Definitely not Android. SONY? You gotta be kidding. No Microsoft hardware (yet). Samsung maybe? Nay.

Um…

It had to be Nokia then. It’s smartphone marketshare in decline, Microsoft couldn’t go anywhere else.

Which leads nicely to the next big boy, Nokia itself.

Nokia is a giant. Massive with customers, it has created remarkable legendary mobile designs. However, its fortunes -or at least its marketshare lead- got reversed and were showing a worrying trend.

The new CEO put it very well himself, the company is in dire straits indeed, at least in the long term.

Nokia also considered its options.

Nokia understands systems and consumers very well, probably much more than Microsoft does or has ever done. With that in mind, Elop and senior managers understood that Symbian is a thing of the past that. Meego wasn’t ready and perhaps would never be good enough.

But, wait? A company the size of Nokia has lots of engineering talent! Yes, but also lots of middle managers and lacklustre leadership, if one is to believe Nokia’s top brass.

Therefore, culling the company of useless meddlers and taking the time to isolate a dedicated, motivated and driven team to push Meego (much like what Palm managed to do with its WebOS) up to scratch would take time, too long. If I remember correctly, it took Palm the better part of two years or more to get WebOS near 1.0-status and that is an heroic feat. Time though, is not something Nokia had available. Option out.

Acquisition? Nokia doesn’t strike me as an acquisition-crazy company.

Firstly, a nimble startup would understandably have a hard time integrating into a company of 12.000+ employees.

Secondly, while the Trolltech acquisition had some technical merit, it was completed in 2008 and Nokia has’t really made Qt onto the big promised paradise it was intented to. And they have had plenty of time to do it.

And finally, who to buy? RIM? Are you kidding? Making the company bigger and having a huge corporate cultural clash to boot?!? And getting RIM’s loony CEO(s) onboard?

Nay, purchasing its way out of trouble, choice taken away.

Which lead Nokia to consider Android.

One can imagine Nokia seeing Android as the forbidden fruit of sin. Juicy, tempting, free for the taking… and ultimately damning.

Oh, there is the legal swamp. Before taking a bite out of the fruit, Nokia must have had it’s lawyer legions take a good look at it.

If that was not to scare the Nokia strategist what would?

Well, I would argue that the real reason is freedom. Freedom to differentiate. As the Symbian guys put it “[…] surrender too much of the value and differentiation ability, most obviously in services and advertising, to Google.” Exactly. Nokia would play on the same field as all the other OEM. Bummer, from leading systems developer down to mere squabbling OEM.

Surely Nokia had been following what happened to Motorola and the Skyhook guys.

Apparently, both Motorola and ‘Company X’ (believed to be Samsung) have been prevented by Google from shipping handsets bearing its location technology. Remeber Navteq? Ever heard of Nokia maps?

Jump onto the Android bandwagon and kiss goodbye to all that.

So Android being out what was left as an option? A company in desperate need?

Microsoft.

Which leads to the idea of Google having courted Nokia precisely for many of the reasons Microsoft ‘bought’ them in the first place.

Finally, Google is the third big one having run out of options on some of its most strategic options.

Consider the official gospel on why Google created Android and released it to the wild. Well, most of it, anyway.

According to Vic Gundotra from Google:

“If Google didn’t act, it faced a draconian future where one man, one phone, one carrier were our choice. That’s a future we don’t want.”

Sounds chivalrous and following the ‘Don’t be evil’ mantra. Google released Android to save us all from that bleak future.

Well, it’s not true.

Consider Google’s business model. It revolves around advertising and to do well in advertising you need eyeballs, lots of them. Furthermore, to do very well you need targeted eyeballs, that is, to show the right adverts to the right people, to maximise chances of these people hitting the ads.

Google’s business model elevator pitch: person would like to get a service or product on the Web. Cool, she fires up Google on her desktop or laptop and does a search for the terms she thinks will help her find that product or service. Maybe clicks on the ads, maybe finds what she is looking for, maybe not. Bonus points: does more searches through Google, refining the terms (translation: getting more targeted ads), finally finds what she is looking for. This happens millions of times. Google makes tons of money.

Whereas if that same person would like to get a service or product on her iPhone, the story is completely different. Cool, she fires up the App Store app, does a search or browses the categories. Looks at the top charts, reads the reviews and how many stars the candidate solutions have, ponders wether to pay or get a free app, etc. Person chooses app and installs it. Google gets zero money. Zaroo, zip, nada, nothing.

There are a few things that need to be considered:

  • Not all services are available through the App Store and probably will never be a 100%, but there are zillions of apps (read, services) waiting to be downloaded.
  • People trust Google to deliver good search results but how to compare one result from the other can basically only be done by the rank itself or by clicking on the link and spending sometime on the site. The App Store offers a simplified interface allegedly easier the average consumer (ranks, reviews, stars, etc.).
  • People trust the App Store as well, it has never allowed malware on people’s devices.
  • Apple is very careful and vocal about not having any influence on the ranking, like, erm, Google.
  • To get iPhone apps, geeks will do research on Google and the Web. Will look for extended reviews, comparisons and specialised sites. Consumers just use the App Store.

Google knew that smartphones and later tablets would be the devices the mass market consumer will use to access the Internet. iOS was looking good and demolishing the competition at the time. It seems to be doing the same now.

The App Store is a ‘Google search’ replacement.

I would say that Apple wasn’t expecting it to become so successful. I would add that Apple didn’t set out to build a Google search replacement when the App Store was built. Apple engineers and product managers only wanted to create a better experience for consumers than googling vague terms, shuffling through specialised review sites, getting malware off dubious websites, etc. Google is collateral damage.

Google knew it had to act, very fast and with very decisive steps. Apple’s competition was in shambles and couldn’t get their stuff together for the life of them. Microsoft Windows Mobile 6.x was crap and any real future releases years into the future, RIM was -is- a shadow of its former self and Palm was on its last legs. Google couldn’t wait on them.

Android was born and released for free. Had to be. Google didn’t have any other choice. What they really imagined was a mobile future sans Android, where Apple would rule. Imagine if Android didn’t exist what the marketshare of Apple would be on smartphones today and Google wouldn’t be able to continue making money on it. That is the future Google didn’t want.

A corporate strategy is about having a vision and making it happen and the three giants have lost the initiative.

Templating the OSGi way with Freemarker

After some well-deserved rest the OSGi components on the server series is back with a vengeance and a somewhat independent post. For some background please check the other posts in the series.

A common feature of Web applications is the need to output structured content, be it XML, XHTML markup, JSON or many others. A number of technologies is used to do that but few seem to be that dynamic, usually only reloading templates when they change or loading them from URLs. Surely we can leverage OSGi to make things more interesting…

Therefore we should be following the OSGi dynamic philosophy as much as possible, exploiting the features made available by the framework such as dynamic discovery, services, bundle headers and the lot.

In the case of our cache app we are violating quite a few basic design principles by having the format embedded in the java code. So we should as well use a template separate from the code and if possible reuse some existing well-known templating engine from somewhere.

Given these basic design principles let’s get started.

Firstly, we need a robust and flexible templating engine. We select the mature Freemarker engine which is reasonably speedy and has the power and flexibility we want. Make sure you check out the license, of course.

We could stop at putting the library JAR in a bundle and package it so it can be used by any other bundle and that is what we do to be able to use it in OSGi. That however doesn’t exploit many of the nicer OSGi capabilities so we will create another bundle called ‘com.calidos.dani.osgi.freemarker-loader’.

What we want is to allow bundles to carry templates inside them and have our freemarker-loader templating bundle discover them automagically. This is the same technique that the Spring dynamic modules use and you can find more info here. The mechanism is described in this diagram:

OSGi freemarker templating diagram

That is easy enough with a BundleTracker and a BundleTrackerCustomizer implementation. The BundleTracker class tracks bundles being added to the environment like this:

tracker = new BundleTracker(context, Bundle.RESOLVED, templateTracker);
tracker.open();

With this snippet the tracker instance will look for bundles in the RESOLVED state (which lets us track fragments). The ‘templateTracker’ object is an instance of BundleTrackerCustomizer and will receive callbacks whenever bundles are added to the environment.

For instance, when a bundle is added we check for a special header in the bundle which tells us what is the relative path of available templates in the bundle being resolved:

public Object addingBundle(Bundle bundle, BundleEvent event) {
		
// we look for the header and act accordingly		
String templatesLocation = (String) bundle.getHeaders().get(TEMPLATE_HEADER);
if (templatesLocation!=null) {
			 
	Enumeration bundleTemplates = bundle.findEntries(templatesLocation, "*.ftl", true);
	HashSet templatesFromAddedBundle = new HashSet();
	while (bundleTemplates.hasMoreElements()) {
			
		URL templateURL = bundleTemplates.nextElement();
		addTemplate(bundle, templateURL,templatesLocation);
		templatesFromAddedBundle.add(templateURL);
				
	}
	
	templatesOfEachBundle.put(bundle.getBundleId(), templatesFromAddedBundle);
			
}
return null;
		
}	// addingBundle

An interesting method being used here is ‘findEntries’ which loads all the entries in the provided templates folder and lets us add them to our holding structure. We also take care to implement the methods to remove the templates and update them accordingly whenever bundles are updated or unloaded from the environment.

Having TEMPLATE_HEADER with a value of ‘Freemarker-Templates’ means that bundles having a header such as Freemarker-Templates: /templates will have any templates within that folder (please note that the ‘/templates’ bit is not added to template URLs!).

The next thing we need to do is make the loaded templates available to the environment. To do that we make a freemarker Configuration object accessible as an OSGi service object. That Configuration instance is the main object Freemarker to load and use templates and has an interesting mechanism to override its template loading mechanism we use to make available our OSGi environment templates.

freemarkerConfig.setTemplateLoader( new URLTemplateLoader() {
			
@Override
protected URL getURL(String url) {
Stack templateStack = templates.get(url);
if (templateStack!=null) {
	TemplateEntry templateStackTop = templateStack.peek();
	if (templateStackTop!=null) {
		return templateStackTop.getTemplateURL();
	}
	return null;
}
return null;
}

});

The service Configuration object is set with a template loader inner class that uses our template holding objects to retrieve templates stored in our OSGi context. Cool.

This also allows us to effectively disable the template refreshing cycles that Freemarker does by default (allegedly making it slightly more efficient). Now we only need to refresh a bundle containing the templates to get the new version. This can be modified by using the appropriate methods on the Configuration service of course. (There is another method explained later).

An interesting feature we can add to exploit the dynamic nature of OSGi is to make templates available in a stack. This means different bundles can dynamically overwrite templates by the same name. Moreover, once a template is removed the previous version becomes available. This can be used to make temporary changes to templates to add or remove diagnostics information, test new templates temporarily, etc.

We do that using a Stack of TemplateEntry objects, TemplateEntry being a helper class to store template entries.

This is all very nice but we have a problem when having multiple versions of the same bundle that hold multiple copies of the same template, this means they will stack and we have no way to access a particular version of a template. To solve this problem we store each template in three different stacks by three different URLs:

  • ‘path/file.ftl’
  • ‘bundle://bundlename/path/file.ftl’
  • ‘bundle://bundlename:version/path/file.ftl’

In this manner we can use the more generic URL in most cases but still can access specific versions when needed. It is important to think about the dynamic nature of OSGi as well as the possibility of several versions of the same bundle coexisting peacefully in the same environment.

From the perspective of any bundle using the service in the simplest case it only needs to look for a service named ‘freemarker.template.Configuration’. For convenience, the property ‘dynamicConfiguration’ is set to ‘true’ to coexist peacefully with other Configuration services (maybe coming from an official Freemarker bundle). For instance, if we know for sure our dynamic Configuration service is the only one present we can do:

context.getServiceReference(Configuration.class.getName());

That will give us the highest ranking Configuration service. If there are several such services we can use a call like this to get the one that has the dynamic OSGi loader:

context.getServiceReferences(Configuration.class.getName(), "dynamicConfiguration=true");

There is one last feature which lets bundle users feed an already configured template configuration object to the bundle by publishing a Configuration service with property ‘preparedConfiguration’ set to ‘true’. This will get picked up by the bundle and its template loader put in sequence with the dynamic OSGi template loader. This means that any original Configuration settings are maintained (For further information on service filtering based on properties please check the BundleContext javadoc.).

Best thing to do is to go and try it by downloading the bundles here. Source is also available.

Components on the server (6): adding Integration Testing

In this installment of the server-side OSGi series, we add integration testing capabilities to our project. Integration testing goes beyond plain unit testing and checks the interactions between real components. This is in contrast with unit testing, which generally uses mockups to represent components outside the one being tested. Please take a look at previous installments, as usual.

In the case of integration testing, it is manly used in a pre-production environment, with a valid build that has all unit tests passed. It can even be used in production to just after a deployment is made, taking care not to have destructive checks or massive load tests in the integration test code. YMMV.

To achieve integration testing we need to check the various OSGi components deployed interact in the way that is expected of them. Therefore we need to test the components in a group and not in isolation. To do that in the OSGi world means we need to have access to the OSGi context from within the tests to access services, call them and check their responses, etc.

To allow for this kind of integration testing within the OSGi environment, we make a slight modification to the excellent test.extender we have already patched in the previous installment.

Basically, the basic test.extender seeks out any JUnit test classes within the fragment bundle, creates an instance using an empty constructor and then fires up the tests. This is activated either by default when the fragment is loaded or by using ‘test ‘ in the console. For further information please see the previous post about this subject.

For our integration testing, we add an extra command to test.extender:

public Object _integrationTest(CommandInterpreter intp) {
        String nextArgument = intp.nextArgument();
    	testExtender.integrationTest(Long.parseLong(nextArgument));
    	return null;
}

And we refactor the TestExtender to add the integrationTest method which reuses some of the code to instantiate test cases using a constructor that accepts the OSGi context as a parameter.

Constructor[] constructors = clazz.getConstructors();
boolean foundConstructor = false;
for (int i = 0; i < constructors.length && !foundConstructor; i++) {
	Constructor constructor = constructors[i];
	Class[] types = constructor.getParameterTypes();
	if (types.length==1 && types[0].isInstance(context)) {
		foundConstructor = true;
		EClassUtils.testClass(inspectClass, constructor.newInstance(context));
	}
} // for

The OSGi context is passed onto the constructor and then the test class is run. It is obviously up to the test class to use the context appropriately for its integration testing.

In our cache project setup, we can do some useful integration testing on the cache.controller component, basically checking if the interaction with the provider components is behaving as we expect it. The integration testing is also added to a fragment that can be deployed optionally, of course.

We start by creating the fragment and adding a testing class like this:

Adding test class

Next, we add the constructor that accepts an OSGi context, which is very simple:

public CacheIntegrationTest(BundleContext ctx) {
	super();
	this.context = ctx;
}

In the setup and teardown methods we get and unget the cache service to perform the testing:


public void setUp() throws Exception {
	serviceReference = context.getServiceReference(Cache.class.getName());
	controller = (CacheControllerCore) context.getService(serviceReference);

}

public void tearDown() throws Exception {		
	context.ungetService(serviceReference);
	controller = null;		
}

In this case we get the controller cache service and store it in an instance used to perform the tests. This is quite simple and fulfills our intended purpose but we still have the flexibility to make more complex integration testing if needed.

Next we create as many test cases as needed:

public void testGet() {
	try {
		controller.init();
		double v = Math.random();
		String k = "/k"+v;
		controller.set(k, v);
		assertEquals(v, controller.get(k));
	} catch (CacheProviderException e) {
		e.printStackTrace();
		fail(e.getMessage());
	}

}

It should be noted that while the code looks like regular testing code, it is actually using real services from the OSGi environment as opposed to mockups. This means we are testing the real integration between components as well as the individual controller component code. The disadvantage here is that if there is an error in the controller we might mistake the problem with an issue with the services used. In conclusion, having integration code doesn’t negate the need to have unit tests.

Once we load the fragment onto the environment, first we need to obtain the bundle id of the integration fragment and then launch the integration testing in this manner:


osgi> integrate 125
Bundle : [125] : com.calidos.dani.osgi.cache.controller.integration
_
CLASS : [com.calidos.dani.osgi.cache.controller.CacheIntegrationTest]
___________________________________________________________________________
Method : [ testInit ] PASS
Method : [ testInitInt ] PASS
Method : [ testSize ] PASS
14:21:43,077 WARN CacheControllerCore Couldn't clear some of the provider caches as operation is unsupported
14:21:43,077 WARN CacheControllerCore Couldn't clear some of the provider caches as operation is unsupported
Method : [ testClear ] PASS
Method : [ testSet ] PASS
Method : [ testGet ] PASS
Method : [ testGetStatus ] PASS
___________________________________________________________________________

The results tell us that all operations are OK but we need to bear in mind that the clear operation is not supported in some backend caches. If this is what is expected by the operator then all is fine.

We take advantage of the new integration testing functionality to make some extensive changes to logging and exception handling of the controller code. By running the integration tests we make sure all seems to work fine (even though we still need some proper unit testing of the controller). Modifications are made quite quickly thanks to the integration tests.

To recap, we’ve added integration testing support to the existing ‘test.extender’ bundle and created integration testing code for the cache controller component. This has allowed us to make code changes quickly with less risk of mistakes.

Here you can find a patch for the test extender project as well as the patched testing bundle already compiled. Enjoy!

El manifest – Unlink your feeds

Gràcies al Roger per enllaçar al manifest ‘Unlink your feeds‘, em subscric totalment a la iniciativa.

La idea és anar amb molt de compte en enllaçar els posts i updates de microblogging entre els diferents serveis, no sempre és lo òptim. Com diu el mateix Roger:

Un clam al cel perquè la gent deixi de reaprofitar el mateix missatge per a totes les xarxes socials. Crea soroll innecesari, molts cops perd context (hashtags a facebook, etc…) i molesta. Ok a que em diguis “avui he dinat molt bé”, però no necessito rebre-ho per triplicat.