We wanted to set up an automated testing environment for the Blackberry in which we could test against multiple versions of a real BES. We have a pretty solid framework for automated testing of OfficeWriter, and since it proved itself to be very beneficial, we decided to try to do the same for our BlackBerry product. We already have a framework in place which runs automated tests on the simulator using the MDS simulator. We even extended this framework to run against an actual SharePoint server, although that’s a topic for another blog post. So, if precedence counts for anything, setting up an automated test environment for BlackBerry should have been a snap, right? Wrong. Here I wish to describe the path we walked down as we tried to set up a test environment that would run automatically against a real BES.

Setup

In order for a simulator to run against a real BES, it needs to connect to Desktop Manager, so we have set up two (Windows 7 64bit) virtual machines, each with a simulator and a Desktop Manager set up with a user provisioned through BES – one for BES 4 and one for BES 5. In order to automatically access and control these machines we chose to use PsExec from our Ant script using the exec tag and passing in the needed arguments. So, for example the followings, as part of an Ant target, will restart MyRemoteMachine using PsExec:

One important thing to know when using psexec on a remote machine is that psexec generates the service, copies it into the ADMIN$ location on the remote machine and then installs it. Therefore the account under which psexec is running, needs to have admin rights on the remote machine.

Simulator Commands

Originally we had Desktop Manager start on startup and the simulator start from Ant. The command line options we used to start the simulator are:

The important ones being the last three:

• comm-cable-connected: starts up the simulator with the usb connected so it can connect to desktop manager. Note: after a few times of running the simulator this option becomes grayed out (it is no longer available) so we have set up our scripts to restart the vm before running the tests.
• no-radio-on-at-startup and no-load-networks: these switches make sure all networks are turned off. If there is a network transport turned on, the simulator (as well as a real tethered device) will try going through them (and will most likely fail) instead of going through Desktop Manager.

Problems

Once we started the simulator and allowed time for loading, we used fledgecontroller to load and run our application. However, it seemed that the simulator was not recognizing Desktop Manager. We couldn’t see what was happening on session 0 where the test ran (we’ll discuss this shortly), so we couldn’t know what was going wrong… A quick google search showed that blackberry products such as the desktop manager and the simulator have many issues with windows 7 64bit machines, so we decided to try it on a Windows 2003 32bit vm. We have set up the new vm with all the required software and ran the test. Again the simulator was unable to connect to desktop manager. Where were we supposed to go from here?

Sessions

Before we proceed let’s spend a few minutes discussing the impact of sessions in general and session 0 isolation on Windows 7 in particular, both on PsExec and on our testing. As can be read in the links above, session 0 is the Console session that is always running and PsExec by default runs on session 0. When a user RDP‘s into a remote machine (or vm) it creates a new session (on Windows 7, applications run on sessions other than 0 as needed). It is possible to toggle which session PsExec will interact with using the i switch (see PsExec specs). However that session needs to preexist – as PsExec can only interact with existing sessions, not create new ones. It is important to note here that on Windows 7, trying to run Desktop Manager or Fledge using PsExec and interact with a session other than 0 (using the i switch) didn’t work. It seems that since PsExec is a service and has to run on session 0 for Windows 7, it failed to start the applications on other sessions. As long as we interacted with session 0 it was able to start Desktop Manager and Fledge (on session 0), but once we tried to have it interact with any other session, it failed silently – it returned that the process started with id xxx, but no such process was found on the machine.

Troubleshooting

What we were able to do at this point was to remote desktop into the vm on session 1 and instruct psexec to interact with session 1 instead of session 0 on windows 2003. Then we were finally able to see what the tests were doing and why they were failing:

The first thing we noticed was that if Desktop Manager is running, it tries to connect to the simulator as soon as the simulator starts, but since the simulator is still loading (as we all know this takes a very long time), the desktop manager pops up an error that it failed to connect. Now, pop up dialogs are one of the automated test’s worst enemy – since the test runs with no human operator, on a dialog it just hangs… This was the first cause why desktop manager and the simulator did not connect. So we reversed the order, after restarting the vm we first started the simulator, only then the desktop manager. But still, they would not talk to each other! Looking again at the tests run on session 1, we saw that if the desktop manager starts just before the simulator finished loading (which happened to be the case) it does not pop up an error but it also does not complete the connection. It tries to connect and then hangs, never finishing sync. So this was an easy fix – we simply increased the time we wait for the simulator to start.

So we ran the tests again on session 1 of the win2003 machine, and it actually got the simulator and desktop manager connected and successfully started the tests! Which then immediately failed. We were still happy we got desktop manager and the simulator talking and before moving to work on why the tests themselves were failing, we ran the build back on session 0 (in which it will have to run since it is the only session that is reliably active). And it failed to connect the simulator to Desktop Manager. It seems that since session 0 is the console session and does not interact with the desktop, the two are not able to communicate to each other. In other words, if desktop manager and the simulator run in session 0 – they are not able to connect.

Conclusion

We did a brief search and could not find a way to programmatically start a new session, and automating an RDP session might be possible but is not obvious. Eventually, since we could not see a clear end to the problems, and since the benefits do not justify spending so much time on it, we have decided to declare defeat. The difficultness of Blackberry has won; we will not have automated tests that connect to a real BES. However, if you have a suggestion or have successfully run automated tests that connect to a real BES, please, please let us know in the comments!

Follow

Related posts: