Test Automation requires constant maintenance

Introduction

We spent the greater part of a decade working on a Test Automation system. That time spent was one of great productivity and really enjoyable. However, the question is what should a test automation focus on keeps coming back. Should one write a system that incorporates test reporting, scheduling, simple UI etc.. or should one write test cases that are effective? Obviously everyone wants test cases but there is always pressure to make the reports look 'nice', make it simple so that even the CEO's assistant can run it!

Test Automation in TCL/TK

The first system we wrote was in TCL/TK and used XML to configure different test suites, tests, and parameters. Using TCL/Expect, it automated interactions with a SoC on a board via a serial interface. It was designed to test Linux and RTOS network drivers. However, the interfaces kept changing and it really didn't do what we wanted to. The only place it made sense was the performance and long duration tests where it was hooked up to an IXIA network simulator and controlled network traffic.

Test Automation in Ruby

The second system was written simply because a PHB at the top of the corporate ladder decided that we can get better 'efficiency' if all the test automation teams worked together to write a comprehensive one-shoe-fits-all system. Our counterparts had used Ruby and it seemed pretty cool option to try. This system needed work for testing RTOS'es, Linux drivers, Multimedia frameworks, Compilers, SDKs and the Kitchen Sink. After all, the PHBs thought, testing is a single domain and everything else is incidental. The big problem was that we needed to talk to teams the other side of the world everyday to do anything useful. The project management was a nightmare and we ended writing a lite version of the software which is still being used.

Costs

Looking back I would say we spent the following

1. 4 resources worked full time for 10 years on test automation with us. That's 4800 days spent on building, maintaining and enhancing an internal tool.
2. Effort spent was 9600 person days on an average. There were times when we had more people working on the project but really that doesn't matter.
3. Assuming that each resource was billed $4000 per month that works out to $20 per hour. A low figure I picked on purpose so that we are conservative in our cost estimate.
4. It works out $192000 at a minimum. If we are being realistic, then $100 per hour is probably the right ballpark number. 

So, I would estimate that its atleast $960000 spent on test automation frameworks. This is a cost separate from actual test case implementation, test execution, project management, managing expectations, vendor management, outsourcing management etc..

Summary

My advice - don't spend too much test frameworks, test automation systems. Instead focus on the actual tests. Focus on automating your tests with simple scripts and don't worry too much about tying them all together. Is it better to have a beautiful system with few tests, great reporting, fantastic UI OR a large bank of tests, crummy reports and command-line runners? The latter gives you better bang for the buck for sure. End of the day, XKCD sums it up neatly.

Cross compiling TCL for ARM

Refer http://objectmix.com/tcl/15449-how-cross-compile-tcl8-4-tk8-4-arm-linux.html

Also change the Makefile. Replace "./tclsh" with "tclsh". That way we run the build system's tclsh to install rather than the freshly built tclsh.

GUI with Gambit

There doesn't seem to be any real GUI libraries for Scheme. I spent
some time looking for a GUI framework that I can interface with
Gambit. WxWidgets was the first place I looked. I wasn't fully sure
that I could interface a C++ GUI framework with Gambit. The docs
mentioned it, but I didn't want to spent time exploring and debugging
C++ issues this time around.

Tk is a GUI library that is cross platform, works almost everywhere
without a hitch. I had spent a lot of time coding in Tcl/Tk a few
years back, and so getting back on the Tk bandwagon looked
appealing. The latest "Tile" updates to Tk also improves the
look-and-feel considerably. So, what options did I have for Scheme +
Tk?

On obvious way is to look at STKlos. The only reason, I didn't want to
do that, was because I wanted to stick to Gambit for the
moment. Gambit allows me to create a standalone application. Having a
standalone exe simplifies application delivery, esp. on Windows.

My first thought was to have a native interface to Tk. I thought about
a FFI interface to the Tk dll itself. Again no luck here. There was a
Snow package for Tk available but Snow itself seems to be dead with no
development happening on it.

I came across PS/Tk. PS/Tk uses Tcl as a bridge to Tk. You need to
have TCL installed though. We can get past this, if we use tclkit and
so, it looked like a good starting point.

The whole PS/Tk is just one file - pstk.scm that you download. Just
load the file and it should be good to go.

A-ha one Gotcha!

That doesn't work. What you need is to modify the file for specific
scheme support. There is example code written up for each scheme
implementation that PS/Tk supports and its just a question of
uncommenting the right one. For Gambit the code looks like so -

; GAMBIT
;   (run-program
;     (lambda (program)
;       (let ((port
;               (open-process
;                 (list path: "/bin/sh"
;                 arguments: (list "-c" (string-append "exec " program))
;                 stderr-redirection: #t))))
;         (list port port))))
;
;   (flush-output-port force-output)

Uncommenting this won't do as its Linux specific. Some small changes
and we should be able to use it on Windows. The changes are -

; GAMBIT / Windows
(run-program
(lambda (program)
 (let ((port
   (open-process
    (list path: "c:/tcl/bin/tclsh85.exe"
   stderr-redirection: #t))))
   (list port port))))

(flush-output-port force-output)

Once that's done, we are good to go. We can create a simple frame with
a button like so -

(define f (tk 'create-widget 'frame))
(define b (f 'create-widget 'button))
(b 'configure 'text: "Hello World")
(tk/pack b)
(tk/pack f)

Next stop - change from a full fledged Tcl/Tk installation to Tclkit.