Introduction
We spent the greater part of a decade working on a Test Automation system. That time spent was one of great productivity and really enjoyable. However, the question is what should a test automation focus on keeps coming back. Should one write a system that incorporates test reporting, scheduling, simple UI etc.. or should one write test cases that are effective? Obviously everyone wants test cases but there is always pressure to make the reports look 'nice', make it simple so that even the CEO's assistant can run it!Test Automation in TCL/TK
The first system we wrote was in TCL/TK and used XML to configure different test suites, tests, and parameters. Using TCL/Expect, it automated interactions with a SoC on a board via a serial interface. It was designed to test Linux and RTOS network drivers. However, the interfaces kept changing and it really didn't do what we wanted to. The only place it made sense was the performance and long duration tests where it was hooked up to an IXIA network simulator and controlled network traffic.Test Automation in Ruby
The second system was written simply because a PHB at the top of the corporate ladder decided that we can get better 'efficiency' if all the test automation teams worked together to write a comprehensive one-shoe-fits-all system. Our counterparts had used Ruby and it seemed pretty cool option to try. This system needed work for testing RTOS'es, Linux drivers, Multimedia frameworks, Compilers, SDKs and the Kitchen Sink. After all, the PHBs thought, testing is a single domain and everything else is incidental. The big problem was that we needed to talk to teams the other side of the world everyday to do anything useful. The project management was a nightmare and we ended writing a lite version of the software which is still being used.Costs
Looking back I would say we spent the following- 4 resources worked full time for 10 years on test automation with us. That's 4800 days spent on building, maintaining and enhancing an internal tool.
- Effort spent was 9600 person days on an average. There were times when we had more people working on the project but really that doesn't matter.
- Assuming that each resource was billed $4000 per month that works out to $20 per hour. A low figure I picked on purpose so that we are conservative in our cost estimate.
- It works out $192000 at a minimum. If we are being realistic, then $100 per hour is probably the right ballpark number.
So, I would estimate that its atleast $960000 spent on test automation frameworks. This is a cost separate from actual test case implementation, test execution, project management, managing expectations, vendor management, outsourcing management etc..
