GREE Haskell test-sandbox

Description

test-sandbox is a framework to manage external applications and communicate with them via TCP or standard I/O for system testing in a sandboxed environment. The Test.Sandbox monad can either be used stand-alone or in conjunction with HUnit, QuickCheck and the test-framework packages to build a complete test suite. The API is meant to be simple to understand yet flexible enough to meet most of the needs of application testers.

Features

• Register, start and stop programs in a sandboxed environment.
• Automatic cleaning at shutdown: started processes are shutdown, temporary files are deleted.
• Ask the framework to provide you with random, guaranteed not bound TCP ports for your tests: no more collisions when running 2 sets of tests at the same time.
• Generate your temporary configuration files programatically in a secure manner.
• Easily share variables between your tests and modify them at runtime.
• Combine with the test-framework package for standardized output and XML test result generation.
• Use the QuickCheck library to write property tests and generate automatic test cases for your external application; enjoy the full power of the Haskell test harness, even if the application to test is written in a different language!

Examples

A simple test for the memcached NoSQL key-value store, with communication via TCP.

import Test.Sandbox
import Test.Sandbox.HUnit

setup :: Sandbox ()
setup = do
  port <- getPort "memcached"
  register "memcached" "memcached" [ "-p", show port ] def

main :: IO ()
main = sandbox $ do
  setup
  start "memcached"
  output <- sendTo "memcached" "set key 0 0 5\r\nvalue\r\n" 1
  assertEqual "item is stored" "STORED\r\n" output

A simple hypothetic test for the "sed" UNIX command-line utility, with communication via standard I/O.

import Test.Sandbox
import Test.Sandbox.HUnit

main :: IO ()
main = sandbox $ do
  start =<< register "sed_regex" "sed" [ "-u", "s/a/b/" ] def { psCapture = CaptureStdout }
  assertEqual "a->b" "b\n" =<< interactWith "sed_regex_ "a\n" 5

Here is how to cram the above in the Test.Framework module.

import Test.Framework
import Test.Framework.Providers.Sandbox
import Test.Sandbox
import Test.Sandbox.HUnit

setup :: Sandbox ()
setup = start =<< register "sed_s/a/b/" "sed" [ "-u", "s/a/b/" ] def { psCapture = CaptureStdout }

main = defaultMain [
    sandboxTests "sed_tests" [
        sandboxTest "setup" setup
      , sandboxTest "sed a->b" $ assertEqual "a->b" "b\n" =<< interactWith "sed_s/a/b/" "a\n" 5
      , sandboxTest "sed aa->ba" $ assertEqual "aa->ba" "ba\n" =<< interactWith "sed_s/a/b/" "aa\n" 5
    ]
  ]

Generated output

benjamin-surma@g-pc-3964:~/Documents/memcached-tests$ ./dist/build/test/test
[sed_tests]
  [setup]

##------------------------------------------------------------------------------
 ## sed_tests end-to-end test environment                                     --
## ##---------------------------------------------------------------------------
-- Data directory: /tmp/sed_tests_20474
-- Allocated ports: 
-- Configuration files: 
-- Registered processes: sed_s/a/b/
--------------------------------------------------------------------------------

Starting process sed_s/a/b/... Done.
[OK]
  [sed a->b]  [OK]
  [sed aa->ba]  [OK]
sed_tests:
  setup: [OK]
  sed a->b: [OK]
  sed aa->ba: [OK]
  cleaning: [OK]

         Sandbox tests  Total      
 Passed  4              3          
 Failed  0              0          
 Total   4              4          
benjamin-surma@g-pc-3964:~/Documents/memcached-tests$ 

Synopsis

At GREE, we spend lots of time meticulously testing our internally-developed middleware. We have solutions not only developed in Haskell, but also C++ and PHP, but wanted a simple and robust test framework to perform end-to-end testing, and this is how test-sandbox is born.