akaros/Documentation/testing.txt
<<
>>
Prefs
   1Automated Testing 
   2----------------------------
   3This document outlines our plans for automated testing with Akaros.
   4When completed, the testing framework will be used to both verify successful
   5compilation of akaros (and its supporting user libraries and cross compiler),
   6as well as perform automated tests of these.
   7
   8Currently, our testing is done in a very ad-hoc manner.  For kernel tests, we
   9simply add a function to the file 'kern/src/testing.c' and run it periodically
  10from the kernel monitor to verify that it still works. For user tests, we
  11create a new file in 'tests/' and run 'make tests' followed by 'make fill-kfs'
  12to make and install the tests so akaros can run them once booted.  We then just
  13peridically run them when we think we might have broken something.
  14
  15We obviously need something better.
  16
  17At a minimum we need a framework that can automate the following on every commit:
  18        - Verify successful kernel compilation
  19        - Verify successful compilation of our user libraries
  20        - Verify successful compilation of the cross compiler for all supported
  21          archs on commits that touch a file in the cross compiler
  22        - Compile and run a suite of kernel and user tests and verify their
  23          success
  24
  25The framework should support easily adding/removing tests, as well as a way to
  26easily turn them on and off, without actually removing them. We should also
  27have a way of notifying us if something breaks, and a way to turn off
  28notifications for individual tests, if things start to get too noisy.
  29Independent of how we do notifications, we should have a dashboard indicating
  30the current state of what's passing/failing, and a way to look through the
  31history of this for every commit.
  32
  33An example of such a dashboard can be found at: http://build.golang.org/
  34Our's does not need to be this sophisticated at first.  A simple text file with
  35all the relevant information will suffice, and it's history can be kept track
  36of by versioning it in a git repo.
  37
  38
  39Unorganized stuff
  40----------------------------
  41Use configure variables integrated with KBuild for kernel test on/off
  42Mention difficulty of running some tests.
  43        - Need specific kernel configs
  44        - Specific timing between injecting inputs for some tests
  45Mention technologies we are considering using
  46        - Junit like framework for writing tests
  47        - Autotest? https://github.com/autotest/autotest
  48Send email notification if cross compiler needs a rebuild
  49
  50
  51Links 
  52----------------------------
  53- Converting VirtualBox VDI image to Xen:
  54  https://www.serverstack.com/blog/2012/11/20/converting-virtualbox-vm-to-a-xen-hypervisor-virtual-machine/
  55
  56
  57================================================================================
  58======================== W.I.P. OF ACTUAL DOCUMENTATION ========================
  59================================================================================
  60
  61testing.txt
  62Alfonso Gomez Jordana and Kevin Klues
  63
  64This document explains how the AKAROS testing framework works: It contains both
  65straight-to-the-point instructions on where & how to set up a new unit test, to
  66a detailed and thorough explanation of how all the components work together 
  67under the hood to make tests run.
  68
  69Part 1: Overview
  70Part 2: Instructions for adding a new test
  71Part 3: How to run tests
  72Part 4: Kernel testing internals
  73Part 5: Userspace testing internals
  74Part 6: Continuous integration server
  75
  76
  77Revision History:
  782014-02-26 - Initial version
  79
  80Part 1: Overview 
  81=====================================
  82
  83The AKAROS testing framework consists on an automated Continuous Integration
  84server that runs several kinds of test whenever a new commit is pushed to the
  85master branch in the repository, and generates reports with the results.
  86
  87Some or all of the following things will be tested when a new commit is pushed
  88to the server, depending on the parts of the code that are modified:
  89* Cross-compiler compiles without errors
  90* Kernel compiles without errors
  91* Userspace libs compile without errors
  92* Userspace programs compile without errors
  93* Kernel tests pass without errors (this includes during-boot kernel tests and
  94  post-boot kernel ones)
  95* Userspace tests pass without errors
  96
  97
  98
  99Part 2: How to add a test
 100=====================================
 101
 1022.1 Determining the type of test to create
 103------------------------------------------------------------------
 104There are three different types of tests that you can create now in Akaros:
 105* Unit tests in kernel space that run at some specific point during boot.
 106* Unit tests in kernel space that run after boot.
 107* Unit tests in user space that run after boot.
 108
 109(These may be extended in the future to include, for example, cross-compiler 
 110unit tests.)
 111
 112The reasons for choosing one or another would be: 
 113TODO: Add reasons
 114
 1152.2 During-boot kernel unit tests
 116------------------------------------------------------------------
 117TODO: Write
 118
 1192.3 Post-boot kernel unit tests
 120------------------------------------------------------------------
 1211. Go to akaros/kern/src/tests_pb_kernel.c
 1222. Create a new function for the test. Follow the following pattern:
 123        bool test_[NAME](void)
 124        {
 125                [CODE]
 126                return true;
 127        }
 1283. Inside the [CODE] section, use the following assertion functions:
 129        * KT_ASSERT(boolean-expression): runs the expression and, if it fails,
 130          it finishes the full unit test with a result of FAILED.
 131        * KT_ASSERT_M(message, boolean-expression): like KT_ASSERT but, in case
 132          of failure, the message gets printed as cause of failure.
 1334. Go to akaros/kern/src/tests.c
 1345. Add inside the pb_kernel_tests array a line like the following:
 135        PB_K_TEST_REG([NAME], true),
 136        (Optionally, disable temporarily the test by setting the second arg to
 137        false).
 138
 139        If you'd like a test to be ran only in a given architecture, surround
 140        it here by #ifdefine CONFIG_[ARCH] ... #endif tags.
 141
 1422.4 Userspace unit tests
 143------------------------------------------------------------------
 144TODO: Write
 145
 146
 147
 148Part 3: How to run a suite of tests
 149=====================================
 150
 151By default, tests are not run automatically. Tests have to be enabled somehow 
 152during compilation.
 153
 154The easiest way to activate tests is to activate them all by running 
 155        make ARCH=XXX deftestconfig
 156
 157If you need more specific control about which tests to activate, you can run
 158        make ARCH=XXX menuconfig
 159and then select, from the Testing submenu, which tests to run.
 160
 161Once you activate the tests you want, you will just need to compile kernel, 
 162userspace tests... and then boot akaros, and they'll run automatically and 
 163output to console :)
 164
 165Additionally, every time a new commit is pushed to master in the github repo, 
 166the Continuous Integration server (see part 6) will automatically run relevant
 167tests and report on them.
 168
 169
 170
 171Part 4: Kernel testing internals
 172=====================================
 173
 1744.1 During-boot testing
 175------------------------------------------------------------------
 176TODO: Write
 177
 1784.2 Post-boot testing
 179------------------------------------------------------------------
 180To accomplish post-boot testing, the following files are involved:
 181        akaros/kern/src/manager.c
 182        akaros/kern/src/tests.c
 183        akaros/kern/src/tests_pb_kernel.c
 184        akaros/include/test_infrastructure.h
 185
 186test_infrastructure.h:
 187        This file defines macros used for assertions, as well as data
 188        structures, variables and a macro for registering postboot tests.
 189
 190tests_pb_kernel.c:
 191        This file contains the implementation of all the tests in this category.
 192
 193tests.c:
 194        This file is where all test names are registered for them to be ran.
 195
 196manager.c:
 197        In this file, via the function postboot_kernel_tests(), all tests are
 198        executed after the kernel has loaded. This function is in charge of printing
 199        all test results too.
 200
 2014.3 Userspace testing
 202------------------------------------------------------------------
 203TODO: Write
 204
 2054.4 Enabling / disabling tests
 206------------------------------------------------------------------
 207For enabling and disabling tests, we use CONFIG variables. In particular, the
 208following are the variables used:
 209        * CONFIG_POSTBOOT_KERNEL_TESTING
 210        * TODO: Fill this in once we have more.
 211
 212These variables are defined in the akaros/KConfig file.
 213
 214TODO: Speak here about deftestconfig once we have set it up
 215
 216
 217
 218TODO: Write rest of sections
 219