akaros/GETTING_STARTED.md
<<
>>
Prefs
   1Getting Started with Akaros
   2==========================
   3Barret Rhoden
   4
   5Last thorough update: **2013-02-07**
   6
   7Contents
   8-----------
   9+ Overview
  10+ Need Help?
  11+ From Download to Hello World
  12  - Cross compiler (and glibc)
  13  - Kernel
  14  - Userspace
  15  - Busybox
  16  - Building and Loading a Virtual Machine Image
  17  - Running Qemu
  18  - Running on Hardware
  19  - Hello World
  20  - Other Dev Workflow Stuff
  21+ RISCV
  22+ Other Developer's Stuff
  23
  24
  251. Overview
  26---------------------
  27In this document, I outline what steps I go through to set up my development
  28environment.  Some devs use other setups, and they can put sections farther
  29down in this document describing their steps.
  30
  31Due to the nature of different workflows and different Linux distros, you'll
  32need to do some configuration of your environment.  Things won't magically work
  33out of the box.
  34
  35
  362. Need Help?
  37---------------------
  38First off, if you get stuck, email someone.  You can join our mailing list by
  39sending an email to
  40[akaros+subscribe@googlegroups.com](mailto:akaros%2Bsubscribe@googlegroups.com)
  41or visit the [group](http://groups.google.com/group/akaros).  Once you've
  42joined, send your messages to <mailto:akaros@googlegroups.com>.
  43
  44Alternatively, you can poke your head in #akaros on `irc.freenode.net`.  I'm
  45usually idling in there (alone), and if I'm at my computer, I'll respond.
  46
  47
  483. From Download to Hello World
  49----------------------------
  50I'll describe how to get x86 working.  RISCV is similar.
  51
  52To start off, make sure AKAROS_ROOT and AKAROS_XCC_ROOT are set in your
  53environment.  AKAROS_ROOT is the Akaros repo directory.  AKAROS_XCC_ROOT is a
  54directory of your choosing where the toolchain will be installed (more on that
  55in Section 3.1 below).
  56
  57The first step is to configure the kernel.  Targets like `config`,
  58`menuconfig`, and some of the other KBuild targets work.  Defconfig gives you a
  59default configuration.  For example, to config for 64-bit x86:
  60
  61`$ make ARCH=x86 defconfig`
  62
  63Alternatively, you can run menuconfig to customize what settings you want:
  64
  65`$ make ARCH=x86 menuconfig`
  66
  67For x86, you can choose between 32 and 64 bit when you `make menuconfig`.  This
  68selection must match your cross compiler `make` command.  The default is 64
  69bit.
  70
  71There are a lot of other settings when you `make config`, and you should browse
  72through to decide what you want to enable/disable.
  73
  74Most everyone wants KFS turned on (Filesystems --> KFS filesystem).  This is
  75the in-memory filesystem that the kernel uses.  The kernel build scripts will
  76look at the "KFS/Initramfs paths" string and take any of those directories and
  77add them to a CPIO archive that will eventually become the root filesystem when
  78Akaros runs. These settings are set by default when you do a `make defconfig`.
  79
  80There are also settings for `ext2`.  If you turn on `ext2` support, you need to
  81point to an `img` file that has been formatted with `ext2` and has files in it.
  82If you aren't messing with filesystems at all, feel free to ignore this.  It's
  83an in-memory filesystem, like KFS (linked to the end of the kernel), so you
  84won't gain much by using it for now.
  85
  86### 3.1 Cross compiler (and glibc)
  87The second step is to build the cross compiler, which lives in
  88`tools/compilers/gcc-glibc`
  89
  90`$ cd tools/compilers/gcc-glibc`
  91
  92In this directory, you first need to set up your Makelocal file.  There is a
  93template to work from.
  94
  95`$ cp Makelocal.template Makelocal`
  96
  97You need to set your `INSTDIRS` to some place where you want the cross compiler
  98installed.  I have a directory named `akaros-gcc-glibc` for this.
  99
 100Additionally, you must set the environment variable `$AKAROS_XCC_ROOT` to point
 101to the installation directory for your architecture.  For instance, my
 102AKAROS_XCC_ROOT is:
 103
 104`/home/brho/classes/akaros/akaros-gcc-glibc/install-x86_64-ucb-akaros-gcc/`
 105
 106You also need to add `bin` directories to your `PATH` where the cross compiler
 107will be installed.  This will vary based on your value for `INSTDIRS`.  For
 108instance, my path contains:
 109
 110`/home/brho/classes/akaros/akaros-gcc-glibc/install-x86_64-ucb-akaros-gcc/bin`
 111
 112You can also set up `MAKE_JOBS`, so you don't over or under load your system when
 113building.  I have a 2 core laptop, so I use `MAKE_JOBS := 3`
 114
 115At this point, you can build (for example):
 116
 117`$ make x86_64`
 118
 119This might take a while (10-20 minutes for me on a 2007 era laptop).
 120
 121Just to double check everything installed correctly, you should be able to run
 122`x86_64-ucb-akaros-gcc` from your shell.
 123
 124Now, you have a cross compiler ready, and you can start to build Akaros.
 125
 126### 3.2 Kernel
 127`cd` back into the repo root.
 128
 129Like the cross compiler, the kernel has its own `Makelocal`.
 130
 131`$ cp Makelocal.template Makelocal`
 132
 133This file is used to set up custom make targets that are not part of the
 134default `Makefile`, but fit nicely into your personal workflow. This file is
 135not under version control and can me made to contain just about anything.
 136
 137Now you're ready to build the kernel:
 138
 139`$ make`
 140
 141So the kernel built, but you can't do much with it, and you probably have no
 142programs.
 143
 144Notice that we didn't have to set the `ARCH` variable this time.  The make
 145system knows what architecture we are set up for and will always build for that
 146architecture until a new `ARCH` is selected (i.e. via `make ARCH=xxx defconfig`
 147etc.)
 148
 149### 3.3 Userspace
 150First, you'll need to build a few common applications and libraries:
 151
 152`$ make apps-install`
 153
 154Then you can build the tests and small utility programs:
 155
 156`$ make tests`
 157
 158You now have programs and libraries, and need to put them in KFS.  To do this,
 159we provide a `fill-kfs` make target.
 160
 161`$ make fill-kfs`
 162
 163The `fill-kfs` target copies your cross compiler's shared libraries and all
 164test binaries into the first "KFS/Initramfs path" you set during configuration
 165(or `kern/kfs/lib` if you just kept the default).
 166
 167Now that you've changed the contents of KFS's source, remake the kernel.  You
 168should see something like the following before the kernel links.  If you don't
 169see this, then you probably didn't actually fill KFS properly.
 170
 171```
 172        Building initramfs:
 173                Adding kern/kfs to initramfs...
 174```
 175
 176
 177### 3.4 Busybox
 178Busybox provides our shell and some basic utilities.  You almost certainly want
 179to build and install busybox.
 180
 181Userspace programs like busybox need to be compiled with the cross compiler and
 182then have their binaries copied to `kern/kfs/bin`.  Since most everyone wants
 183busybox and we have a few patches of our own, we have support for automatically
 184building and installing it to KFS.
 185
 186For the default build (`x86_64`):
 187
 188```
 189$ cd tools/apps/busybox
 190$ make [x86_64|riscv]
 191$ cd -
 192```
 193
 194And you should be set.  Check `kfs` to make sure everything installed.  You
 195should get sane results from:
 196
 197```
 198$ ls -l kern/kfs/bin | grep cat
 199lrwxrwxrwx 1 brho brho       7 Jan 23 09:19 cat -> busybox
 200```
 201
 202You can customize your busybox installation, including the install prefix, the
 203`.config` file, and `make` jobs.  Check out the makefile in
 204`tools/apps/busybox` for details.
 205
 206Now that you've changed KFS, don't forget to remake the kernel.
 207
 208### 3.5 Building and Loading a Virtual Machine Image
 209At this point, you probably have a runnable kernel with programs in KFS.  It
 210should be sitting at `obj/kern/akaros-kernel`.  When running in a VM, you can
 211either run the kernel directly from `qemu`, or put it in a virtual machine
 212image file.
 213
 214If you don't want to bother with the image, skip this section.  I tend to run
 215my images off an image file, since `qemu` acts more like hardware (as far as
 216multiboot goes).  The downside is the boot up is slower, especially if you have
 217a large kernel (>100MB).  It also takes some effort to set up the VM image.
 218
 219If you are still reading, you'll need an image file that looks like a hard disk
 220to boot `qemu` off of.  I put one similar to mine at:
 221<http://akaros.cs.berkeley.edu/files/hdd268mb.img>
 222
 223It's around 268MB (256MiB, or whatever).  If you want to make your own, check out
 224[Documentation/howtos/make-bootable-grub-hdd.txt](Documentation/howtos/make-bootable-grub-hdd.txt).
 225That's actually the original document I made back when I first figured it out
 226back in 2009, which was updated again in 2013.  In between, I wrote it up
 227online at
 228<http://www.omninerd.com/articles/Installing_GRUB_on_a_Hard_Disk_Image_File>,
 229which has some other tidbits in the comments.  Both methods still use `grub1`.
 230
 231Anyway, I put that img in `AKAROS-ROOT/mnt/`, and make a folder next to it:
 232`AKAROS-ROOT/mnt/hdd`.  `mnt/hdd` is the mount point where I mount `hdd.img`
 233(Note I don't call it `hdd64mb.img` on my dev machine).
 234
 235Personally, I always have `hdd.img` mounted.  Some of the other devs have make
 236targets that mount and umount it.  Whenever I reboot my development machine, I
 237run a script (as root) that mounts the image file and sets up a few things for
 238networking.  I put a script I use for this in `scripts/kvm-up.sh`.  You'll likely
 239want to copy it to the directory **above** the akaros root directory and edit it
 240accordingly. Feel free to comment out the networking stuff.  That's for using
 241networking in `qemu`.
 242
 243Now that your image file is mounted at `mnt/hdd`, you'll want to copy your
 244freshly built kernel to the root of the image.  I have a make target in my
 245makelocal for this, so that whenever I do a `make kvm`, it builds the kernel
 246and copies it to my `hdd.img`.
 247
 248I added edited versions of my KVM (and USB) make targets to the
 249`Makelocal.template`.  Uncomment the KVM one (at least).
 250
 251Incidentally, I also have the following in my `Makelocal`, so that `make` (and
 252`make all`) also make kvm:
 253
 254`all: kvm`
 255
 256Now, `make kvm`.  You should be able to see the new kernel in `mnt/hdd/` (do an
 257`ls -l` and check the timestamp).
 258
 259### 3.6 Running Qemu
 260Here is the command I use to run `qemu`/`kvm`.  It's evolved over the years,
 261and it will vary based on your linux distribution.  Don't run it just yet:
 262
 263```
 264$ qemu-system-x86_64 -s -enable-kvm -cpu kvm64 -smp 8 -m 4096 -nographic -monitor /dev/pts/3 -net nic,model=e1000 -net user,hostfwd=tcp::5555-:22 -net dump,file=/tmp/vm.pcap -drive file=mnt/hdd.img,index=0,media=disk,format=raw
 265```
 266
 267If you skipped making a virtual machine image or want to run the kernel
 268directly without emulating a disk, replace the "`-drive`" parameter (all the way
 269to `format=raw`) with "`-kernel obj/kern/akaros-kernel`".
 270
 271The `-monitor` is the qemu monitor, which is a CLI for qemu.  Pick a
 272tab/terminal/pty in Linux that you will only use for qemu monitoring, and enter
 273`tty'.  Whatever it tells you, put in place of `/dev/pts/3`.  I've been using
 274the same tab for about 4 years now.  In that tab, enter '`sleep 999999999`'.
 275Qemu will still access it, but you won't have to worry about bash trying to
 276handle your inputs.
 277
 278`-nographic` allows qemu to work in the terminal you run qemu from, instead of
 279spawning off a fake cpu crt/monitor.
 280
 281The command as written uses qemu's user networking.  It's emulated and a little
 282slow.  The example I have alo forwards port `5555` on the host to port `22` on
 283the guest.  Customize it according to your needs.
 284
 285Another option for networking is to set up a tun/tap device.  I use this on
 286some machines, and the kvm-up script has some commands to set it up.  It's
 287tricky, and may vary for your distribution.  If you do use the tun/tap
 288networking, replace the "`-net user`" section with:
 289
 290`-net tap,ifname=tap0,script=no`
 291
 292The "`-net dump`" option saves a pcap trace of the network traffic.  This is very
 293useful for debugging, but probably not needed for most people.
 294
 295Feel free to pick different values for the number of cpus and RAM (8 and 4096
 296in the example).
 297
 298Once you finally run it, you can stop the VM by entering '`q`' to the qemu
 299monitor (or just killing the process)..  Other help commands from the monitor
 300include '`info cpus`', '`info registers`', '`x`', and '`help`'.
 301
 302In more recent versions of qemu, `CTRL-C` will not get sent to the guest;
 303instead it will kill the VM.  If this gets annoying, you can remap "interrupt"
 304to something other than `CTRL-C` in the terminal where you run qemu:
 305
 306`$ stty intr ^]`
 307
 308To change it back:
 309
 310`$ stty intr ^c`
 311
 312
 313### 3.7 Running on Hardware
 314I have a few bootable USB sticks with grub set up to run Akaros.  The make usb
 315target (example in `Makelocal.template`) will copy freshly made kernels to your
 316USB device.  You'll need to adjust those paths according to your distro.  My
 317usb sticks are usually `/dev/sdc`, for instance (some odd USB device in the
 318last couple of years has taken over `/dev/sdb`.  Probably something to do with
 319`udev` changing over the years).
 320
 321Anyway, you'll need to mess around a bit to get that working.  Or I can `dd` one
 322for you (I have 4GB disks in my office that I use).  If you make your own, the
 323critical part is getting grub to pick the right device (from what I remember),
 324and its fairly similar to installing grub on any old hard drive (since it's
 325just a bloc device).  Much easier than a hard disk image file.
 326
 327
 328### 3.8 Hello World
 329So now you can run the kernel.  It's time to edit a program (or make your own).
 330In this, I'll go through my workflow for making changes.
 331
 332```
 333$ vi tests/hello.c
 334(edit, save)
 335
 336$ make tests
 337(new version in obj/tests/hello)
 338
 339$ make fill-kfs
 340(updates kfs)
 341
 342$ make
 343(rebuilds kernel with the new KFS)
 344
 345$ qemu...
 346
 347(following commands are in Akaros)
 348Shift-G (to get to the kernel monitor)
 349
 350ROS(Core 0)> bb (to run busybox)
 351
 352/ $ hello
 353(Should print your message)
 354```
 355
 356### 3.9 Other Dev Workflow Stuff
 357One thing to note is that while we use dynamic linking for `libc`, `parlib`
 358libraries are statically linked with applications.  In fact, nowadays **all**
 359Akaros programs need to be linked againt `parlib` (used to be that single-core
 360processes (SCPs) didn't need it).
 361
 362The makefiles won't notice if you change a file in `parlib` and then remake a
 363binary.  So if you edit `user/parlib/uthread.c` for example,
 364`tests/pthread_test` won't get rebuilt.  Here's what I do:
 365
 366```
 367$ vi user/parlib/uthread.c (make awesome change)
 368
 369$ touch tests/pthread_test.c ; make tests
 370```
 371
 372This will force the rebuild of `pthread_test`.  Older, untouched binaries (e.g.
 373`block_test`), won't get rebuilt.  I actually want this in some cases
 374(different versions of `parlib` when I'm running certain tests).  Anyway, just
 375pay attention to what you're building.  There's not much output in the console,
 376so you should be able to see what's going on all the time.  (unlike when
 377building `glibc`...).
 378
 379Oh, and don't forget to:
 380
 381`$ make fill-kfs`
 382
 383to make sure you run the new `pthread_test`.
 384
 385Additionally, when switching between 32 and 64 bit x86, `make objclean` before
 386filling KFS.  This is the easiest way to make sure you get the appropriate
 387libraries loaded in KFS.
 388
 389Early on as a dev, there are lots of times where you accidentally don't run the
 390right program (or kernel) and won't understand why your change isn't happening.
 391A few `printk("WTF\n")`'s later, you realize you didn't have the `hdd.img`
 392mounted, or you didn't fill KFS, or you didn't relink your binaries, or you
 393forgot to save all files in `vi1 (and not just the current buffer).  But after
 394doing a couple `hello worlds`, you're set.
 395
 396Alternatively, you could have a make target to run qemu, which also touches all
 397binaries (or otherwise enforces a rebuild), auto-fills KFS, remakes the kernel,
 398and mounts/copies/unmounts your `hdd.img`.
 399Personally, I like to keep track of what is going on under the hood, esp if I
 400want to do something a little differently (like with testing `ext2`, having
 401different versions of `parlib` with some binaries, or being picky about my
 402mount/umounts).
 403
 404
 4054. RISCV
 406---------------------
 407**TODO**.
 408
 409For now, you need a 64 bit distro to build the RISCV stuff, so I don't do it
 410very often.  I'll eventually sync up with Andrew and we'll get this part sorted
 411out.
 412
 413
 4145. Other Developer's Stuff
 415--------------------------
 416Nothing for now...
 417