akaros/GETTING_STARTED.md
<<
>>
Prefs
   1Getting Started with Akaros
   2==========================
   3Barret Rhoden
   4
   5Last thorough update: **2013-02-07**
   6
   7Contents
   8-----------
   9+ Overview
  10+ Need Help?
  11+ From Download to Hello World
  12  - Cross compiler (and glibc)
  13  - Kernel
  14  - Userspace
  15  - Busybox
  16  - Building and Loading a Virtual Machine Image
  17  - Running Qemu
  18  - Running on Hardware
  19  - Hello World
  20  - Other Dev Workflow Stuff
  21+ RISCV
  22+ Other Developer's Stuff
  23
  24
  251. Overview
  26---------------------
  27In this document, I outline what steps I go through to set up my development
  28environment.  Some devs use other setups, and they can put sections farther
  29down in this document describing their steps.
  30
  31Due to the nature of different workflows and different Linux distros, you'll
  32need to do some configuration of your environment.  Things won't magically work
  33out of the box.
  34
  35
  362. Need Help?
  37---------------------
  38First off, if you get stuck, email someone.  You can join our mailing list by
  39sending an email to
  40[akaros+subscribe@googlegroups.com](mailto:akaros%2Bsubscribe@googlegroups.com)
  41or visit the [group](http://groups.google.com/group/akaros).  Once you've
  42joined, send your messages to <mailto:akaros@googlegroups.com>.
  43
  44Alternatively, you can poke your head in #akaros on `irc.freenode.net`.  I'm
  45usually idling in there (alone), and if I'm at my computer, I'll respond.
  46
  47
  483. From Download to Hello World
  49----------------------------
  50I'll describe how to get x86 working.  RISCV is similar.
  51
  52To start off, make sure AKAROS_ROOT and AKAROS_XCC_ROOT are set in your
  53environment.  AKAROS_ROOT is the Akaros repo directory.  AKAROS_XCC_ROOT is a
  54directory of your choosing where the toolchain will be installed (more on that
  55in Section 3.1 below).
  56
  57I also suggest running `scripts/one-time-setup.sh`, once per `git clone`.  This
  58performs various checks and other setup.  Check it out for details.
  59
  60The first step is to configure the kernel.  Targets like `config`,
  61`menuconfig`, and some of the other KBuild targets work.  Defconfig gives you a
  62default configuration.  For example, to config for 64-bit x86:
  63
  64`$ make ARCH=x86 defconfig`
  65
  66Alternatively, you can run menuconfig to customize what settings you want:
  67
  68`$ make ARCH=x86 menuconfig`
  69
  70For x86, you can choose between 32 and 64 bit when you `make menuconfig`.  This
  71selection must match your cross compiler `make` command.  The default is 64
  72bit.
  73
  74There are a lot of other settings when you `make config`, and you should browse
  75through to decide what you want to enable/disable.
  76
  77Most everyone wants KFS turned on (Filesystems --> KFS filesystem).  This is
  78the in-memory filesystem that the kernel uses.  The kernel build scripts will
  79look at the "KFS/Initramfs paths" string and take any of those directories and
  80add them to a CPIO archive that will eventually become the root filesystem when
  81Akaros runs. These settings are set by default when you do a `make defconfig`.
  82
  83There are also settings for `ext2`.  If you turn on `ext2` support, you need to
  84point to an `img` file that has been formatted with `ext2` and has files in it.
  85If you aren't messing with filesystems at all, feel free to ignore this.  It's
  86an in-memory filesystem, like KFS (linked to the end of the kernel), so you
  87won't gain much by using it for now.
  88
  89### 3.1 Cross compiler (and glibc)
  90The second step is to build the cross compiler, which lives in
  91`tools/compilers/gcc-glibc`
  92
  93`$ cd tools/compilers/gcc-glibc`
  94
  95In this directory, you first need to set up your Makelocal file.  There is a
  96template to work from.
  97
  98`$ cp Makelocal.template Makelocal`
  99
 100You need to set your `INSTDIRS` to some place where you want the cross compiler
 101installed.  I have a directory named `akaros-gcc-glibc` for this.
 102
 103Additionally, you must set the environment variable `$AKAROS_XCC_ROOT` to point
 104to the installation directory for your architecture.  For instance, my
 105AKAROS_XCC_ROOT is:
 106
 107`/home/brho/classes/akaros/akaros-gcc-glibc/install-x86_64-ucb-akaros-gcc/`
 108
 109You also need to add `bin` directories to your `PATH` where the cross compiler
 110will be installed.  This will vary based on your value for `INSTDIRS`.  For
 111instance, my path contains:
 112
 113`/home/brho/classes/akaros/akaros-gcc-glibc/install-x86_64-ucb-akaros-gcc/bin`
 114
 115You can also set up `MAKE_JOBS`, so you don't over or under load your system
 116when building.  I have a 2 core laptop, so I use `MAKE_JOBS := 3`
 117
 118At this point, you can build (for example):
 119
 120`$ make x86_64`
 121
 122This might take a while (10-20 minutes for me on a 2007 era laptop).
 123
 124Just to double check everything installed correctly, you should be able to run
 125`x86_64-ucb-akaros-gcc` from your shell.
 126
 127Now, you have a cross compiler ready, and you can start to build Akaros.
 128
 129### 3.2 Kernel
 130`cd` back into the repo root.
 131
 132Like the cross compiler, the kernel has its own `Makelocal`.
 133
 134`$ cp Makelocal.template Makelocal`
 135
 136This file is used to set up custom make targets that are not part of the
 137default `Makefile`, but fit nicely into your personal workflow. This file is
 138not under version control and can me made to contain just about anything.
 139
 140Now you're ready to build the kernel:
 141
 142`$ make`
 143
 144So the kernel built, but you can't do much with it, and you probably have no
 145programs.
 146
 147Notice that we didn't have to set the `ARCH` variable this time.  The make
 148system knows what architecture we are set up for and will always build for that
 149architecture until a new `ARCH` is selected (i.e. via `make ARCH=xxx defconfig`
 150etc.)
 151
 152### 3.3 Userspace
 153First, you'll need to build a few common applications and libraries:
 154
 155`$ make apps-install`
 156
 157Then you can build the tests and small utility programs:
 158
 159`$ make tests`
 160
 161You now have programs and libraries, and need to put them in KFS.  To do this,
 162we provide a `fill-kfs` make target.
 163
 164`$ make fill-kfs`
 165
 166The `fill-kfs` target copies your cross compiler's shared libraries and all
 167test binaries into the first "KFS/Initramfs path" you set during configuration
 168(or `kern/kfs/lib` if you just kept the default).
 169
 170Now that you've changed the contents of KFS's source, remake the kernel.  You
 171should see something like the following before the kernel links.  If you don't
 172see this, then you probably didn't actually fill KFS properly.
 173
 174```
 175        Building initramfs:
 176        Adding kern/kfs to initramfs...
 177```
 178
 179
 180### 3.4 Busybox
 181Busybox provides our shell and some basic utilities.  You almost certainly want
 182to build and install busybox.
 183
 184Userspace programs like busybox need to be compiled with the cross compiler and
 185then have their binaries copied to `kern/kfs/bin`.  Since most everyone wants
 186busybox and we have a few patches of our own, we have support for automatically
 187building and installing it to KFS.
 188
 189For the default build (`x86_64`):
 190
 191```
 192$ cd tools/apps/busybox
 193$ make [x86_64|riscv]
 194$ cd -
 195```
 196
 197And you should be set.  Check `kfs` to make sure everything installed.  You
 198should get sane results from:
 199
 200```
 201$ ls -l kern/kfs/bin | grep cat
 202lrwxrwxrwx 1 brho brho       7 Jan 23 09:19 cat -> busybox
 203```
 204
 205You can customize your busybox installation, including the install prefix, the
 206`.config` file, and `make` jobs.  Check out the makefile in
 207`tools/apps/busybox` for details.
 208
 209Now that you've changed KFS, don't forget to remake the kernel.
 210
 211### 3.5 Building and Loading a Virtual Machine Image
 212At this point, you probably have a runnable kernel with programs in KFS.  It
 213should be sitting at `obj/kern/akaros-kernel`.  When running in a VM, you can
 214either run the kernel directly from `qemu`, or put it in a virtual machine
 215image file.
 216
 217If you don't want to bother with the image, skip this section.  I tend to run
 218my images off an image file, since `qemu` acts more like hardware (as far as
 219multiboot goes).  The downside is the boot up is slower, especially if you have
 220a large kernel (>100MB).  It also takes some effort to set up the VM image.
 221
 222If you are still reading, you'll need an image file that looks like a hard disk
 223to boot `qemu` off of.  I put one similar to mine at:
 224<http://akaros.cs.berkeley.edu/files/hdd268mb.img>
 225
 226It's around 268MB (256MiB, or whatever).  If you want to make your own, check out
 227[Documentation/howtos/make-bootable-grub-hdd.txt](Documentation/howtos/make-bootable-grub-hdd.txt).
 228That's actually the original document I made back when I first figured it out
 229back in 2009, which was updated again in 2013.  In between, I wrote it up
 230online at
 231<http://www.omninerd.com/articles/Installing_GRUB_on_a_Hard_Disk_Image_File>,
 232which has some other tidbits in the comments.  Both methods still use `grub1`.
 233
 234Anyway, I put that img in `AKAROS-ROOT/mnt/`, and make a folder next to it:
 235`AKAROS-ROOT/mnt/hdd`.  `mnt/hdd` is the mount point where I mount `hdd.img`
 236(Note I don't call it `hdd64mb.img` on my dev machine).
 237
 238Personally, I always have `hdd.img` mounted.  Some of the other devs have make
 239targets that mount and umount it.  Whenever I reboot my development machine, I
 240run a script (as root) that mounts the image file and sets up a few things for
 241networking.  I put a script I use for this in `scripts/kvm-up.sh`.  You'll
 242likely want to copy it to the directory **above** the akaros root directory and
 243edit it accordingly. Feel free to comment out the networking stuff.  That's for
 244using networking in `qemu`.
 245
 246Now that your image file is mounted at `mnt/hdd`, you'll want to copy your
 247freshly built kernel to the root of the image.  I have a make target in my
 248makelocal for this, so that whenever I do a `make kvm`, it builds the kernel
 249and copies it to my `hdd.img`.
 250
 251I added edited versions of my KVM (and USB) make targets to the
 252`Makelocal.template`.  Uncomment the KVM one (at least).
 253
 254Incidentally, I also have the following in my `Makelocal`, so that `make` (and
 255`make all`) also make kvm:
 256
 257`all: kvm`
 258
 259Now, `make kvm`.  You should be able to see the new kernel in `mnt/hdd/` (do an
 260`ls -l` and check the timestamp).
 261
 262### 3.6 Running Qemu
 263Here is the command I use to run `qemu`/`kvm`.  It's evolved over the years,
 264and it will vary based on your linux distribution.  Don't run it just yet:
 265
 266```
 267$ qemu-system-x86_64 -s -enable-kvm -cpu kvm64 -smp 8 -m 4096 -nographic -monitor /dev/pts/3 -net nic,model=e1000 -net user,hostfwd=tcp::5555-:22 -net dump,file=/tmp/vm.pcap -drive file=mnt/hdd.img,index=0,media=disk,format=raw
 268```
 269
 270If you skipped making a virtual machine image or want to run the kernel
 271directly without emulating a disk, replace the "`-drive`" parameter (all the way
 272to `format=raw`) with "`-kernel obj/kern/akaros-kernel`".
 273
 274The `-monitor` is the qemu monitor, which is a CLI for qemu.  Pick a
 275tab/terminal/pty in Linux that you will only use for qemu monitoring, and enter
 276'`tty`'.  Whatever it tells you, put in place of `/dev/pts/3`.  I've been using
 277the same tab for about 4 years now.  In that tab, enter '`sleep 999999999`'.
 278Qemu will still access it, but you won't have to worry about bash trying to
 279handle your inputs.
 280
 281`-nographic` allows qemu to work in the terminal you run qemu from, instead of
 282spawning off a fake cpu crt/monitor.
 283
 284The command as written uses qemu's user networking.  It's emulated and a little
 285slow.  The example I have alo forwards port `5555` on the host to port `22` on
 286the guest.  Customize it according to your needs.
 287
 288Another option for networking is to set up a tun/tap device.  I use this on
 289some machines, and the kvm-up script has some commands to set it up.  It's
 290tricky, and may vary for your distribution.  If you do use the tun/tap
 291networking, replace the "`-net user`" section with:
 292
 293`-net tap,ifname=tap0,script=no`
 294
 295The "`-net dump`" option saves a pcap trace of the network traffic.  This is
 296very useful for debugging, but probably not needed for most people.
 297
 298Feel free to pick different values for the number of cpus and RAM (8 and 4096
 299in the example).
 300
 301Once you finally run it, you can stop the VM by entering '`q`' to the qemu
 302monitor (or just killing the process)..  Other help commands from the monitor
 303include '`info cpus`', '`info registers`', '`x`', and '`help`'.
 304
 305In more recent versions of qemu, `CTRL-C` will not get sent to the guest;
 306instead it will kill the VM.  If this gets annoying, you can remap "interrupt"
 307to something other than `CTRL-C` in the terminal where you run qemu:
 308
 309`$ stty intr ^]`
 310
 311To change it back:
 312
 313`$ stty intr ^c`
 314
 315
 316### 3.7 Running on Hardware
 317I have a few bootable USB sticks with grub set up to run Akaros.  The make usb
 318target (example in `Makelocal.template`) will copy freshly made kernels to your
 319USB device.  You'll need to adjust those paths according to your distro.  My
 320usb sticks are usually `/dev/sdc`, for instance (some odd USB device in the
 321last couple of years has taken over `/dev/sdb`.  Probably something to do with
 322`udev` changing over the years).
 323
 324Anyway, you'll need to mess around a bit to get that working.  Or I can `dd` one
 325for you (I have 4GB disks in my office that I use).  If you make your own, the
 326critical part is getting grub to pick the right device (from what I remember),
 327and its fairly similar to installing grub on any old hard drive (since it's
 328just a bloc device).  Much easier than a hard disk image file.
 329
 330
 331### 3.8 Hello World
 332So now you can run the kernel.  It's time to edit a program (or make your own).
 333In this, I'll go through my workflow for making changes.
 334
 335```
 336$ vi tests/hello.c
 337(edit, save)
 338
 339$ make tests
 340(new version in obj/tests/hello)
 341
 342$ make fill-kfs
 343(updates kfs)
 344
 345$ make
 346(rebuilds kernel with the new KFS)
 347
 348$ qemu...
 349
 350(following commands are in Akaros)
 351Shift-G (to get to the kernel monitor)
 352
 353ROS(Core 0)> bb (to run busybox)
 354
 355/ $ hello
 356(Should print your message)
 357```
 358
 359### 3.9 Other Dev Workflow Stuff
 360One thing to note is that while we use dynamic linking for `libc`, `parlib`
 361libraries are statically linked with applications.  In fact, nowadays **all**
 362Akaros programs need to be linked againt `parlib` (used to be that single-core
 363processes (SCPs) didn't need it).
 364
 365The makefiles won't notice if you change a file in `parlib` and then remake a
 366binary.  So if you edit `user/parlib/uthread.c` for example,
 367`tests/pthread_test` won't get rebuilt.  Here's what I do:
 368
 369```
 370$ vi user/parlib/uthread.c (make awesome change)
 371
 372$ touch tests/pthread_test.c ; make tests
 373```
 374
 375This will force the rebuild of `pthread_test`.  Older, untouched binaries (e.g.
 376`block_test`), won't get rebuilt.  I actually want this in some cases
 377(different versions of `parlib` when I'm running certain tests).  Anyway, just
 378pay attention to what you're building.  There's not much output in the console,
 379so you should be able to see what's going on all the time.  (unlike when
 380building `glibc`...).
 381
 382Oh, and don't forget to:
 383
 384`$ make fill-kfs`
 385
 386to make sure you run the new `pthread_test`.
 387
 388Additionally, when switching between 32 and 64 bit x86, `make objclean` before
 389filling KFS.  This is the easiest way to make sure you get the appropriate
 390libraries loaded in KFS.
 391
 392Early on as a dev, there are lots of times where you accidentally don't run the
 393right program (or kernel) and won't understand why your change isn't happening.
 394A few `printk("WTF\n")`'s later, you realize you didn't have the `hdd.img`
 395mounted, or you didn't fill KFS, or you didn't relink your binaries, or you
 396forgot to save all files in `vi` (and not just the current buffer).  But after
 397doing a couple `hello worlds`, you're set.
 398
 399Alternatively, you could have a make target to run qemu, which also touches all
 400binaries (or otherwise enforces a rebuild), auto-fills KFS, remakes the kernel,
 401and mounts/copies/unmounts your `hdd.img`.
 402Personally, I like to keep track of what is going on under the hood, esp if I
 403want to do something a little differently (like with testing `ext2`, having
 404different versions of `parlib` with some binaries, or being picky about my
 405mount/umounts).
 406
 407
 4084. RISCV
 409---------------------
 410**TODO**.
 411
 412For now, you need a 64 bit distro to build the RISCV stuff, so I don't do it
 413very often.  I'll eventually sync up with Andrew and we'll get this part sorted
 414out.
 415
 416
 4175. Other Developer's Stuff
 418--------------------------
 419Nothing for now...
 420