akaros/GETTING_STARTED.md
<<
>>
Prefs
   1Getting Started with Akaros
   2==========================
   3Barret Rhoden
   4
   5Last thorough update: **2013-02-07**
   6
   7Contents
   8-----------
   9+ Overview
  10+ Need Help?
  11+ From Download to Hello World
  12  - Cross compiler (and glibc)
  13  - Kernel
  14  - Userspace
  15  - Building and Loading a Virtual Machine Image
  16  - Running Qemu
  17  - Running on Hardware
  18  - Hello World
  19  - Other Dev Workflow Stuff
  20+ RISCV
  21+ Other Developer's Stuff
  22
  23
  241. Overview
  25---------------------
  26In this document, I outline what steps I go through to set up my development
  27environment.  Some devs use other setups, and they can put sections farther
  28down in this document describing their steps.
  29
  30Due to the nature of different workflows and different Linux distros, you'll
  31need to do some configuration of your environment.  Things won't magically work
  32out of the box.
  33
  34
  352. Need Help?
  36---------------------
  37First off, if you get stuck, email someone.  You can join our mailing list by
  38sending an email to
  39[akaros+subscribe@googlegroups.com](mailto:akaros%2Bsubscribe@googlegroups.com)
  40or visit the [group](http://groups.google.com/group/akaros).  Once you've
  41joined, send your messages to <mailto:akaros@googlegroups.com>.
  42
  43Alternatively, you can poke your head in #akaros on `irc.freenode.net`.  I'm
  44usually idling in there (alone), and if I'm at my computer, I'll respond.
  45
  46
  473. From Download to Hello World
  48----------------------------
  49I'll describe how to get x86 working.  RISCV is similar.
  50
  51To start off, make sure AKAROS_ROOT and AKAROS_TOOLCHAINS are set in your
  52environment.  AKAROS_ROOT is the Akaros repo directory.  AKAROS_TOOLCHAINS is a
  53directory of your choosing where the toolchain will be installed (more on that
  54in Section 3.1 below).
  55
  56I also suggest running `scripts/one-time-setup.sh`, once per `git clone`.  This
  57performs various checks and other setup.  Check it out for details.
  58
  59The first step is to configure the kernel.  Targets like `config`,
  60`menuconfig`, and some of the other KBuild targets work.  Defconfig gives you a
  61default configuration.  For example, to config for 64-bit x86:
  62
  63`$ make ARCH=x86 defconfig`
  64
  65Alternatively, you can run menuconfig to customize what settings you want:
  66
  67`$ make ARCH=x86 menuconfig`
  68
  69For x86, you can choose between 32 and 64 bit when you `make menuconfig`.  This
  70selection must match your cross compiler `make` command.  The default is 64
  71bit.
  72
  73There are a lot of other settings when you `make config`, and you should browse
  74through to decide what you want to enable/disable.
  75
  76Most everyone wants KFS turned on (Filesystems --> KFS filesystem).  This is
  77the in-memory filesystem that the kernel uses.  The kernel build scripts will
  78look at the "KFS/Initramfs paths" string and take any of those directories and
  79add them to a CPIO archive that will eventually become the root filesystem when
  80Akaros runs. These settings are set by default when you do a `make defconfig`.
  81
  82There are also settings for `ext2`.  If you turn on `ext2` support, you need to
  83point to an `img` file that has been formatted with `ext2` and has files in it.
  84If you aren't messing with filesystems at all, feel free to ignore this.  It's
  85an in-memory filesystem, like KFS (linked to the end of the kernel), so you
  86won't gain much by using it for now.
  87
  88### 3.1 Cross compiler (and glibc)
  89The second step is to build the cross compiler, which lives in
  90`tools/compilers/gcc-glibc`
  91
  92`$ cd tools/compilers/gcc-glibc`
  93
  94You should already have set your `AKAROS_TOOLCHAINS` to some place where you
  95want the cross compiler installed.  I have a directory named `akaros-gcc-glibc`
  96for this. My bash environment variables are -
  97
  98```bash
  99export AKAROS_ROOT=/home/brho/classes/akaros/akaros-kernel
 100export AKAROS_TOOLCHAINS=/home/brho/classes/akaros/akaros-gcc-glibc
 101```
 102
 103You can also set up `MAKE_JOBS`, so you don't over or under load your system
 104when building.  I have a 2 core laptop, so I use `MAKE_JOBS := 3`
 105
 106At this point, you can build (for example):
 107
 108`$ make x86_64`
 109
 110This might take a while (10-20 minutes for me on a 2007 era laptop).
 111
 112Next you will need to add `bin` directories to your `PATH` where the cross
 113compiler was installed.  This will vary based on the architecture you built the
 114toolchain for.  Refer below for updating your PATH -
 115
 116* For riscv, add `$AKAROS_TOOLCHAINS/riscv-ucb-akaros-gcc/bin` to your PATH.
 117* For x86_64, add `$AKAROS_TOOLCHAINS/x86_64-ucb-akaros-gcc/bin` to your PATH.
 118* For x86_64 native, add `$AKAROS_TOOLCHAINS/x86_64-ucb-akaros-gcc-native/bin`
 119to your PATH.
 120
 121Just to double check everything installed correctly, you should be able to run
 122`x86_64-ucb-akaros-gcc` from your shell.
 123
 124Now, you have a cross compiler ready, and you can start to build Akaros.
 125
 126### 3.2 Kernel
 127`cd` back into the repo root.
 128
 129Like the cross compiler, the kernel has its own `Makelocal`.
 130
 131`$ cp Makelocal.template Makelocal`
 132
 133This file is used to set up custom make targets that are not part of the
 134default `Makefile`, but fit nicely into your personal workflow. This file is
 135not under version control and can be made to contain just about anything.
 136
 137Now you're ready to build the kernel:
 138
 139`$ make`
 140
 141So the kernel built, but you can't do much with it, and you probably have no
 142programs.
 143
 144Notice that we didn't have to set the `ARCH` variable this time.  The make
 145system knows what architecture we are set up for and will always build for that
 146architecture until a new `ARCH` is selected (i.e. via `make ARCH=xxx defconfig`
 147etc.)
 148
 149### 3.3 Userspace
 150First, you'll need to build a few common applications and libraries:
 151
 152`$ make apps-install`
 153
 154Then you can build the tests and small utility programs:
 155
 156`$ make tests`
 157
 158You now have programs and libraries, and need to put them in KFS.  To do this,
 159we provide a `fill-kfs` make target.
 160
 161`$ make fill-kfs`
 162
 163The `fill-kfs` target copies your cross compiler's shared libraries and all
 164test binaries into the first "KFS/Initramfs path" you set during configuration
 165(or `kern/kfs/lib` if you just kept the default).
 166
 167Now that you've changed the contents of KFS's source, remake the kernel.  You
 168should see something like the following before the kernel links.  If you don't
 169see this, then you probably didn't actually fill KFS properly.
 170
 171```
 172        Building initramfs:
 173        Adding kern/kfs to initramfs...
 174```
 175
 176### 3.4 Building and Loading a Virtual Machine Image
 177At this point, you probably have a runnable kernel with programs in KFS.  It
 178should be sitting at `obj/kern/akaros-kernel`.  When running in a VM, you can
 179either run the kernel directly from `qemu`, or put it in a virtual machine
 180image file.
 181
 182If you don't want to bother with the image, skip this section.  I tend to run
 183my images off an image file, since `qemu` acts more like hardware (as far as
 184multiboot goes).  The downside is the boot up is slower, especially if you have
 185a large kernel (>100MB).  It also takes some effort to set up the VM image.
 186
 187If you are still reading, you'll need an image file that looks like a hard disk
 188to boot `qemu` off of.  I put one similar to mine at:
 189<http://akaros.cs.berkeley.edu/files/hdd268mb.img>
 190
 191It's around 268MB (256MiB, or whatever).  If you want to make your own, check out
 192[Documentation/howtos/make-bootable-grub-hdd.txt](Documentation/howtos/make-bootable-grub-hdd.txt).
 193That's actually the original document I made back when I first figured it out
 194back in 2009, which was updated again in 2013.  In between, I wrote it up
 195online at
 196<http://www.omninerd.com/articles/Installing_GRUB_on_a_Hard_Disk_Image_File>,
 197which has some other tidbits in the comments.  Both methods still use `grub1`.
 198
 199Anyway, I put that img in `AKAROS-ROOT/mnt/`, and make a folder next to it:
 200`AKAROS-ROOT/mnt/hdd`.  `mnt/hdd` is the mount point where I mount `hdd.img`
 201(Note I don't call it `hdd64mb.img` on my dev machine).
 202
 203Personally, I always have `hdd.img` mounted.  Some of the other devs have make
 204targets that mount and umount it.  Whenever I reboot my development machine, I
 205run a script (as root) that mounts the image file and sets up a few things for
 206networking.  I put a script I use for this in `scripts/kvm-up.sh`.  You'll
 207likely want to copy it to the directory **above** the akaros root directory and
 208edit it accordingly. Feel free to comment out the networking stuff.  That's for
 209using networking in `qemu`.
 210
 211Now that your image file is mounted at `mnt/hdd`, you'll want to copy your
 212freshly built kernel to the root of the image.  I have a make target in my
 213makelocal for this, so that whenever I do a `make kvm`, it builds the kernel
 214and copies it to my `hdd.img`.
 215
 216I added edited versions of my KVM (and USB) make targets to the
 217`Makelocal.template`.  Uncomment the KVM one (at least).
 218
 219Incidentally, I also have the following in my `Makelocal`, so that `make` (and
 220`make all`) also make kvm:
 221
 222`all: kvm`
 223
 224Now, `make kvm`.  You should be able to see the new kernel in `mnt/hdd/` (do an
 225`ls -l` and check the timestamp).
 226
 227### 3.5 Running Qemu
 228Here is the command I use to run `qemu`/`kvm`.  It's evolved over the years,
 229and it will vary based on your linux distribution.  Don't run it just yet:
 230
 231```
 232$ qemu-system-x86_64 -s -enable-kvm -cpu kvm64 -smp 8 -m 4096 -nographic -monitor /dev/pts/3 -net nic,model=e1000 -net user,hostfwd=tcp::5555-:22 -net dump,file=/tmp/vm.pcap -drive file=mnt/hdd.img,index=0,media=disk,format=raw
 233```
 234
 235If you skipped making a virtual machine image or want to run the kernel
 236directly without emulating a disk, replace the "`-drive`" parameter (all the way
 237to `format=raw`) with "`-kernel obj/kern/akaros-kernel`".
 238
 239The `-monitor` is the qemu monitor, which is a CLI for qemu.  Pick a
 240tab/terminal/pty in Linux that you will only use for qemu monitoring, and enter
 241'`tty`'.  Whatever it tells you, put in place of `/dev/pts/3`.  I've been using
 242the same tab for about 4 years now.  In that tab, enter '`sleep 999999999`'.
 243Qemu will still access it, but you won't have to worry about bash trying to
 244handle your inputs.
 245
 246`-nographic` allows qemu to work in the terminal you run qemu from, instead of
 247spawning off a fake cpu crt/monitor.
 248
 249The command as written uses qemu's user networking.  It's emulated and a little
 250slow.  The example I have alo forwards port `5555` on the host to port `22` on
 251the guest.  Customize it according to your needs.
 252
 253Another option for networking is to set up a tun/tap device.  I use this on
 254some machines, and the kvm-up script has some commands to set it up.  It's
 255tricky, and may vary for your distribution.  If you do use the tun/tap
 256networking, replace the "`-net user`" section with:
 257
 258`-net tap,ifname=tap0,script=no`
 259
 260The "`-net dump`" option saves a pcap trace of the network traffic.  This is
 261very useful for debugging, but probably not needed for most people.
 262
 263Feel free to pick different values for the number of cpus and RAM (8 and 4096
 264in the example).
 265
 266Once you finally run it, you can stop the VM by entering '`q`' to the qemu
 267monitor (or just killing the process)..  Other help commands from the monitor
 268include '`info cpus`', '`info registers`', '`x`', and '`help`'.
 269
 270In more recent versions of qemu, `CTRL-C` will not get sent to the guest;
 271instead it will kill the VM.  If this gets annoying, you can remap "interrupt"
 272to something other than `CTRL-C` in the terminal where you run qemu:
 273
 274`$ stty intr ^]`
 275
 276To change it back:
 277
 278`$ stty intr ^c`
 279
 280
 281### 3.6 Running on Hardware
 282I have a few bootable USB sticks with grub set up to run Akaros.  The make usb
 283target (example in `Makelocal.template`) will copy freshly made kernels to your
 284USB device.  You'll need to adjust those paths according to your distro.  My
 285usb sticks are usually `/dev/sdc`, for instance (some odd USB device in the
 286last couple of years has taken over `/dev/sdb`.  Probably something to do with
 287`udev` changing over the years).
 288
 289Anyway, you'll need to mess around a bit to get that working.  Or I can `dd` one
 290for you (I have 4GB disks in my office that I use).  If you make your own, the
 291critical part is getting grub to pick the right device (from what I remember),
 292and its fairly similar to installing grub on any old hard drive (since it's
 293just a bloc device).  Much easier than a hard disk image file.
 294
 295
 296### 3.7 Hello World
 297So now you can run the kernel.  It's time to edit a program (or make your own).
 298In this, I'll go through my workflow for making changes.
 299
 300```
 301$ vi tests/hello.c
 302(edit, save)
 303
 304$ make tests
 305(new version in obj/tests/hello)
 306
 307$ make fill-kfs
 308(updates kfs)
 309
 310$ make
 311(rebuilds kernel with the new KFS)
 312
 313$ qemu...
 314
 315(following commands are in Akaros)
 316Shift-G (to get to the kernel monitor)
 317
 318ROS(Core 0)> bb (to run busybox)
 319
 320/ $ hello
 321(Should print your message)
 322```
 323
 324### 3.8 Other Dev Workflow Stuff
 325One thing to note is that while we use dynamic linking for `libc`, `parlib`
 326libraries are statically linked with applications.  In fact, nowadays **all**
 327Akaros programs need to be linked againt `parlib` (used to be that single-core
 328processes (SCPs) didn't need it).
 329
 330The makefiles won't notice if you change a file in `parlib` and then remake a
 331binary.  So if you edit `user/parlib/uthread.c` for example,
 332`tests/pthread_test` won't get rebuilt.  Here's what I do:
 333
 334```
 335$ vi user/parlib/uthread.c (make awesome change)
 336
 337$ touch tests/pthread_test.c ; make tests
 338```
 339
 340This will force the rebuild of `pthread_test`.  Older, untouched binaries (e.g.
 341`block_test`), won't get rebuilt.  I actually want this in some cases
 342(different versions of `parlib` when I'm running certain tests).  Anyway, just
 343pay attention to what you're building.  There's not much output in the console,
 344so you should be able to see what's going on all the time.  (unlike when
 345building `glibc`...).
 346
 347Oh, and don't forget to:
 348
 349`$ make fill-kfs`
 350
 351to make sure you run the new `pthread_test`.
 352
 353Additionally, when switching between 32 and 64 bit x86, `make objclean` before
 354filling KFS.  This is the easiest way to make sure you get the appropriate
 355libraries loaded in KFS.
 356
 357Early on as a dev, there are lots of times where you accidentally don't run the
 358right program (or kernel) and won't understand why your change isn't happening.
 359A few `printk("WTF\n")`'s later, you realize you didn't have the `hdd.img`
 360mounted, or you didn't fill KFS, or you didn't relink your binaries, or you
 361forgot to save all files in `vi` (and not just the current buffer).  But after
 362doing a couple `hello worlds`, you're set.
 363
 364Alternatively, you could have a make target to run qemu, which also touches all
 365binaries (or otherwise enforces a rebuild), auto-fills KFS, remakes the kernel,
 366and mounts/copies/unmounts your `hdd.img`.
 367Personally, I like to keep track of what is going on under the hood, esp if I
 368want to do something a little differently (like with testing `ext2`, having
 369different versions of `parlib` with some binaries, or being picky about my
 370mount/umounts).
 371
 372
 3734. RISCV
 374---------------------
 375**TODO**.
 376
 377For now, you need a 64 bit distro to build the RISCV stuff, so I don't do it
 378very often.  I'll eventually sync up with Andrew and we'll get this part sorted
 379out.
 380
 381
 3825. Other Developer's Stuff
 383--------------------------
 384Nothing for now...
 385