2Barret Rhoden
   3Last updated: 2010-08-22
   5This document is an attempt to gather knowledge about how to work with the
   6glibc port.  This is based on my half-assed struggling with the port done by
   7kevin and andrew.
   9When to recompile (other than when changing libc)
  11Whenever you change a kernel header (something in kern/include/ros),
  12technically you should rebuild.  It isn't critical in all places, such as for
  13files that aren't included by glibc, or for files that haven't changed
  14something that glibc relies on.
  16After any recompile, don't forget to rebuild userspace apps (make userclean,
  17make install-libs) and to refresh your FS.
  19How to recompile glibc
  21Normally, make x86 (or whatever) works fairly well.  However, sometimes you
  22need to rebuild all of glibc.  This might be on a hunch, or to get rid of
  23things that might have compiled, but are failing to link and don't seem to
  24rebuild.  This happens when you change parts of glibc.
  26If you know what file(s) you changed and it doesn't seem like the changes are
  27being applied, you can copy the file from the -ros directory to the glibc-XXX
  28directory.  The glibc make process doesn't apply diffs or track the changes
  29from -ros to the glibc directory, so your recompile won't notice the changes.
  31If things are stil messed up, the next step is to remove the build directories
  32(i686-ros-glibc-*) and also the hidden files (.i686-ros-glibc*).  If you get
  33errors from make very early on about not finding some targets (such as
  34install-headers), you forgot to delete the hidden files.
  36Then next step up would be to remove the glibc-XXX folder (whatever the
  37version is).  Sometimes you aren't sure if old changes are sitting around in
  38this folder that aren't getting overwritten by the real source of our changes,
  39the -ros folder.  A lovely example of this is an old Makefile for the nss
  40subdir.  It was altered to remove some things, notably the files service,
  41which translates into libnss_files - which is needed by other things (nis,
  42etc).  To fix this, I had to notice an -ros version of the Makefile (after I
  43realized it was the problem), and that the Makefile was being included - it
  44was just faulty.
  46Finally, you can always make clean, but the overall XCC Makefile will clean
  47everything - including gcc and binutils.
  49Note that if you are making a trivial addition to a kernel header, you can get
  50away with just copying it to its location in the XCC install directory
  53The -ros folder
  55All changes to glibc must be made in the glibc-XXX-ros folder.  Any changes
  56here will be copied over to the real glibc folder when it is made.  If you
  57want to remove a file that was originally in the -ros folder, you need to
  58manually remove it from the glibc-XXX folder as well.  The updating scripts
  59will not remove a file.
  61The sysdeps folder
  63Most of your changes will go here.  Every system that has a glibc port should
  64have one of these folders, with its system-specific ports.  The files here are
  65normally files expected elsewhere in libc, and the glibc build system knows to
  66look in these places.
  68Simply dropping a file in here will not get it built.  It will for many files,
  69but some others that weren't expected will need to be manually added to a
  70Makefile.  If you notice some files not getting compiled, (drop a #error in
  71there), you'll need to add it to a Makefile.  In the main sysdeps/Makefile,
  72add the filename (minus .c) to the sysdeps var.  Look at sa_len for an
  73example.  Sometimes you'll need to be careful about adding it for some
  74subdirectories and not others (you probably only want to add a C file for one
  75subdir).  Check out Linux's port for help.
  77Sometimes you have files that you want to change outside of the sysdeps
  78folder.  These still go in the glibc-XXX-ros folder, but not in sysdeps.  The
  79main example we have right now is features.h.  At one point, I wanted to
  80change sys/socket.h.  Simply adding it to sysdeps/ros/sys/socket.h would not
  81change the main sys/socket.h, nor would the sysdep version get included first.
  82Putting it in the -ros/sys/ folder did it (though ultimately I didn't want the
  83change).  The point is, sysdeps doesn't mirror and override the main tree for
  84all files - it is behind some others in the search/include path.
  86Another situation requiring a change outside of the sysdeps directory was
  87sunrpc/netname.c.  I wanted to change the functions (stub out the ones that
  88used NSS).  Adding the sysdep worked, but it turns out that *both* the sysdep
  89netname.c and the original sunrpc/netname.c were being compiled.  The root
  90cause seems to be compat-netname.os.  There are make rules in sunrpc to
  91generate some compatibility routines.  Grep for rpc-compat-routines.os.  The
  92rule seems to ignore sysdeps and just use the normal C file - in this case
  97As a note, these 'subdirectories' are the "primary folders" (i.e. addons),
  98such as inet, ncsd, libio, whatever.  These are the root folders in glibc,
  99representing some major functionality.  They can be built with the
 100--enable-addons switch, which by default builds all of them.  Sort of!
 102To really get them to even be available for a port, you need to "include" them
 103in a certain manner.  There are two ways.  One is the Subdirs file in the
 104sysdeps/ros/ directory.  Putting a subdir name in here means glibc will try to
 105build it; it is available to be an addon.  Careful with these, since a lot of
 106the folders tend to need each other (like most all of the ones in unix/inet).
 108If you want a massive subsystem, such as "unix/inet" or "posix", you can add
 109it to the sysdeps/ros/Implies file.  You will get a bunch of these folders at
 110once, plus some other unspecified stuff (necessary for the overall system,
 111perhaps?).  If you add "unix/inet", you get more than just its Subdirs.
 113Also note that these subdirs can depend on each other, noted in the "Depends"
 114file.  Presumably this will cause them to get made...
 116Errno & TLS
 118errno is a macro that may point to different locations depending on where you
 119are in Glibc. errno_location is simply one of the options. 
 120During dynamic linking, the linker points errno to a different location, so it
 121is usable and can be referenced before TLS is fully set up in TLS_INIT_TP. 
 122Because errno is valid when the linker runs, regular syscalls can be made.
 124However for statically linked apps, several syscalls cannot use the ros_syscall
 125macro, because there is no valid errno set up. 
 127Unimplemented Stubs
 129There are a lot of things we haven't ported, but we have the stub functions so
 130that we can at least compile things against it.  When you compile (including
 131regular programs), you'll get warnings about this.  
 133Linux ASM bits
 135We've included some header files from Linux's kernel.  For now, we just need
 136something to keep glibc happy, even though ROS doesn't have any networking
 137syscalls.  Don't rely on this stuff too much.  These files, and other future
 138glibc compatibility kernel headers, are in kern/include/ros/glibc-asm.
 140Weak Symbols, start.c, and ros_syscall_blockon
 142For a long time, __ros_syscall_blockon was not getting overridden by the
 143strong symbol in parlib.  This means that glibc's syscalls were not blocking
 144uthreads properly.  Why did the weak symbols work for vcore_entry() and
 145vcore_event_init(), but not blockon?
 147Side note: we needed to force the linker to find out about vcore_entry and
 148vcore_event_init, via the -u tricks of commit f188983.  The linker is not
 149obligated to look in libraries to override a weak symbol.
 151The crux of the matter is that start.c is not linked with applications in the
 152same manner as the rest of glibc (notably all the I/O syscalls).  start.c will
 153get linked with the program at compile time, while libc can be linked
 154dynamically.  Because of this, start.c's weak symbols would get (correctly)
 155overridden by the strong symbols of libparlib.a.  But glibc would build
 156libc.so, and that would not get a chance to link against the binary (and
 157libparlib.a) until load time.  The weak symbols in libc get promoted to strong
 158symbols when it is done linking libc.so, and when it later is linked against
 159the program, there is no longer an opportunity to override the weak symbol.
 161Also note that rtld will never link with parlib, so any weak symbol in there
 162will never get overriden.  I briefly considered linking rtld with -lparlib,
 163but it blows up in a nasty way: parlib needs lots of libc, which rtld is not
 164built with (memset for example).
 166Anyway, the moral of the story is to be careful with your weak
 167symbols/aliases.  Only put them in start.c, or similar files.
 169Adding a Global Variable to the ABI
 171It's not enough to simply 'extern' a variable (or declare a function, which is
 172extern by default).  At some point, glibc will change those symbols from
 175You need to add an entry to the Versions file (sysdeps/ros/Versions).  Be sure
 176to indent with spaces, not tabs, or else glibc's scripts will not handle your
 179Putting them in the libc, glibc 2.0 section seems to work fine.
 181Tips, Questions, and Misc Notes
 183- Grep and find are your friend.
 184- Watch what Linux does.
 185- The kernel headers end up getting installed to the sys-include/ directory,
 186  while the regular glib headers go to the include directory.
 187- atomic_swap might have issues (warnings about signedness differences)
 188- It's not always clear which files in the -ros folder actually need to be
 189  there (presumably they all do), and in what manner they were modified and
 190  from which original copy.  Ideally, the ROS-specific changes to large files
 191  will be commented.
 192- the SHARED flag is a bit of a mess - things get compiled differently for
 193  shared vs static, and it can get complicated with start.c, tls.c, etc.
 194- What things in one file rely heavily on another file?  Are there non-obvious
 195  gotchas?  (yes, and no one documented them).
 196- Is the build failing without any clear error messages?  Scroll up a lot, and
 197  there may be messages farther up (like a hundred+ lines up).  I've had some
 198  gcc stage2 builds that fail with no obvious issue in the short term console
 199  output, but the real error is much higher.  Some aspect of the build system
 200  will continue on failures and only fail much later, after building other
 201  packages.
 202- Note that libstdc++ is a subpart of gcc, built during stage2, and has its
 203  own configure script and settings.
 205Ghetto Things (Feel free to fix them):
 207- ptsname: we needed to have the sysdep (was being looked for by the login
 208  subdir make process), but having the actual functions in it caused a link
 209  error (multiple declarations).  So we have an empty file...