Note that the content has moved to a better home on github, https://github.com/anaaktge/cheribuild/wiki !
We hope to move into the main CheriBuild repository soon!
Below is the original content for posterity.
Hello! Below is my attempt to condense what a new contributor might need to know about CheriBSD, standing up applications on CheriBSD, and to some extent the broader CHERI (Capability Hardware Enhanced RISC Instructions) project, and its community into something reasonably digestible. It is based on my own experience, far from exhaustive, and at the rate things are moving, likely to be out of date before the metaphorical ink is dry. If you are interested in the hardware aspect, please get in touch with the team.
Still, I hope it will help you understand a bit more about how the project operates and help you get oriented and comfortably settled into the team. We would like to have an expanded, diverse, and welcoming contributor community. One of the greatest strengths of this project is its people. This has been the most supportive and approachable OSS (Open Source Software) community I have ever worked with, and I hope your experience will mirror mine.
I hope eventually to get things like this onto a wiki, with regular maintenance; for now something is better than nothing.
Assumptions About You
I assume that you’re already somewhat familiar with the CHERI project. If not, now might be a good time to at least skim through cheri-cpu.org. (For the enthusiastic reader, there are links to additional content later.)
I assume you are reasonably technical, interested in security, software, and/or computer architecture. I assume you largely know how to program and are at least familiar with the C language and pointers. Perhaps you are a motivated student in an undergraduate computer science program, or a software engineer looking for an interesting OSS project or considering work with the upcoming Arm Morello Board , or a graduate student looking for an interesting and novel research substrate.
If you feel you are not quite at that level, it is certainly still possible for you to contribute. Despite its decade of history, CHERI remains a young project and so has room for everything from documentation through small prototypes of novel system features through porting applications or even entire operating systems. Please do not look down on or underestimate the value of noncode contributions. Tests and documentation are always in need of more attention and are often more valuable than new feature contributions over a project lifetime. Tutorials would also be extremely helpful at this juncture.
Why contribute now?
The CHERI project began in 2010, and has been a small team of researchers, most of which are housed in a single building. Typically when onboarding a new contributor, someone either flies out to them or they travel to that building.
That model isn’t terribly scalable, and with the announcement of real, physical CHERI hardware landing in 2021, the Arm Morello Board , there is a growing interest, and an increasing number of engineers and companies who want or need to contribute to the project. The UK government will also be funding UK universities and companies to perform research and build demonstrators/prototypes around CHERI as part of the Digital Security by Design initiative.
What are capabilities anyway?
Capabilities as a design concept are relatively old and relatively intuitive: something should only be able to interact with resources for which it has (unforgeable) rights. Capabilities can be viewed as a way to implement the principles of least privilege and privilege separation. They have been implemented in hardware, software, and a combination of the two, at least in a research setting. Wikipedia’s Capability-based security article has a good list of capability supporting systems that managed to escape the lab. Traditionally, cost of re-development/re-implementation or performance has kept capability systems out of wider use.
Do they replace FreeBSD Capsicum capabilities?
They do not. Capsicum capabilities in FreeBSD (and by extension CheriBSD) is analogous to seccomp on Linux. It’s used at a per process level in user space to restrict system calls. Nor are they related to the withdrawn POSIX capabilities.
CHERI by contrast, is an architectural feature that extends existing architectures which can be used for things like low level memory safety. CHERI capabilities are most often used like fat pointers carrying additional benefits. As a general rule, the closer you are tied to the hardware, ie compiler, language runtime, or operating system, the more you care about the CHERI features.
Ideally, a security conscious application would make use of both OS-based sandboxing and CHERI protections, although it may not be aware of the CHERI features it uses through a trusted computing base (like a compiler).
Where are capabilities used in CheriBSD?
CheriBSD is a CHERI-enabled adaptation of the FreeBSD operating system. CheriBSD tries to stay up to date with FreeBSD plus or minus a few weeks. At the time of this writing, userland, the level at which normal applications and users operate, runs in pure capability mode, where old school, unprotected C style pointers are not allowed. The kernel itself however, is still not fully ported over to using capabilities within itself. There is ongoing effort on a pure capability kernel on a branch. It’s one the many areas where we’d love some help.
Is CheriBSD the same as CheriOS ?
CheriBSD is not the same thing as cheriOS. CheriBSD is a CHERI-fied version of FreeBSD, which was designed long before CHERI, and whose internals may not all be able to benefit from all of CHERI’s features. For example, employing CHERI memory safety seems to work well with the UNIX design and implementation, but using CHERI-based compartmentalization requires greater rethinking of OS software abstractions. In contrast, CheriOS is a new microkernel built from scratch with CHERI capability hardware in mind. I have not worked with it, however it’s a very intriguing design space. If that interests you, please see the CheriOS github page and reach out on the dedicated channel on the CHERI CPU slack.
Where can I contribute to CheriBSD?
I have a few suggestions; however, don’t feel limited by them. The team is bursting with ideas and they’re open to hearing more.
Some of the eternal suggestions include creating and improving documentation, and expanding test coverage. If you feel your C language mastery is weak, test writing might be a really great way to build that skill up and contribute at the same time.
Converting popular utilities, libraries, and even full-scale applications to use capabilities would certainly be handy. The first step is to ensure that they work well with CHERI C/C++, which is often straightforward and consists of helping older code better differentiate its integer types from capability types. There are some subtleties, and the CHERI developers will be happy to help out. There is a technical report documenting CHERI C/C++.
Sounds awesome! How do I communicate with the community?
I’m so glad you asked! Please join us on the mailing list, slack, and github. The mailing list can be found at https://lists.cam.ac.uk/mailman/listinfo/cl-cheri-discuss and if you are not from Cambridge, please ignore the scary red box and treat it as a normal mailman list. The slack is at cheri-cpu.slack.com and please feel free to explore the channel list. The CHERI main github page is a bit overwhelming with 120+ repositories. The two you’ll probably want to start with are the CheriBSD github and Cheribuild github.
Where can I find out more?
In the immortal words of Lego Batman, “prepare yourself for some….reading”! The whole list of publications can be found on CHERI Publications page. I don’t recommend you start there. Academic papers represent truth at time of publication, and may no longer be current.
I’ve taken to renaming any CHERI related pdf I download, because the names are not descriptive for end users.
I recommend you go in the approximate following order, using the Architecture document as a reference as you read the other things.
- Warmup, video : Dr. Robert Watson presenting CHERI
- Warmup, text: CHERI Frequently Asked Questions (FAQ)
- Medium: An Introduction to CHERI
- Medium: CHERI C/C++ Programming Guide (CCPG’s github)
- Hard: CHERI Instruction-Set Architecture (Version 7), for reference
- Bonus: CHERI: Notes on the Meltdown and Spectre Attacks for the inevitable questions about hardware attacks
- Bonus: CheriABI: Enforcing Valid Pointer Provenance and Minimizing Pointer Privilege in the POSIX C Run-time Environment (2019)
- Bonus: Cornucopia: Temporal Safety for CHERI Heaps (2020)
What tools would be good to know beforehand?
QEMU is a processor emulator, sometimes classified as a userland hypervisor. Whichever way you slice it, there is a forked version that supports CHERI-MIPS, CHERI-RISC-V, and Morello. The CHERI-RISC-V is the principal focus for development, but Qemu CHERI-MIPS was the stable option until earlier this year.
Note that I was using MIPS, however new users should now start with RISCV invocations. I have both listed where applicable.
- FreeBSD Basics
CheriBSD is based on FreeBSD. Much of the team has extensive FreeBSD experience. I don’t however, and definitely wandered in a few weeds as a result. If you would like to learn more about FreeBSD and how the kernel is structured, there is the FreeBSD Handbook. If you are more of a textbook learner, or would like a more polished survey of the kernel than the Handbook, The Design and Implementation of the FreeBSD Operating System 2nd ed (ACM link, amz link ) was recommended to me by a long time FreeBSD committer. Yes, one of the authors is part of CHERI and worked on Capsicum.
- C Language
CheriBSD is primarily written in C and the principal CHERI targets are TCBs (Trusted Computing Bases), which are generally written in C or in something that secretly uses C. As a result, you’ll probably have a more pleasant time if you are conversant with C, and pointers in particular. I tend to have my trusty K&R C book within reach whenever I revisit C. You may want your C reference of choice handy as you read the CHERI C/C++ Programming Guide. I understand that C is no longer taught as often and C fluency is much less common now, so if this a weak area for you, please be patient with yourself.
If you’re in the mood for a spot of UNIX & C history, consider reading The UNIX time-sharing system, by Dennis M. Ritchie & Ken Thompson (1974) . It may help you understand FreeBSD and C a little better.
- Command Line Interface (CLI)
Cheribuild, qemu, and various other bits largely live and run out of your terminal. The build & install process largely assumes familiarity with the CLI. Familiarity with your package manager might also save you some pain and troubleshooting later on.
- Git & Github
As I mentioned earlier, the CHERI main github page is a bit overwhelming. However, the documentation on the build system is very good, Alex has done a stellar job, and can be found on Cheribuild github. If you are interested in CheriBSD, then of course you should also check out the CheriBSD github.
How do I set this up? Starting with Cheribuild
Cheribuild is a nontrivial feat of engineering in its own right and it’ll be your best friend for getting started. The documentation is excellent and kept up to date. Alex works very hard on it and it shows. The overall set up is to install some requirements, set up the directory structure, clone a repo, and let Cheribuild work perhaps while you do some light CHERIreading. Remember that ‘-f’ is your friend if you tend to wander away from your terminal during installations.
My systems are Ubuntu, I understand this works on FreeBSD and OSX, however I have not tried it and cannot speak to the installation process.
Be aware that when you re-start Qemu and rebuild, it will prompt you, but likely overwrite the CheriBSD disk image and all the changes you might have made to it. That’s intentional, but may not suit your development style. Pay careful attention to the Cheribuild documentation!
I did not know about it when I was setting up, but there is a –force option to automatically suppress the y/n prompts. I recommend you try it out for installation. If you run into issues, please file a bug. We’d appreciate it.
Output from my terminal:
a@a:~$ sudo apt-get install libtool pkg-config clang bison cmake ninja-build samba flex texinfo libarchive-dev libglib2.0-dev libpixman-1-dev libbz2-dev libarchive-tools
a@a:~$ mkdir ~/cheri
a@a:~$ git clone https://github.com/CTSRD-CHERI/cheribuild ~/cheri/cheribuild
Can I set up it on Windows?
Well, no not really. Your best bet is to run the Qemu setup in a VM. The team thinks Cygwin or WSL might work, maybe, but they have never tried it and no one has any plans to either. Perhaps a place to contribute if you are a Windows programmer?
How do I start/stop Qemu CheriBSD?
risv (what you should use):
a@a:~$ ~/cheri/cheribuild/cheribuild.py qemu llvm cheribsd-riscv64-purecap disk-image-riscv64-purecap run-riscv64-purecap
mips (what I used):
a@a:~$ ~/cheri/cheribuild/cheribuild.py qemu llvm cheribsd-mips64-purecap disk-image-mips64-purecap run-mips64-purecap
(Assuming you have installed Cheribuild from the above section, this will build the development environment and then start the mips version, not the RISC-V one)
Default account is ‘root’ with no password. I’m told this is a tradition from the FreeBSD live boot CD, although I’ve never seen one of these mythical things in person. 😀 I generally run ‘uname -mrs’ after starting CheriBSD for a sanity check, though that’s just me.
By the way, the cleanest way to exit the whole thing is to run ‘shutdown -p now’ at the CheriBSD terminal prompt. Be advised that it may take a surprisingly long amount of time to shutdown and exit. That’s normal.
If you don’t care about ruining the disk image or keeping your install stable, i.e. you’re going to rebuild it anyway, “Ctrl-A x” will kill QEMU. I generally don’t do this, however most of the team does.
I’ve got it running!
Yay! Congratulations! Do a happy dance, then poke around a bit and explore!
How do I test stuff in CheriBSD ?
Cheribuild to the rescue! There’s a ‘–test’ argument to run the existing tests. It’s not as extensive as it could be, but it does exist. For new work, there is still a lot of manual testing by the developer and the reviewer. At this time, there is no formal QA verification, although Alex has been working hard to improve the CI (Continuous Integration) and run Jenkins on master.
Your test invocation is:
a@a:~$ ~/cheri/cheribuild/cheribuild.py qemu llvm cheribsd-riscv64-purecap disk-image-riscv64-purecap run-riscv64-purecap --test
My test invocation was:
a@a:~$ ~/cheri/cheribuild/cheribuild.py qemu llvm cheribsd-mips64-purecap disk-image-mips64-purecap run-mips64-purecap --test
New stuff lands very frequently, so you may find a clean master breaks. Don’t panic. We’re getting better, and most of the main repositories stage changes through a “dev” branch, though it still happens. If you run into issues with new changes, Speak up! Please!Related to testing, Alfredo has syzkaller, a kernel fuzzer, running regularly. The ISA(Instruction Set Architecture) and other parts have testing and verification procedures of their own, however my focus is primarily on CheriBSD.
How do I debug stuff in CheriBSD?
There are 3 flavors of GDB to consider: inside-inside, inside-outside, outside-outside. Inside-inside GDB is cross-compiled, hybrid aware, running in CheriBSD to debug CHERI userspace. inside-outside GDB is a CHERI aware host executable used to debug the CheriBSD kernel with a debug server, such as the one provided by QEMU. Outside-outside GDB is a host process debugging a host executable like QEMU itself. Outside-outside is the least used and most likely not one you’ll reach for often. For the CHERI cases, you’ll want to check out the latest debugging over at Building GDB
For working in the CheriBSD kernel, remote gdb debugging is the way to go. You’ll be looking for the extra flag –wait-for-debugger, wait on cheribuild, then follow the directions it prints for the type of debugging you’d like to do. –wait-for-debugger is most useful for kernel debugging rather than userland debugging.
For me, this tends to be fragile, and you’ll need to be _very_ close attention to the directions that cheribuild spits out for attaching the debugger to the kernel or userland. Very, very close attention. I personally use two terminals side by side when setting it up.
For working *on* CheriBSD, in userspace, gdb is present, but not built by default. You’ll need to add an extra flag to the starting incantation. Please check the Cheribuild documentation for the latest details. Ultimately you’ll be interested in running gdb-run.sh, which will start a program in GDB and print a backtrace once it stops, which can save a lot of your sanity.
Your (kernel) debugger invocation is:
a@a:~$ ./cheri/cheribuild/cheribuild.py qemu llvm cheribsd-riscv64-purecap disk-image-riscv64-purecap run-riscv64-purecap gdb-native --wait-for-debugger
My (kernel) debugger invocation was:
a@a:~$ ./cheri/cheribuild/cheribuild.py qemu llvm cheribsd-mips64-purecap disk-image-mips64-purecap run-mips64-purecap gdb-native --wait-for-debugger
How do I find a Good First BugTM ?
I’m afraid that is a luxury that this project does not yet have. However, what we lack in organization, we make up for in engagement. Ask in slack or on the mailing list with your interest and skills, and we’ll try to match you as best we can.
If you were interested in compiling and maintaining such a list as a project, that would be a nice gift to future contributors.
How do I find a project or task generally on CHERI?
As of this writing, it’s the same friendly process as above. On the bright side, there is a high level of interpersonal interaction. There’s a good chance the whole team will know your name by the time your contribution lands.
How do I find a Mentor?
We don’t have a formal mentorship program at this stage (also a luxury of bigger projects). That doesn’t mean folks aren’t interested in helping new contributors, yet it does mean that you’ll need to be a bit more proactive about asking for help. If it helps you, you can always ask someone to mentor you explicitly. I recommend slack over the mailing list for seeking a mentor.
What are the development style requirements?
The style guide for CheriBSD is the venerable, cryptic style(9), as it is for FreeBSD. There is a top level clang format file in the source tree and there exist plugins for Vim & Emacs. If you prefer VScode, as far as I can tell you’re out of luck at this time.
The other style note worth mentioning, is that since CHERI is an acronym, it is usually capitalized. CheriBSD and Cheribuild are amongst the most prominent exceptions.
How do I ask for a code review?
There is no formal process for asking for review or who should be assigned a given review. Ideally, whoever you’re working on changes with will be able to advise you or do the review. The github suggestions are generally good, and those folks can redirect you if the algorithm makes an incorrect recommendation. If all else fails, please ask in slack and/or make a Pull Request(PR). Mark it with ‘work in process’ and with a description as informative as possible to help things move smoothly.
I think I found an issue, but I’m not sure. What should I do?
If you think you found something amiss, you probably did. Please let us know either by filing an issue on the appropriate github repo, or asking in slack if you aren’t sure which repo it belongs in. Thanks!
If you found something sensitive or non public? Please reach out on slack instead of github. If it’s a flaw in an upstream source, that’s a bit more sensitive and should probably be treated as nonpublic at first to be safe. If you found something like a security flaw in FreeBSD, please reach out directly on slack to Brooks Davis.
Is there a Code of Conduct?
The short answer is ‘not yet’. The team is working towards a proper CoC. It’s not an overnight process. The CheriBSD group will look over the new FreeBSD one, and if it checks out, may decide to adopt it.
How are decisions made? What is the governance model ?
There is no formal or official governance model. Originating as a small research project, decisions were made by the research team. There are now numerous private contracts between various organizations that are involved, and those details are nonpublic. I understand that currently much is still sorted in closed meetings and then disseminated around.
It’s not clear at this time how much influence a volunteer contributor outside that structure can have, or what recognition is possible. It is a question the community will need to resolve as it grows and expands beyond labs and corporate research groups.
I would like to thank the CHERI team for their patience and support. In particular, Dr. Robert Watson, Dr. Wes Filardo, & Dr. Alex Richardson deserve extra thanks and credit for their continued feedback.