IOQuake3

The following is what I've learned about the ioquake3 engine, which I hope will be helpful to you guys while I'm away.

FIXME: Currently, I'm using this page as more of a "hey guys, here's the sit" rather than an actual public page. We can clean it up later.

General Terms & Tips

Finding Things

Trying to find a struct or function definition? Your best bet is probably grep. Most structs (though not all), you'll do better looking for structname_s rather than structname_t, since the _s version is usually only used in the definition, and _t will return a lot of chaff. Function definitions you can probably narrow down by only selecting lines that also contain a "{". Example:

grep -r "BotQuagentAI" code | grep "{"

Most things that you're looking for will probably be in the code/game directory or the code/qcommon directory.

Frames

We call an iteration of a game-driving loop a "frame". Because the client and the server are on different machines, client frames and server frames may operate at completely different frequencies. Quagent AI is called every single server frame (unlike normal Quake bot AI). This is unlikely to cost much computationally on a modern machine, since they are far more powerful than machines made in the early 90s.

Server, Client

Quake is a multiplayer game, so it runs on multiple machines and struggles to keep everything consistent. Clients do prediction and interpolation, while the server periodically gives authoritative snapshots of the game state. Honestly I don't know too much about this: I gave Xu a book that covers it a little.

This might be a problem for our model loading bug - since events that result in the loading of a new model occur on the server, but model loading takes place on the client, we'll have to find a clever way to realize a model is missing in the client (though for all I know, it calls "ModelMissing?()" or something...)

The Virtual Machine

To protect users from malicious mods, Quake has a virtual machine that runs large sections of its code. In fact, most of our modifications are to virtual machine code. On both the server and the client, there is some code running inside the VM, and some code running outside it.

Generally, code inside the following folders are available inside the VM:

• game
• cgame
• ui?

while any code inside the following folders are strictly outside the VM:

• botlib
• client

some code is shared:

• qcommon

and the rest I don't know about.

Getting In and Out of the VM

So say you want to call something inside the VM from a function outside it. I have no idea how to do that, but can point you at VM_Call in the qcommon/vm_* files.

Going the other way is easier, however. You need to add a "trap function" that when called, actually drops out of the VM, reinterprets any memory pointers, and does whatever you need it to. Adding a trap function is a bit of a pain, especially if you're doing it to botlib (if so you'll have to change the function export structure: just grep -i "Stupidly" and you'll find all the changes I made for one of our botlib traps: trap_BotMoveStupidlyInDirection). Adding traps like we did for reading/writing sockets is much easier, though: just follow the step-by-step instructions below and copy the style of the existing entries there. Typically our edits are at the bottom of the files, but if you search "quagents" (case-insensitive) you'll turn then up quickly.

Adding a Trap Function

Edit these files to insert a trap function accessible to the game module (where most of our modifications take place):

• game/g_public.h: add an element to the gameImport_t enum (order matters - try to maintain the values of all the other entries (just add at the bottom)).
• game/g_local.h: add trap function declarations (see bottom of file). These will be the function names inside the VM.
• game/g_syscalls.asm: add equ statements to the file reflecting the (negative) value of the enum entry you added. It should be off by one like all the others are.
• game/g_syscalls.c: add a patch function that calls syscall with the appropriate enum value and parameters, and returns what it returns unless the return is void.
• qcommon/q_shared.h: add trap function declarations (search "Quagents" and keep them together). These will be the function names outside the VM.
• qcommon/common.c: we've defined all our trap functions here, but you probably don't have to. Still, it's a convenient place to do it, so why don't we keep them together and define the function here.
• server/sv_game.c: edit SV_GameSystemCalls and add a case for your enum value to call your function with. If you're passing memory addresses out of the virtual machine, use VMA(x) to remap address in args[x] from VM-memory to Quake-memory. Keep in mind that there's no way to go the other way - you won't be able to see memory outside the VM from within it.

To insert trap functions into cgame, replace game/g_* with cgame/cg_*, and server/sv_game.c with client/cl_cgame. To insert trap functions into ui, ...? Something similar probably.

Quake Filesystem

Quake uses a private, isolated filesystem for all file I/O. This means that all data generated directly from the game - screenshots, log files, etc., will be stored in this filesystem. The "home directory" for this filesystem, where all loads and stores occur by default, is located at ~/.quagents3/quagents3. The '.' prefix on the directory indicates a hidden directory, which means you will not be able to see it from your standard GUI filesystem browser. Instead, use your terminal to access it.

For more information, see the "files.c" file located in code/qcommon.

Adding New Items

To create a new item, edit the following files:

• bg_misc.c - add your item to the end of the list of items, following the same format as the other quagent items.
• inv.h - define your item at the end of the list of quagent items. Make sure to change the value of QUAGENT_ITEM_END accordingly.
• ai_quagentnodes.c - add a case for your new item in the put down, check inventory, and alchemize functions.
• ai_directorcommands.c - add a case for your new item in the drop function

Quagents Behavior

The entry point for an individual Quagent's behavior is the !BotQuagentAI(...) function in game/ai_main.c. Obviously, this already tells you that Quagent AI is running almost entirely in the virtual machine (the only parts that don't are things we trap out of it, such as socket reads, writes, and elementary actions). BotQuagentAI is called between the motion parts of the frame, so usually you have to set up a motion to execute, you can't actually move stuff (or if you do, it can be undone elsewhere!). The main purpose of BotQuagentAI is to (in the following order): check to see if the bot was just set up (and if so, do some initialization, call some built-in functions to get fresh data structures (such as BotUpdateInventory), read any pending commands from the socket and add them to the command queue, attempt to respawn if dead, update some "latent" values, and then execute as many AINodes as possible.

All Quagent behavior runs through an AINode. An AINode is a function assigned to the bs->ainode pointer in bot_state_t, as you can see in the large switch statement.

Generally all our modifications to ai_main.c are at the bottom of that file.

CommandQueue

As you know, the Quagents store their commands into a command queue that is actually an array 1024 commands long. There are two array indexes that are used to point into this array: bs->finalcommand and bs->currentcommand. If finalcommand is ever greater than currentcommand, there are no orders on the queue. When they are equal (pointing to the same command), that command is the only command. Otherwise, finalcommand points at the last command and currentcommand points at the next to be executed. Most commands are atomic, but some will take some duration of time. All manipulation of the queue itself is done in AIExit_Quagents and AIEnter_Quagents in game/ai_quagentsnodes.c.

The queue is an array of "quagent_command_t"s, which are a struct that contains command state information. Because it must be statically sized, it is as large as the state information of the largest command (rb). It has generic spaces for information - by convention, these values are in the same order as ProtocolZero parameters. So for example, if I issue a "n ro 4 180 0", the command for that will have:

command = QCMD_RO
id = 4
valf1 = 180
valf2 = 0

These values are assigned as part of BotAIQuagentsParseCommand's routine. After the command is executed in one frame, the Quagent has rotated a few degrees, and valf1 is updated to be however much is left to go (so, say, valf1 = 166). This way, if the user issues another command that interrupts the rotate, when the rotate eventually resumes it will only rotate the remaining amount. So typically the valf* track parameters, but in some cases they've been used as state machines.

AINodes

All AINodes are functions that takes a bot_state_t* (which is defined in game/ai_main.h) and returns either qfalse or qtrue.

Adding an AINode

You will need to edit 3 files: game/ai_quagentsnodes.c, game/ai_quagentsnodes.h, and game/ai_main.c. First, in game/ai_quagentsnodes.h, declare your new AINode function (stick to the format, please), and make a #define for the opcode. Next, in game/ai_quagentsnodes.c, add the case for your opcode macro to AI_QuagentsOpCode (the first function in that file). This method is used to correctly print returns and response. You will also have to write your actual AINode function in game/ai_quagentsnodes.c. Finally, in game/ai_main.c, you need to add the case for your command in the large switch statement in BotQuagentAI, and then add a case for your node in BotAIQuagentsParseCommand. The cases are currently in alphabetical order by opcode to help the programmer recognize opcode collisions. All the case should do is call AIEnter_Quagents. Copy the first four parameters from any other call (but replace QCMD_?? with your opcode macro). The remainder of the parameters are what should be assigned into the command's val1, valf1, valf2, valf3, and so on slots. You can reference these from your AINode function by saying:

quagent_command_t* cmd=bs->commandqueue+bs->currentcommand;
float valf1=cmd->valf1;
float valf2=cmd->valf2;
//and so on...

in your AINode function. Remember that anything compiled inside the virtual machine must be in (a restricted form of) ANSI C90, so all variable declarations must be at the beginning of compound statements, etc.

Your AINode function should fulfill these requirements:

• When the behavior is finished, the function should call AIExit_Quagents. The parameters you should pass to this are (1) the botstate, (2) the id of the command (this is cmd->id), (3) the value of your command macro (usually easiest to just do cmd->command), and (4) a string stating the return status ("done" for normal execution, or a meaningful error message (no spaces allowed) otherwise). After exiting, the function should return qfalse so that the next command on the command queue is handled.
• When the function has run everything it can this frame but the behavior is not finished (has not called AIExit_Quagents), the function must return qtrue, else the function will be called again.
• All current Quagent-specific AINodes are named AINode_G*, where * is the task name. The G is for quaGent, and is there to prevent collision with Quake's built-in AINodes.

BotQuagentAILatentNodeHelper()

Some Quagent behaviors require updates after the movement portion of the frame has been executed. However, because the user may interrupt the action with another, the functions cannot rely on being called again on the next frame (because a different command may be executed), and there is never a guarantee that they will ever be called again (for example if the user commands a "n fa", all previous interrupted behaviors are never resumed). For some behaviors (usually ones that track progress of movement), this is unacceptable, so part of their functionality is defined in BotQuagentAILatentNodeHelper() in game/ai_main.c. This function is called every AIFrame, and usually updates a progress value for whatever behavior was active in the most recent frame and then clears its memory of which were active.

If you examine the function you'll see that it updates the values pointed to by pointers in bot_state_t. These pointers point into the commandqueue, at elements of a quagent_command_t. Under our design, a quagent command must store all progress and state information into its quagent_command_t. This is because multiple commands of the same type are allowed to be on the queue, so per-type storage won't work, and because the queue has already allocated space for storing their parameters.

Take mb as an example (AINode_GWalkBy is the function). A moveby command attempts to move the bot in a specified direction for a specific distance. However, it is difficult to tell how far exactly the bot will move on a given frame, so what the function does is update a pointer in bot_state, merely request a move, return, and then when BotQuagentAILatentNodeHelper is called on the next AI frame, it figures out how far the bot has moved and decrements the moveby distance by that amount. So on the first frame, the distance may be 50, but on the second, it may be 41, then 22, then 2, then -5. When it becomes negative the bot enters a "backpedaling" scenario, and (if on the ground) teleports back to where it should have moved to.

Thread Safety

Thread safety is not a concern because Quake is single-threaded. If a user sends a command while an AINode is midway through execution, it will not be read until the next AI frame. You are guaranteed that the state of the command queue is constant through the execution of an AINode. (That is, until you return, no function anywhere can interrupt you.) For this reason, it is safe to make multiple calls to trap_Com_QuagentsWriteStr, as the batch rangefinder does.

The Director

I should have all the framework we want in place for the GOD / Director functionality. Try connecting to Quagents on port 6002 and typing stuff. Currently, it'll just echo every command to stdout. I think pretty much anything we want a "configuration file" for, we should build through the director instead. Here's the points:

• Assuming we implement all the commands, anything a config file can do, we could do by piping it in command form into the director with an easy script (that we could even configure to run on startup or something). Can't quite pipe into telnet, but not hard to write a small program that reads a file and fires it over the socket, or just use netcat.

  cat initialcommands.txt | nc localhost 6002
  

• If we put more general options here, we can do things a config file can't, such as change the configuration on the fly. Sure, it's not a huge feature, but it might be nice to add more items to the world as it ran, or something like that.
• The major drawback is that it's not blindingly obvious how to buffer commands the user issues as soon as Quake starts up (say, before the map is loaded). Then again, I'd be surprised if people choose not to just start Quake using the +map option. At least, that's what everyone I know did in the previous Quagents. Actually, I take this back, maybe. We just have it wait on the "hasn't had a read call yet" bit.

If you want to edit director effects, check out the BotHandleDirectorCommand(char* cmdbuff) function in game/ai_main.c. Currently, it just echoes any director commands to stdout, but it gets called once per command with the command string in cmdbuff (see the comments). So you'll need to add stuff to parse out the commands from whatever format they are in and then evaluate them. I decided it might be nice to keep all the director command functions in ai_directorcommands.c and ai_directorcommands.h. Hopefully that won't be a problem.

The director commands uses a similar format to the bot commands. The only difference is that instead of adding the commands to the command queue, the function that performs the actions is called directly and executed. This is handeled in the BotHandleDirectorCommand(char* cmdbuff) function in game/ai_main.c. All of the director commands are in ai_directorcommands.c