4-1. The module concept
4-2. Structure of a module
4-2-1. Required symbols
4-2-2. Utility macros and functions
4-2-3. Dynamic versus static modules
4-3. Module management
4-3-1. Module loading
4-3-2. Module configuration
4-3-3. Module unloading
4-3-4. Searching for a module
4-3-5. Locking a module in memory
4-3-6. Global initialization and cleanup
4-4. External symbols
4-4-1. Exporting symbols
4-4-2. Importing symbols
4-5. Callbacks
4-5-1. Registering callbacks
4-5-2. Hooking into callbacks
4-5-3. Calling callbacks
4-6. Adding a module to Services
Previous section: Communication (socket) handling | Table of Contents | Next section: IRC server interface
The module system was introduced in Services 5.0, partly to allow greater flexibility in configuring Services, partly to simplify the process of extending Services, and partly to help clarify the boundaries between distinct sets of functionality. Rather than merge every conceivable function into a single monolithic program, Services was divided into a comparatively small core and a number of modules which each implemented distinct sets of features. These modules could then be developed separately from the core and from each other, without concern for the effects of changes in other parts of the program.
Aside from providing a cleaner program structure, modules make it simpler to implement different variations on a particular feature. A good example of this is the protocol modules (discussed in section 5); rather than a complex set of #if and #else directives to handle differences between IRC server types, a separate protocol module can be written for each type of IRC server; this results in a certain amount of code duplication for similar protocols, but makes the code for each protocol much more maintainable, and allows the code for a single protocol to be changed easily without worrying that it will break processing for other protocols.
Modules also make it easier to add new features to Services, whether as part of the Services package or as third-party additions. By connecting to the Services core and standard modules through a set of well-defined callbacks, no changes need to be made to the main source code when the module is updated, and vice versa. The compilation system (see section 10-3-2) also makes it easy for users to add modules on their own, by simply "dropping in" the source code for the new module; Services will automatically find the module and compile it.
Below, section 4-2 discusses the basic structure of a module, along with other background issues; section 4-3 discusses how modules are managed, including the details of module loading, configuration, and unloading; section 4-4 discusses how to make symbols available to other modules and use symbols from other modules; and section 4-5 covers callbacks.
From the viewpoint of the Services core, a module is a single unit of functionality with a known interface. The only direct accesses made to module data are to a limited number of functions and variables (as described in section 4-2-1 below); all other module interactions occur through callbacks, which it is the module's responsibility to install as necessary.
A typical module, therefore, will use its initialization routine (one of the directly-accessed functions) to install callback functions, enabling it to respond to events such as messages from the IRC server or connections on a socket. After performing any other necessary setup actions, the initialization routine returns control to the Services core, which will then call the module's callback functions as appropriate. The bulk of a module is thus usually contained in these callback functions or their helper routines.
Modules are required to define two variables, used by the module loader: int32 module_version, initialized to the value of the macro MODULE_VERSION_CODE, and Module **_this_module_ptr, a pointer to a variable which will receive the module's handle (see section 4-3 for information on module handles). However, both of these variables are normally defined by modules.h when compiling the module's main file (the source file with the same base name as the module object file), so ordinarily there is no need to be concerned about these variables.
Modules may optionally define two functions, which the module loader will call when the module is loaded and unloaded, respectively:
Modules may also optionally define ConfigDirective module_config[] (this must be an array, not a pointer), containing configuration information for the module which will be processed by the module loader. The use of this array is described in section 4-3-2.
The header file modules.h, which must be included by all module source files, includes a few macros for convenience in writing modules:
In addition, many of the routines described in this section take a module or caller parameter, so that the module subsystem knows which module is making the call; however, macros are used to hide this parameter from the module source code, both to improve readability of the code and to avoid potential errors from passing the wrong parameter. These functions are:
It is possible for modules to be compiled either dynamically, so that the module is loaded into memory only when used, or statically, building all modules into the main executable file. The decision of which to use is made by the user via the configure script, and cannot be controlled by individual modules. In general, modules will behave exactly the same whether compiled dynamically or statically; however, it is important to be aware of particular behaviors with each method, to avoid problems that cause the module to fail when using one method or the other.
Dynamic linking involves compiling each module into an independent loadable object file, which is then loaded into memory as needed using the system's dlopen() family of functions. (If these functions are not available, dynamic modules cannot be used; the configure script will detect this and select static modules automatically.) Any direct references to symbols not defined in the module, such as core Services routines, will be marked as unresolved references in the object file. As a result, it is possible to compile a module that references a nonexistent symbol—for example, accidentally typing strmcp instead of strcmp—and the error will not be detected until an attempt is made to load the module.
Static linking, on the other hand, results in every module being linked together with the core code to produce a single executable file. This means that every symbol in the Services core—and in other modules—will be available for use. Static linking can detect references to nonexistent symbols at compile time, unlike dynamic linking, but will not generate an error if one module references a symbol from another. For example, if a protocol module erroneously references a symbol from one of the pseudoclients directly (rather than using the symbol lookup method described in section 4-4-2), the protocol module will fail to load when using dynamic linking unless the appropriate pseudoclient module is loaded first. (Since the current design of Services requires that a protocol module be loaded first, this would result in Services not being able to run at all.)
When using modules composed of two or more source files that share internal-use symbols, static linking can cause spurious errors at link time. This is because, even though the symbols are only used internally, they are still included in the final object file for the module; thus, if two separate modules happen to use the same name for a function or variable, the final link will report a "duplicate symbol" error. This is an actual problem with the NickServ and ChanServ modules, both of which include various internal utility routines in a separate source file, and as a result some ChanServ routines have to be renamed (see the relevant part of section 7-4-1-1).
Additionally, when using static linking, module-global variables will always retain their values between a module unload and a subsequent module load; when using dynamic linking, it is system-dependent whether the values will be retained or reset to their initial values. Thus it is important that a module's cleanup routine not only free all resources used by the module, but also reset all variables to appropriate values for the next call to the initialization routine.
For these reasons, it is best to test modules using both dynamic and static linking, to ensure that unintended errors are not introduced.
The module subsystem uses the routines described in this section to manage modules, performing loading, configuration and unloading as well as checking whether a module is loaded or locking modules into memory.
Most module subsystem routines take or return a module handle, a value of type Module *, to identify a particular module. The Module type is declared as simply struct Module_ in modules.h, rendering the type opaque to callers. Internally, struct Module_ is used to hold information about each loaded module, and has the following fields:
Loaded modules are kept as a doubly-linked list, with the list head in the file-scope variable modulelist. This list always contains at least one entry: a dummy entry for the Services core, defined in the variable coremodule.
The module management routines are:
A module can be loaded by calling the load_module() routine. This routine takes a single parameter, a string giving the name of the module to load (such as "nickserv/main"), and returns the handle for the newly-loaded module or NULL on failure. The routine:
The actual low-level work of loading the module and initializing the Module structure is handled by the internal_load_module() helper routine mentioned above. This routine:
To perform the lookup, internal_load_module() first calls check_module_symbol() to get the symbol value. Then, if dynamic linking is in use, the returned value is checked the this_module_pptr field of each module in the loaded module list to see whether the same address has been used for a different module. This latter check is required because of a problem (on some systems) with the system function dlsym(), used for looking up symbols when dynamic linking is in use: if the requested symbol is not present in the requested module but is present in another module, dlsym() may return the symbol from the other module (see section 4-4-2 for details).
Note that the implementation used with static linking does not have this problem, rendering the check unnecessary. In fact, attempting to perform the check with static linking can actually cause problems on some systems. This is due to constant merging performed by linkers, which results in distinct modules legitimately having the same value for such a symbol. This has been observed with the module_version symbol (see below), so the check is disabled when static linking is in use.
The my_dlopen() function referred to above is one of four wrappers for the system dynamic linking functions: dlopen(), dlclose(), dlsym(), and dlerror(). When using dynamic linking, the wrapper functions essentially just call their system counterparts, although my_dlopen() generates the actual pathname from the module name (prepending the module directory pathname and appending the shared object extension ".so"), and my_dlsym() includes workarounds for systems that prepend symbol names with underscores and that do not handle dlsym(NULL,...). When static linking is in use, the wrapper functions instead search through the pregenerated module and symbol lists for the requested module or symbol (dlclose() does nothing, and dlerror() uses a static error message variable to hold the value to return). See section 10-3-2) for details on the generation of these lists.
Note that load_module() (and its counterpart, unload_module()) are only intended to be called by the Services core to load modules specified by LoadModule directives in ircservices.conf. Modules should never attempt to load other modules on their own, since the user may have intentionally chosen not to load certain modules; instead, an error should be generated if a required module is not available.
Configuration of modules is performed through the same routine, configure(), that handles the Services core configuration; however, a different file is used, modules.conf rather than ircservices.conf, and each module's configuration directives must be enclosed by the Module and EndModule meta-directives. (Configuration files and the configure() routine are described in detail in section 2-3-2.)
Each module can define configuration directives to be parsed by the configuration file reader by creating an array of ConfigDirective entries named module_config. load_module() will check for this symbol and, if found, pass it to configure(), as described above; thus, the variables corresponding to the configuration directives will have already been set when the module's init_module() function is called. The directives will likewise be reprocessed automatically during reconfiguration; however, it may be necessary for the module to detect changes in configuration variables (such as the nickname of a pseudoclient). For this purpose, the module can hook into the "reconfigure" callback, which is called once before reconfiguration takes place and once after (see appendix C for details on the callback).
All configuration variables will be reset to their initial values when the module is unloaded, so the module's exit_module() routine does not need to take any particular actions with respect to such variables.
Loaded modules are unloaded from memory with the unload_module() routine. This routine, which is implemented in a separate internal_unload_module() routine to allow passing a shutdown parameter to the module's cleanup function, performs the reverse of load_module()'s operations, returning nonzero if the module is successfully unloaded, zero if not. The routine:
Aside from unload_module(), modules are also unloaded during the shutdown process (see section 4-3-6).
If one module needs to access another, it can do so with the find_module() routine. This routine takes a module name as parameter, searches through the loaded module list for a module whose name exactly matches the given name (case-sensitively), and returns the handle for that module (NULL if no matching module is found).
find_module() is used most commonly when one module wants to use functionality provided by another; for example, the module implementing the ChanServ pseudoclient (chanserv/main) uses it to retrieve the handle for the NickServ module (nickserv/main). Note, however, that in cases where the module being searched for is not required, it may not be present at initialization time; in this case, it is better to hook into the core's "load module" and "unload module" callbacks, and only use find_module() to check whether the desired module has already been loaded when init_module() is called. For example, the MemoServ module, memoserv/main, takes this approach with respect to the chanserv/main module when deciding whether to enable channel memos.
If one module requires another in order to function properly, the "user" module can request that the "used" module not be unloaded. This is accomplished by calling the use_module() routine, passing the "used" module as the module parameter; this causes the specified module to be locked in memory. When the module is no longer needed (for example, when the "user" module's exit_module() routine is being called for unloading), unuse_module() unlocks the module, allowing it to be unloaded normally.
One problem that can arise when locking resources is the case of a circular lock dependency: if module A has locked module B, but module B has also locked module A, then neither of them can be unloaded until one or the other module releases its lock; if the modules only release their locks at cleanup time, there would be no way to unload them. In order to prevent situations like this, use_module() checks for such circular dependencies and issues a warning if one is detected (in the form of a "BUG" message written to the log file). Naturally, it is also not permitted for a module to attempt to lock itself.
Note that both of these routines are internally named with a preceding underscore (_use_module(), _unuse_module()) and take a second parameter, const Module *caller, which is used to ensure that modules do not leave stale locks around when they are unloaded (as described in section 4-3-3). However, these are hidden by macros which automatically pass THIS_MODULE as the caller parameter.
Before calling any module-related routines, the module subsystem must be initialized by calling modules_init(), which obtains the system handle for the program itself (when using dynamic linking) and sets up the three module subsystem callbacks: "load module", "reconfigure", and "unload module".
Likewise, the modules_cleanup() routine must be called when the program shuts down. Aside from removing the callbacks mentioned above, modules_cleanup() checks that other parts of the Services core have removed any callbacks they registered, and calls the internal routine unload_all_modules() to walk the loaded module list and unload each module. In this case, each module's exit_module() function is called with a nonzero shutdown value, indicating that the program is shutting down and the module must clean up after itself. (The module will be unloaded regardless of the return value of exit_module().)
While some modules only require Services core functions to operate, most interact with other modules. This necessitates a way of accessing functions and data provided by outside modules.
A module can designate certain symbols (functions or variables) to be exported, so that other modules can access them as described in the next section. This is done using one of three macros, depending on what is being exported:
These macros do not expand to anything themselves, but are used by the compilation process (see section 10-3-2) to generate lists of exported symbols when using static linking. Even without these macros, the symbols will still (if not declared static) be recorded in the object file, but since there is no way to access the program's symbols when using static linking—and in fact, the symbols may have been stripped from the executable file by the user—the macros must be used in order for get_module_symbol() (see section 4-4-2 below) to be able to retrieve the symbol's value.
Note that the symbols used by the module subsystem (module_version and so on, as described in section 4-2-1) are automatically made available to the module subsystem, and need not be explicitly exported. This is done by the Makefile controlling module compilation, as described in section 10-3-2.
There are two methods of accessing symbols located in other modules: by calling get_module_symbol() or check_module_symbol(), or by referencing the symbol directly. Each has its benefits and disadvantages.
get_module_symbol() and check_module_symbol() allow a caller to retrieve the value of a given symbol in a specified (or any) module. The functions are declared as follows (note that get_module_symbol() is declared internally as _get_module_symbol() and takes a hidden caller parameter, but these are hidden by a macro in modules.h):
Both of these functions retrieve the value of the symbol symname in the module module; if module is NULL, the symbol is searched for in all loaded modules (if defined in more than one module, then an arbitrary module is used for the lookup). If the symbol is not defined in module but is defined in another module, then the result of the lookup is defined to be either NULL or the value of the symbol in the module in which it is defined. This undesirable behavior unfortunately cannot be avoided due to identical behavior by (at least) the glibc library used on Linux. When using static linking, the functions always return NULL for this case.
The difference between their two functions is how they deal with the case of a nonexistent symbol. get_module_symbol() is intended for use when the symbol is assumed to exist; it returns the symbol's value directly if it exist, and logs a warning and returns NULL otherwise. The danger of using this for symbols that may legitimately not exist (other than generating spurious log messages) is that it is possible for a symbol to have the value zero (NULL), which would be indistinguishable from a symbol lookup failure. check_module_symbol(), on the other hands, takes two extra parameters, a pointer to a variable in which the result is to be stored on success and a pointer to a variable in which the relevant error message is to be stored on failure (either or both of which may be NULL), and returns nonzero on success, zero on failure. On success, the variable pointed to by resultptr is set to the value of the pointer, and the variable pointed to by errorptr is left unchanged; on failure, the variable pointed to by errorptr is set to a human-readable error string, and the variable pointed to by resultptr is left unchanged.
Note that the "value of the symbol" here is the value in linker terms; in other words, when using these functions to access a variable, the value of the symbol is the address of the variable, not its value. To access the value of the s_NickServ string variable in an unspecified module, for example, one would write code like:
It is also important to keep in mind that the value of such variables may change, or the module containing the variable may be unloaded, at any time. Accesses should therefore always be done by looking up the symbol, ensuring that the symbol exists, and referencing it through the returned pointer. Since this can incur significant overhead, however, another option (when the module name is known) is to use the "load module" and "unload module" callbacks to watch for the module to be loaded or unloaded; a single get_module_symbol() or check_module_symbol() will then suffice, and the returned variable pointer can be saved locally until the module is later unloaded. Many of the modules distributed with Services take this approach.
The other method of accessing symbols in other modules is to simply reference the symbol as if it was an ordinary external variable or function. Doing this avoids the overhead of looking up the symbol every time it is to be used, or monitoring the relevant module for loads and unloads. However, it also requires the module containing the symbol to be loaded before the module that uses it; if the modules are not loaded in the correct order, load_module() will report an "unresolved symbol" error. For example, if module A exports a symbol symA which module B accesses directly, attempting to load module B before module A will cause an "unresolved symbol" error for symA, without giving module B a chance to manually resolve the symbol. As a general rule, therefore, it is recommended to avoid direct references to symbols from other modules, unless the other module can be reasonably expected to be loaded first (for example, many of the pseudoclients' sub-modules access symbols from their respective core modules in this fashion).
Callbacks are a method of allowing modules to signal other modules when certain events occur. A module first establishes a callback by registering it, giving it a name unique among all callbacks for that module (two callbacks in different modules can share the same name). Other modules can use that name to refer to the callback, and add functions to be called when the callback's event occurs (also referred to as hooking into the callback). The module which added the callback can then instruct the module subsystem to call each of those functions in turn when the requisite event occurs.
The Services core also sets up a number of callbacks which can be hooked into by modules as necessary. A full list of callbacks available in the Services core and standard modules can be found in Appendix C.
A module that wishes to register a new callback can do so with the register_callback() routine (internally _register_callback()):
This routine checks to ensure that the given module has not already registered a callback of the same name (by calling find_callback(), an internal helper function that looks for a callback by name in a given module), then adds the callback to the module's callback table with an empty callback function list. The return value is the ID value assigned to the callback (a value used when calling or removing the callback, both for efficiency and to eliminate the possibility of mistyping the callback name string when referring to it internally); a return value of -1 signals an error. Note that the module's THIS_MODULE macro (which evaluates to NULL when called by the core) is automatically passed to the function by the register_callback() macro in modules.h, like the caller parameter to use_module() and unuse_module().
When the callback is no longer needed, it can be unregistered by calling unregister_callback() (internally _unregister_callback()):
The routine returns 1 if the callback was found and freed, 0 if the ID value given was invalid.
Once a callback has been registered, other modules can add functions to or remove functions from the callback with the add_callback(), add_callback_pri(), and remove_callback() functions (internally _add_callback_pri() and _remove_callback—add_callback() is defined in terms of add_callback_pri()):
The type of a callback function is defined as callback_t:
The number and type of parameters passed to the function depends on the particular callback, and thus the callback_t type is not a true prototype. The int return value is used both to return a result from the callback function itself and to control execution of the list of callback functions, with a nonzero value terminating execution, as described in section 4-5-3 below.
add_callback() and add_callback_pri() add the given function to the selected callback, returning nonzero on success, zero on error. In the case of add_callback_pri(), the callback function can be given a numeric priority; callback functions with higher priority values are called before those with lower values, and callback functions with the same priority value are called in the same order they were added. add_callback() is equivalent to add_callback_pri() with a priority of zero, and in fact is implemented as a macro that does exactly that. Implementation note: Priorities were added to allow the operserv/sessions module to have its "user check" callback function, which increments the user count for a session if the user passes the session check, called last. This is arguably a bad design decision, as it would be better (if less efficient) to move the session count incrementing to the "user create" callback, and the current method cannot stop another module from adding a callback function at an even lower priority and causing the session count to "leak" if the lower-priority function rejects the user.
remove_callback() removes the given function from the selected callback, returning nonzero if the function was found and successfully removed, else zero. remove_callback() deliberately does not generate a log warning when used with a callback function that has not been added, in order to simplify modules' cleanup routines (so that they can call remove_callback() for all callbacks that may have potentially been added without having to keep track of which ones were in fact added).
A module can instruct the module subsystem to call all callback functions for a particular callback with the call_callback() set of functions (actually macros, with the implementing function internally named _call_callback_5()):
As with register_module() and unregister_module(), the module handle of the calling module is automatically passed in the hidden module parameter.
These functions all call the selected callback (specified by callback ID value, as returned from register_callback()), and differ only in the number of parameters passed to the callback. The parameters arg1 through arg5 are passed to the callback functions unchanged; the parameters are declared as the (more or less) generic type void *, but may be of any type, as the call_callback() macros take care of converting the parameters to void * to avoid type conversion warnings (but see section 11-1 for problems with this method of passing parameters). It is the callback function's responsibility to ensure that it is declared with the correct parameter list.
After checking the validity of the module and callback ID parameters, call_callback() calls each callback function that has been added to the callback, in priority and time order (see section 4-5-2 about calling order). If any callback function returns a nonzero value, call_callback() stops immediately and returns that value, skipping the rest of the callback functions; if all callback functions return zero (or if there are no callback functions for the given callback), zero is returned. (-1 is returned if the callback ID is invalid.) Thus, callbacks should be designed so that a return value of zero from a callback function means "continue processing" and a nonzero value means "stop processing or take some other action".
Adding a new module to Services consists generally of the following steps:
For submodules of pre-existing modules, like new NickServ or ChanServ features, the module can naturally go in the appropriate pre-existing directory instead, with the corresponding Makefile modified to compile the new module.
For modules intended to be distributed separately from Services itself, it is sufficient to distribute the subdirectory containing the module source code and Makefile; by copying that directory into the modules directory of a fresh Services distribution, the module Makefile will automatically detect it and compile the module without any further intervention on the part of the user. (See also section 3-10 of th user's manual.)
In addition to this section, sections 2-2 (concerning utility functions) and 2-4 (concerning logging), along with Appendix C (containing a list of core and standard module callbacks), provide useful information for module development. Additionally, the following sections of this manual can be helpful in writing different types of modules:
Previous section: Communication (socket) handling | Table of Contents | Next section: IRC server interface