INCLUDES, AUTOLOAD, BYTECODE CACHES and OPTIMIZATION The Problem ----------- HTML Purifier contains a number of extra components that are not used all of the time, only if the user explicitly specifies that we should use them. Some of these optional components are optionally included (Filter, Language, Lexer, Printer), while others are included all the time (Injector, URIFilter, HTMLModule, URIScheme). We will stipulate that these are all developer specified: it is conceivable that certain Tokens are not used, but this is user-dependent and should not be trusted. We should come up with a consistent way to handle these things and ensure that we get the maximum performance when there is bytecode caches and when there are not. Unfortunately, these two goals seem contrary to each other. A peripheral issue is the performance of ConfigSchema, which has been shown take a large, constant amount of initialization time, and is intricately linked to the issue of includes due to its pervasive use in our plugin architecture. Pros and Cons ------------- We will assume that user-based extensions will be included by them. Conditional includes: Pros: - User management is simplified; only a single directive needs to be set - Only necessary code is included Cons: - Doesn't play nicely with opcode caches - Adds complexity to standalone version - Optional configuration directives are not exposed without a little extra coaxing (not implemented yet) Include it all: Pros: - User management is still simple - Plays nicely with opcode caches and standalone version - All configuration directives are present Cons: - Lots of (how much?) extra code is included - Classes that inherit from external libraries will cause compile errors Build an include stub (Let's do this!): Pros: - Only necessary code is included - Plays nicely with opcode caches and standalone version - require (without once) can be used, see above - Could further extend as a compilation to one file Cons: - Not implemented yet - Requires user intervention and use of a command line script - Standalone script must be chained to this - More complex and compiled-language-like - Requires a whole new class of system-wide configuration directives, as configuration objects can be reused - Determining what needs to be included can be complex (see above) - No way of autodetecting dynamically instantiated classes - Might be slow Include stubs ------------- This solution may be "just right" for users who are heavily oriented towards performance. However, there are a number of picky implementation details to work out beforehand. The number one concern is how to make the HTML Purifier files "work out of the box", while still being able to easily get them into a form that works with this setup. As the codebase stands right now, it would be necessary to strip out all of the require_once calls. The only way we could get rid of the require_once calls is to use __autoload or use the stub for all cases (which might not be a bad idea). Aside ----- An important thing to remember, however, is that these require_once's are valuable data about what classes a file needs. Unfortunately, there's no distinction between whether or not the file is needed all the time, or whether or not it is one of our "optional" files. Thus, it is effectively useless. Deprecated ---------- One of the things I'd like to do is have the code search for any classes that are explicitly mentioned in the code. If a class isn't mentioned, I get to assume that it is "optional," i.e. included via introspection. The choice is either to use PHP's tokenizer or use regexps; regexps would be faster but a tokenizer would be more correct. If this ends up being unfeasible, adding dependency comments isn't a bad idea. (This could even be done automatically by search/replacing require_once, although we'd have to manually inspect the results for the optional requires.) NOTE: This ends up not being necessary, as we're going to make the user figure out all the extra classes they need, and only include the core which is predetermined. Using the autoload framework with include stubs works nicely with introspective classes: instead of having to have require_once inside the function, we can let autoload do the work; we simply need to new $class or accept the object straight from the caller. Handling filters becomes a simple matter of ticking off configuration directives, and if ConfigSchema spits out errors, adding the necessary includes. We could also use the autoload framework as a fallback, in case the user forgets to make the include, but doesn't really care about performance. Insight ------- All of this talk is merely a natural extension of what our current standalone functionality does. However, instead of having our code perform the includes, or attempting to inline everything that possibly could be used, we boot the issue to the user, making them include everything or setup the fallback autoload handler. Configuration Schema -------------------- A common deficiency for all of the conditional include setups (including the dynamically built include PHP stub) is that if one of this conditionally included files includes a configuration directive, it is not accessible to configdoc. A stopgap solution for this problem is to have it piggy-back off of the data in the merge-library.php script to figure out what extra files it needs to include, but if the file also inherits classes that don't exist, we're in big trouble. I think it's high time we centralized the configuration documentation. However, the type checking has been a great boon for the library, and I'd like to keep that. The compromise is to use some other source, and then parse it into the ConfigSchema internal format (sans all of those nasty documentation strings which we really don't need at runtime) and serialize that for future use. The next question is that of format. XML is very verbose, and the prospect of setting defaults in it gives me willies. However, this may be necessary. Splitting up the file into manageable chunks may alleviate this trouble, and we may be even want to create our own format optimized for specifying configuration. It might look like (based off the PHPT format, which is nicely compact yet unambiguous and human-readable): Core.HiddenElements TYPE: lookup DEFAULT: array('script', 'style') // auto-converted during processing --ALIASES-- Core.InvisibleElements, Core.StupidElements --DESCRIPTION--

Blah blah

The first line is the directive name, the lines after that prior to the first --HEADER-- block are single-line values, and then after that the multiline values are there. No value is restricted to a particular format: DEFAULT could very well be multiline if that would be easier. This would make it insanely easy, also, to add arbitrary extra parameters, like: VERSION: 3.0.0 ALLOWED: 'none', 'light', 'medium', 'heavy' // this is wrapped in array() EXTERNAL: CSSTidy // this would be documented somewhere else with a URL The final loss would be that you wouldn't know what file the directive was used in; with some clever regexps it should be possible to figure out where $config->get($ns, $d); occurs. Reflective calls to the configuration object is mitigated by the fact that getBatch is used, so we can simply talk about that in the namespace definition page. This might be slow, but it would only happen when we are creating the documentation for consumption, and is sugar. We can put this in a schema/ directory, outside of HTML Purifier. The serialized data gets treated like entities.ser. The final thing that needs to be handled is user defined configurations. They can be added at runtime using ConfigSchema::registerDirectory() which globs the directory and grabs all of the directives to be incorporated in. Then, the result is saved. We may want to take advantage of the DefinitionCache framework, although it is not altogether certain what configuration directives would be used to generate our key (meta-directives!) Further thoughts ---------------- Our master configuration schema will only need to be updated once every new version, so it's easily versionable. User specified schema files are far more volatile, but it's far too expensive to check the filemtimes of all the files, so a DefinitionRev style mechanism works better. However, we can uniquely identify the schema based on the directories they loaded, so there's no need for a DefinitionId until we give them full programmatic control. These variables should be directly incorporated into ConfigSchema, and ConfigSchema should handle serialization. Some refactoring will be necessary for the DefinitionCache classes, as they are built with Config in mind. If the user changes something, the cache file gets rebuilt. If the version changes, the cache file gets rebuilt. Since our unit tests flush the caches before we start, and the operation is pretty fast, this will not negatively impact unit testing. One last thing: certain configuration directives require that files get added. They may even be specified dynamically. It is not a good idea for the HTMLPurifier_Config object to be used directly for such matters. Instead, the userland code should explicitly perform the includes. We may put in something like: REQUIRES: HTMLPurifier_Filter_ExtractStyleBlocks To indicate that if that class doesn't exist, and the user is attempting to use the directive, we should fatally error out. The stub includes the core files, and the user includes everything else. Any reflective things like new $class would be required to tie in with the configuration. It would work very well with rarely used configuration options, but it wouldn't be so good for "core" parts that can be disabled. In such cases the core include file would need to be modified, and the only way to properly do this is use the configuration object. Once again, our ability to create cache keys saves the day again: we can create arbitrary stub files for arbitrary configurations and include those. They could even be the single file affairs. The only thing we'd need to include, then, would be HTMLPurifier_Config! Then, the configuration object would load the library. An aside... ----------- One questions, however, the wisdom of letting PHP files write other PHP files. It seems like a recipe for disaster, or at least lots of headaches in highly secured setups, where PHP does not have the ability to write to its root. In such cases, we could use sticky bits or tell the user to manually generate the file. The other troublesome bit is actually doing the calculations necessary. For certain cases, it's simple (such as URIScheme), but for AttrDef and HTMLModule the dependency trees are very complex in relation to %HTML.Allowed and friends. I think that this idea should be shelved and looked at a later, less insane date. An interesting dilemma presents itself when a configuration form is offered to the user. Normally, the configuration object is not accessible without editing PHP code; this facility changes thing. The sensible thing to do is stipulate that all classes required by the directives you allow must be included. Unit testing ------------ Setting up the parsing and translation into our existing format would not be difficult to do. It might represent a good time for us to rethink our tests for these facilities; as creative as they are, they are often hacky and require public visibility for things that ought to be protected. This is especially applicable for our DefinitionCache tests. Migration --------- Because we are not *adding* anything essentially new, it should be trivial to write a script to take our existing data and dump it into the new format. Well, not trivial, but fairly easy to accomplish. Primary implementation difficulties would probably involve formatting the file nicely. Backwards-compatibility ----------------------- I expect that the ConfigSchema methods should stick around for a little bit, but display E_USER_NOTICE warnings that they are deprecated. This will require documentation! New stuff --------- VERSION: Version number directive was introduced DEPRECATED-VERSION: If the directive was deprecated, when was it deprecated? DEPRECATED-USE: If the directive was deprecated, what should the user use now? REQUIRES: What classes does this configuration directive require, but are not part of the HTML Purifier core? vim: et sw=4 sts=4