Last weekend, Matt Mullenweg spoke at length about what the future holds for WordPress development. I took some time to share a brief proposal for one change that could be made farther down the road. Today I’d like to share another – more in line with Matt’s proposal that future WordPress development be focused around plugins.
Make WordPress modular.
Aren’t Plugins Already Modular?
WordPress already implements a robust plugin framework. Just about anyone can write code to extend the functionality of WordPress core by building a plugin that either hooks in to a WordPress action or modifies the return of a WordPress filter.
It’s immensely powerful, but still has limits.
For one, plugins have an inherent dependency on the underlying WordPress plugin API. This means the API has to be loaded before the plugins are loaded. It also means that plugins can merely extend the default functionality of WordPress rather than replace it.
A Modular Framework
The idea behind a modular framework requires a couple of concepts that aren’t implemented in WordPress just yet:
- Libraries as pluggable modules
- A dynamic module loader
WordPress today uses blocks of require() and include() statements to load functionality, power up the API, and make the server ready for action. This works, but it’s not very flexible.
A modular framework would first register a module loader, then register available modules, and finally load the modules as needed.
A modular framework would permit developers to replace parts of the core WordPress API with newer implementations wholesale.
What We Gain
If the future of core WordPress development lies in plugins, then we would optimally need a way to not just extend the core API with these plugins but to replace parts of it with new functionality. API refinements could then be tested by installing and activating a module rather than installing a new copy of WordPress.
Developers could also build out core API implementations specific to differing versions of PHP, without needing to force these rewrites back to the core API. Running PHP 5.5-optimized code then becomes a matter of installing a plugin rather than forking WordPress.
The Challenge
Right now, plugins are only loaded after the majority of the WordPress API. To build out a modular system, we’d need to first add a module loader, then somehow load third-party modules alongside the core modules.
One potential loading order would become:
- Load module platform
- Register core modules
- Detect and register pluggable modules
- Begin the WordPress API loading process
This means we’d need to build a module loader that is added before WordPress core. Then, rather than require()ing core modules, we would register them with the loader. Next we would detect any plugin modules (perhaps drop-ins in a /wp-content/modules directory). Finally we would process the rest of the API loading like normal.
If we decide that a core API or a set of core PHP files need to be updated in a future version, we register the new version as a pluggable module and WordPress would automatically load them in place of the core files.
Will it Work
Rather than just spout off about an idea, I plan to actually implement a prototype of the module loader over the coming weeks. As I do so though, I’m interested in what you think.
Will this potentially help further WordPress core development? If you could build a plugin that replaces a core API, which one would you work on? Is this an idea we should pursue, or just an interesting talking point?
Feel free to continue the conversation below.
Looking forward to checking it out!
It seems there’s already some of that happening; with drop-ins like advanced cache and wpdb, this seems like formalizing what is currently an adhoc system.
On the other hand, what benefit do we gain by allowing folks to completely replace core modules when most of them can be filtered/hooked into, etc?
This idea is heavily based on the system pioneered by things like object caching. The advantage over hooks is that it would allow complete replacement of whole swaths of the API.
Think refactoring between versions to replace function calls with optimized, experimental code. Or testing the addition of a hook or filter. Or a trial method deprecation.
Been doing it this way for a few years – nothing new here, just WP catching up with the times and figuring out a better way to do things. Perhaps one day it will be fast and flexible 🙂
It’s already fast and flexible. I’ve actually seen first hand how much the hook system in WordPress has influenced “event-driven” APIs on other frameworks, so this isn’t a sign of WordPress “catching up” to others.
But my hope is that a model like this could help it become even more flexible.
I suppose it all depends on what you consider fast 🙂
WordPress by itself is quite fast. It’s when we start adding plugins that invoke custom post types and post meta that it begins to slow down. Post meta uses non-indexed plain text fields for queries and makes more advanced queries on a large database somewhat cumbersome and slow. The bottleneck for speed and performance is often the DB and not the code.
But I have seen indications that will be changing in the future (snippets of more refined DB schema) and I’m hoping a more flexible, modular API system will help prevent the code from becoming the bottleneck.
Again, it all depends what you consider fast, personally a page load from start to finish (including rendering) with no caching inside 300ms is what I call fast and I am yet to see that from WordPress.
There is no difference between slow code and slow database queries – they’re both as important as each other in an application. Shifting blame from one to the other and taking a head in the sand approach (which is what happens with most things like this) doesn’t help.
If these “bottlenecks” had been first thought out in a different or more modular way then there would be nothing wrong hence my original comment of WordPress catching up with the times. I am not just picking on WP – most of not all open source CMS/Blogs do not look closely enough at performance when starting out and then backpedal and retro fit hacks on top of hacks to achieve performance.
Considering how many projects (OS and otherwise) are embracing the new-ish trend of “agile” development, the idea that planning everything out from the beginning is “catching up with the times” feels backwards to me. That said, not trying to shift blame from code to DB or anything, just pointing out what I’ve seen in direct performance profiling with WordPress.
Adding a cache layer (i.e. an object cache to alleviate DB load) is a huge performance gain for WordPress. Take a look at large, high-traffic sites like TechCrunch. This site runs on WordPress, and thanks to Batcache’s page cache, content is delivered from memory in ~0.006s. Network latency then becomes the bottleneck, which unfortunately isn’t something we could solve by refactoring code anyway.
I would be hugely interested if you know of a dynamic (i.e. code + database) CMS that can fit the “load from start to finish (including rendering) with no caching inside 300ms” bill. That’s a tall order, but if it exists, looking at how it achieves that kind of performance would be a great way for WordPress – and other devs – to learn.
I will be open sourcing the CMS that achieves this at some point in the near future so hopefully devs can look at how it can and should be done and we can all have a faster internet 🙂
I dig it.
Agreed, nice look into this and see how we can improve WP developments in the future.
Hi @Eric,
It would be nice to see WordPress become more modular. However, I don’t there are too many things that we’d need replacement of components except for in services used by core such as caching, email sending, maybe database, etc.
Where I see do modularity really being beneficial is to allow site builders to remove component that a site is not using. For example, the ability to remove the feed system or the ‘post’ post type or the importer just to make a few. But it would only be really beneficial it said removal could be done using a package builder like jQuery UI.
What I think would be really beneficial in addition to or instead of (because the scope of moving to modules as we are discussing would be really large) would be to add a “library” system to WordPress. Specifically I’m referring to libraries that would often be shared between multiple plugins and the version management logic to make it workable. Currently there’s no good, reliable way to allow plugin developers to build and share libraries.
I have actually built such a library manager and been using it, but it in plugins for a while but it could be made more robust if it were included in code:
– https://github.com/newclarity/imperative
The basic idea is you say “I need this library with this “well-known” name (a GitHub URL path) and I’m expecting this numbered version” and you do this in the entry point of your plugin. You also create a “loader” file that contains the code you would normally put in your plugin, code that the library manager loads after it decides what needs to be loaded.
Here’s an example of a plugin entry point that uses the library loader:
– http://plugins.svn.wordpress.org/lexity-live-for-wp-e-commerce/tags/1.0.9/lexity-live-for-wp-e-commerce.php
That file uses the “Imperative” library loader and loads Sidecar and Restian libraries.
Here is example of a loader that loads the plugin:
– http://plugins.svn.wordpress.org/lexity-live-for-wp-e-commerce/tags/1.0.9/loader.php
The loader follows these rules: anything that is x.y is compatible with anything x+n.z is not compatible so it loads the largest “y” but won’t allow a plugin to be activated if another plugin it has a library incompatible with a library used by an activated plugin.
For example, if plugin A required v1.3 and plugin B is loaded that requires v1.7 then both A and B will use v1.7. However if plugin C requires v2.1 it can’t be activated if plugin A or B are active.
It works really well for me, but would be more robust if in core and could work for many if there was a well-accepted standard for library names and how versioning works.
-Mike
P.S. BTW, one of my first posts to the wp-hackers list was one where I proposed “modules;” it was that experience which taught me how unwelcoming open-source teams can be to ideas presented by newcomers. 🙂 FWIW.
Modules wouldn’t serve (in this case) as just extensions, but as experimental replacements of core APIs. Rather than forcing someone to patch WordPress (or install trunk) to test out an API change like an update to WP_HTTP or refactoring of WP_Query or a rewrite of WP_IXR_Client, you could just add a drop-in module that would substitute the new functionality.
It wouldn’t be a perfect solution at first, but would give us the opportunity to gradually refactor WP’s internals away from require() so that every API/module/component/library component becomes pluggable and therefore replaceable. In the long term, it might also pave the path for making it easier to unload unneeded modules … but that’s not the use case I’m looking at just yet.
Ah, as a method for feature branching core code, that makes sense.
No comments on my discussion of need for version managed library loader?
Versioning libraries is a bit beyond the scope of what I want to accomplish here. You’ve done some great stuff with it so far, and I’ve dabbled in it as well. But in terms of getting library/component management into core I think baby steps are easier for the community to take than huge swaths of rewrites.
I’m also a bit unconvinced that dynamic versioned library management is truly feasible in the world of interpreted code. With compiled binaries, your libraries are usually bundled together with the project. Interpreted code, or anything with shared libraries usually just has the one version available for sharing. If your code is tightly coupled to a specific version of a library, I’d argue that library should be part of the project itself and not a shared one.
Like I said, though, this is a bit outside the scope of what I’m trying to accomplish at the moment and deserves a much broader discussion. I’m perfectly willing to change my opinion on versioned libraries, but that’s a conversation for another day.
The problem is that WordPress provides no way for plugins to share *library* code other than code that is in core vs. end user code which get shared as plugins already.
This current state really stifles the potential innovation that could be occurring in the WordPress development community as it means each plugin has to reinvent the wheel. Because there is no an easy way to share code WordPress plugin developers have a culture of not reusing other’s code yet most other developers are benefitting from the explosion of code reuse on GitHub.
In other languages like Ruby they have reusable code like Gems because site development team are typically also the ones who do deployment. In WordPress deployment is done by the user so the developers needs a mechanism to allow them to use a shared library and have it co-exist with someone else’s use of the shared library that they know nothing about.
As for baby steps, the fact I was able to implement as a working plugin is a much smaller baby step then converting WordPress core into a collection of modules (not that I disagree with modules; I think they would be great.) Core could literally take what I have written and roll into core with some small changes to core to make it more robust. That’s a pretty tiny baby step to me.
As for your comment that a library should be part of the project, that’s exactly what I’m proposing. A plugin contains all required libraries, but if another plugin happens to use the same library then only one of them is loaded, whichever is the most compatible version. In the plugin I used as an example above, note the libraries are bundled at their required versions:
– http://plugins.svn.wordpress.org/lexity-live-for-wp-e-commerce/tags/1.0.9/libraries/
Only ever have one version of a library? I know you well enough to know that you must see the conceptual problem with that. All code has bugs, some have just not been surfaced yet. If my plugin exposes the bugs of a library whereas yours plugin doesn’t, mine needs a newer fixed version of the library and currently if yours gets to load the lib first my plugin will break.
That said; if versioned libraries are a conversation for another day, which day? 🙂
P.S. It’s issues like these that are making me seriously considering moving on from WordPress to greener Javascript-based pastures. As I’ve seen while learning Backbone, Javascript library developers seem to recognize the requirements of building applications that most of the WordPress community seems to deny. We need to enable code reuse would solve real world problems and allow WordPress to improve a lot more rapidly yet I feel like I get deer-in-the-headlights when I talk about it to WordPress developers. And I don’t really mean you, you are generally one of the best in the WP community, I just mean in general.
WP Touch Pro 3 has adopted this model. If you want to have CPT’s you load the CPT module. If you want a custom font library, you register the module with the plugin and it calls it.
This concept make total sense. Many of us are already building our plugins with modular overrides to allow for other developers to use their imagination. Having this ability in core would really allow us to program outside the box.
Nice work
I can’t program my way out of a paper bag, but I understand the logic and flow and this makes total sense. Ideas like this that really push WordPress forward excite me.
This is one of the biggest things I don’t like about WordPress. It’s so hard to remove core functionality, even things such as disabling comments and posts completely for a CMS site.
So this is something a bit above/below DropIns, pluggables and filters? Something like “a poor mans autoloader”? No version checks? Eric, as much as I appreciate your effort, but it sounds like you’re building something like a wheel, while rest of mankind is already flying since a decade. Wouldn’t it make more sense to just push to PHP 5.3 and get composer working in core? Wouldn’t that make all that efforts unnecessary?
5.3 isn’t going to happen in core any time soon. I was pushing for it to happen by 3.9 or 4.0, but even that seems like a long shot. Beyond that, I’m not talking about Composer. Yes, I want Composer in core, but for entirely different reasons.
That was the actual question. The rest was just the usual rant 🙂
Something above drop-ins, but not an autoloader. A system to selectively register, override, and load core files without directly calling require(). It will allow you to temporarily replace core files with new versions, but without hacking core itself.
I concur. Modular is the way forward. Fragmentation in the user base will force unmanageable bloat if the application is not partitioned to better suit individual requirements and preferences.