Compared to a commercial environment, open-source developers don’t have access to the same tools, nor the ability to hand pick and hone development teams over many years.  Debugging, optimisation and code tracing tools are simply not the same.  Contrast to a Delphi environment costing thousands of dollars per developer with full production grade debugging support and the ability to use commercially purchased components – the open source development team (if not necessarily the individual developer) must use a mixture of tools not necessarily purpose built to work together.  Teams self select, although inevitably filtering occurs over time, and project direction is committee based and therefore more difficult to keep focused than CEO based direction.  So the standard achieved by all of these components is remarkable.

In our case the instability seemed to be caused by insufficient memory being allocated to PHP.  We upped it to 256MB and the intermittent page failures disappeared.  I suspect we will need to go higher than that as the usage grows – or possibly lower, depending on whether the space is session based or shared. 

I am not sure how the PHP memory management interacts with the OS memory management, not whether the PHP memory space is shared between PHP applications, between sessions or specific to each session or application.  We, therefore, will adopt an approach of incremental growth/tuning until we have had a chance to read up on the PHP memory management system and had more time to understand the loads being placed on the servers.  I suspect that the PHP memory model differs between CGI and in-process implementations implementations anyway.   We have a number of large database and graphical processing systems on the dev cluster  so I don’t want to unnecessary rob memory from those systems, otherwise we would just allocate a few gigabytes to the PHP systems.  Based on the observed behaviour, I suspect it is shared between sessions and apps within the same inprocess server (which would make sense).  Whether is is assigned, or meerely available is another issue, and not clear at this point, nor whether unused memory is free to paged out of RAM or returnes to the OS when not required.

We are noting that we tend to have to specifically instruct the PHP apps to tell browsers not to store the pages in the caches (see the earlier post).  I think this is probably a good point for PHP programmers to note.  There are very few scenarios where a PHP web page should be cached – because by definition, they are generated – so it is a good idea to ensure that the header sent by the PHP page includes a ‘header(“no-store”);’ instruction by default – unless you specifically want it to be cached on the browser.

I must say: the more I look into the architecture and code of the WordPress system, the more impressed I am.  What is going on behind the scenes to display this blog to you is simply outstanding.  It is not so much the display method that is so clever (although that is clever enough), but the architecture and the administration/maintenance and update mechanisms are nothing short of brilliant.  There have been some very, very talented people working on this application.

The BPC dev team site (where this blog and the other interactive apps are located) does not have as much comms bandwidth as the main web site, so we have to be a little circumspect with the comms weight of apps we put here.  The nature of this site is that it hosts our production and experimental apps that face the internet and therefore has highly variable bandwidth consumption and is under a lot of stress.  On the plus side, it gets 24*7 human monitoring because it has to have high availability and is constantly being updated. 

We dropped a second hi-speed link into the dev site a year ago but have retained the slower link as the main comms path as it is 99% reliable, whereas the higher speed link still has emotional problems with the local exchange.  So all up, we need the apps we put here to be as light on comms overhead as possible.

It was quite an issue deciding which way to go.  The riskwiki (using media wiki) running on php and MySQL for about a year or two has proven to be very reliable and reasonably ‘tinker-able’.  As a production language PHP is not my first choice, because like most web languages (JavaScript, PHP, ASP Markup, HTML, DHTML, Java, CSS) it is a syntactic design disaster that begs coding errors.  This is not to pass judgement on these languages from the perspective of performance, versatility, or reliability.  Syntactically PHP is better than most, but still far from perfect.   By syntax we mean the grammar of the language.

On the plus side, PHP is incredibly easy to use, and lends itself to some very neat template based layering decisions and design patterns, works well with HTML/CSS/JavaScript/Ajax based clients (like browsers), gives instant feedback during development and is incredibly cross-platform portable.  So as an open source development language it has some advantages.  Debugging, however, can be a nightmare. 

So when we investigated the blogging options and came across WordPress we were comfortable with it using both PHP and MySQL on the same server as our wiki. 

The previous post discussed our experience in getting the WordPress system to work.  Except for the bug we had to deal with, it was an extremely simple exercise.  If we had not had that issue to address, it would have taken about 30 minutes – if that.   Similarly, setting up the RiskWiki a year or so ago was a fast and simple exercise.

One of the really impressive things about WordPress is the astounding range of quality themes and plug-ins available for it at no cost.  The plug-ins expand the functionality of the WordPress system to be way more than a simple blogging application.  There are plug-ins for e-commerce (turning your WordPress into an online shop, petition management (for lobbyists), content management (for web site control), and even a bulletin board/forum plug in that turns WordPress into – you guessed it – a bulletin board.

So when we turned to the question of what system to use for our forum, naturally we first considered using the WP-Forum (or rather its later cousin) for our bulletin board.  In the end, however we decided that it was best to use an application dedicated to an intended purpose rather than a plug-in that converts a blog engine into a bulletin board, a purpose for which the underlying software is possibly not optimised.  It was a close thing however, and we may yet go back and change to the Forum plug-in for WordPress.

The bulletin board we settled on was MyBB – for no particular reason other than it had a plug in for WordPress (that we aren’t actually using yet) and had a long development history, an apparently vibrant development community and seemed to have a huge range of management capabilities available for it.  Also – critically for us – it runs on PHP 5 and MySQL – so it matched the existing open-source software environments used by our RiskWiki wiki and RiskThink blog platforms.

The installation process was similarly straight forward – except that we had to make a few edits to some of the admin pages similar to the fix for WordPress to ensure the browsers were not caching some of the PHP files.  (I am starting to wonder whether IE has changed the caching model with recent security patches).

Now the tricky bit – we now have 3 systems using PHP and MySQL – all open-source.  Our apps of course use Delphi and MS SQL – which means they are all compiled (not interpreted apps). 

When we just had the two systems – RiskWiki and WordPress the shared environment was 100% reliable.  Now that we have added the third – MyBB we are getting periodic page drops (As you may have noticed). Necessitating the server to step in and restart the offending application.  This is particularly likely to occur when you jump from one to another, rather than going via a static web page.  Assuming there is nothing intrinsically wrong with any of the apps, we are thinking that it is a either a MySQL or PHP problem. 

We have turned on full logging to see if we can trace what is happening and hopefully we will have a solution in a few days. 

More on this to come…

The latest version of WordPress is really very good, and once it is working properly is really quite impressive; but in our environment we had some serious issues that made it non-functional initially, requiring two days of solid debugging of the code.

While the general public part of the blog worked fine, the administration screen was essentially non operational.  Various menus would not display on the browsers (IE7 IE8 and FF 2.0) and the blog authoring screen would display properly only the first time a blog was authored on any given browser.  The second and later times, the button would disappear, and the text was white (making it invisible).   The public pages could not be configured properly as the widget drag/drop mechanism stubbornly refused to drag and drop, and the accessibility menu (the alternative widget configuration mode) would not display.

Searching the WordPress forum site and documentation revealed scores of people with the same problem (but by no means universal) and no practical solutions for solving it – except it a few cases where plug-ins were involved, and where the user was using google gears.  We had not plug-ins yet, and are not prepared to install google gears.  So we were faced with either removing the software or debugging it (is is open-source after all!).  So we decided to debug it instead.

Now thinking about what was happening we noted that, firstly, we did not install a little optional tool offered by the WP installer called Google Gears which caches scripts, etc on the browser to improve performance. It emerged later that, for some users, installing google gears fixed their widget-drag drop problem.  We excluded this because our development and support computers have to be as close to yours as we can get them – and larger corporates are not in the habit of allowing modifications to browsers, which Google Gears represents.  Secondly we have been pretty much stripping out google junk from most of our dev systems after we discovered some of there libraries in unauthorised locations (like their very annoying web address redirector that takes you to a google add page when the browser fails to find the site you entered because you typed a single character wrongly).  Lastly, we are increasingly using MS’s Bing since it is proving to be a better search engine than google lately – no advertisements and more accurate searches.  We started wondering if the people without the WP problems had all installed Gears.

Further thinking about the behaviour led us to test what happened when we deleted the browsing history. Surprise, surprise – the blog authoring page worked the first time after the browsing history was deleted – then became dysfunctional on the second access.  So clearly we were looking at some kind of caching anomaly.  Caching is the mechanism whereby the browser stores stuff locally to speed browsing and, theoretically, getting fresh copies if it has changed on the server.  This works fine with static web pages and images, but is less effective when pages are dynamically generated, but seem to have the same name.

So we turned on the debuggers and looked at what was happening.  Looking at the WP page code, however suggested that everything should be fine.  The place where, and the object about which, the debugger was complaining was theoretically correct from the perspective of caching in that it was uniquely named with each access.  Yet the debugger and the browser insisted that they could not find the object bing referenced and housed in the offending script. 

So we looked around on the server code and discovered that the WP programmers had done something a little odd, that in theory should not have mattered.  The WP app generates JavaScript libraries dynamically using a list of required libraries fed to a php page called load-scripts.php.  Most web apps tend to ether have a static JavaScript library that is fed to the browser once for all pages or use some mechanism of creating a dynamic library on a page by page basis. WP uses a shared php script that receives a list of required scripts  for that purpose.  To ensure updating of the script list, the calls to the load-scripts file added a parameter with random  string in the call line – so it should be refreshed each time the calling page executes.  Now, in the \admin\load-scripts.php file they had inserted a header command that defined to the browser the caching rules for the JavaScript code dynamically constructed by the php script.  This is where it got strange – the caching rule said that the generated page should expire some time 12 months from now – but the load-scripts call lines in the hosting pages were clearly intended to cause the load-scripts page to be refreshed with each access (which makes sense for a dynamically generated JavaScript library).  So the browser was being told to keep the script indefinitely, while the access model was designed to ensure it would never be required again.

Not Right! (As my four year old son would say.)  On the face of it, this should not matter – it is just wrong, but the browser should go away and get the scripts again anyway based on the uniqueness of the calling line, but either the browsers all share a common bug, or they are smarter in the cache management than we realise, because they were clearly not getting the scripts again – they had, after all, been told that library of scripts would be current until 2010, but then the DOM in the browser could not find the JavaScript objects, because they had not been retrieved for the current instance of the browser page, and the library containing them was not the same name as the expected one (or something like that!).   Either way, the fix was straight forward – change the caching rule to match the way the load-scripts.php page is called: 

1. Locate the file “\WordPress\wp-admin\load-scripts.php”
2. Open it in a text editor visual studio or equivalent is preferred, but notepad will do. DO NOT USE MS_WORD or similar word processing tool.
3. Locate the following line:
header(“Cache-Control: public, max-age=$expires_offset”);
4. Comment this line out so it looks like this:
/* header(“Cache-Control: public, max-age=$expires_offset”);
*/
5. Add in the following line
header(“Cache-Control: public, no-store”);
6. Double check that you have written the above line EXACTLY as shown.
7. Save the file
8. Now go to your browser and delete all browser file history (sorry, but this is essential or you won’t see the fix.). On IE 7 you do this by:
a. From the menu bar: Tools/Internet Options
b. Select the General tab
c. Under browsing history choose “delete”
d. On the window that opens, choose “Delete Files”
e. Confirm and close windows
f. Close ALL open browsers and restart the browser.

The Result:

All menus in the admin panel now work, widgets can be dragged and dropped and blog editing works correctly – in IE 7, IE 8 and FireFox 2.0

Simple.