The Acer Aspire Easystore NAS units come in a variety of versions.  The latest unit uses a windows OS, but the earlier versions used a dumbed down Linux OS.  The units are, if not especially cheap, at least low priced.  This article is about the earlier versions using the Linux OS and recounts how we recovered our data from a failed unit.

The problem with the Linux based Acer Aspire Easystore is that while the unit will happily allow drives to be replaced, if you lose the motherboard or the Ethernet port you are in serious trouble.   It surprisingly easy to suffer a catastrophic problem.  For example, if you upgrade the firmware from 2.0 to 2.5 your unit can stop responding.   Further you can lose the motherboard  for a variety of  other reasons – for example, they seem especially sensitive to minor DC power spikes.  If the dreaded red light indicating “unrecoverable software” lights up or the unit simply stops talking to the network, you will rapidly discover you  are essentially screwed. 

Acer’s maintenance support, seems excellent when you contact them.  They will happily replace the unit for you (in fact that is all they will do!) and even send out a technician to swap your old still working drives into the new unit.  Unfortunately that is when you discover you have a serious problem.

The replacement unit will not read the RAID array from the old drives – even if your drives are otherwise OK.  If you are lucky, it will recognise the user accounts – but it will not give you back access to your data.  Acer’s helpful response is that you must rebuild the array.  Do not under any circumstances do this or you will lose your data!

This is what you must do…

In our case we always use RAID 1 on low cost RAID systems.  This essentially gives 2 copies of each disk without co-dependence across multiple disks for data recovery.

Recovering your data from the Acer becomes quite simple in this situation.

What you will need:

A spare computer (a Windows desktop is fine) with:

•  at least one spare SATA port,
• a spare SATA cable, (or just unplug an existing drive from the desktop computer and use its cable),
• a spare HD drive  in FAT32 or Linux disk format so it can be accessed easily under Linux (or other NAS on the network) with enough space to store the files you recover from your Acer Easystore drive(s)
• a CD ROM. 

Either the on the same  machine, or another, you need a writable CD drive, a spare blank CD and an internet connection.

Now follow these steps:

Step 1  Assuming you do not have a way to write ISO formats onto a CD ROM go here and download the ISO writer:

http://isorecorder.alexfeinman.com/isorecorder.htm

Just double click to install it and once installed you merely right-click on an ISO image and choose “Copy image to CD”.

Step 2: Get a copy of the latest version of Knoppix Linux from here (or other Knoppix mirror).   I used Knoppix 6.4.4

http://www.mirrorservice.org/sites/ftp.uni-kl.de/pub/linux/knoppix/

You do want the ISO image and unless you are visually impaired you do NOT need the ADRIANE version.

Step 3.  Put your blank CD into your writable CD drive and right click on the ISO image you just downloaded.  Choose “copy image to CD”.  This will give you a bootable version of Knoppix Linux on a CD ROM.

Step 4.  Turn off and open up your spare desktop computer and plug the first RAID1 drive from your dead Acer Aspire Easystore into the SATA data cable and the SATA power cable and plug the other end of the SATA data cable into a spare SATA port on the motherboard (or SATA sub-board).  Note the number written on the motherboard this will help you find the drive later.  You only need to plug in one drive from each RAID1 pair.  The drives are arranged in pairs making one volume each in the easystore – so the top two are one volume and the bottom two are another volume.  You want one drive from each volume.  We will just do one volume in this sequence, and you should repeat the following steps for the second volume after completing the first.

Step 5.  Turn on the computer and if it does not already boot from a CD ROM, trigger the BIOS setup on startup and change the boot order so that it boots from a CD ROM first.  Save and exit and allow the computer boot up.

Step 6.  Once Linux has started (you may have some minor config steps to work through), click on the little filing cabinet in the bottom left hand corner of your screen.  This will show you the mounted devices.  If your machine also has NTFS drives in it or even an existing RAID array in it there will be some drives marked OS which you will not be able to access.  Ignore these.  Your FAT32 drives and your Acer EasyStore Raid1 drive should be visible.  The mounted Acer drive will show as something like “sdc4″, where the “sdc” part might be sda, sdb, sdc, etc.  The number refers to the partition on the drive that could be read.  This will not have your data in it – so don;t get excited just yet.

Step 7.  In the bottom left hand corner of your screen, beside the filing cabinet icon is the Knoppix logo.  This is the equivalent of the windows “start menu”.  Click on this and from “preferences” choose “GParted”.  Let this do its stuff and when it has finished opening select the GParted menu, and from that the “Devices” submenu.  From this look for and select your Acer drive.  In my case it was on /dev/sdc -  yours may be different – but will be the one that was showing the “4″ in its name when you looked in the file system display.

Step 8.  After GParted has scanned it, you will notice this has 4 partitions.  The Linux swap and the system – both of which will be mounted and then two others – which will not be mounted.  The big one has your data, and the little one is just some spare space.  The big one will probably have a “2″ in its name.  In my case it was “sdc2″.  It will also be showing the file system type – probably “ext2″.  Note both the name (eg. sdc2) and the file system (eg “ext2″) – you will need them in a second.

Step 9.  Select this line by left clicking on on it and then right click to bring up the context menu.  From the context menu that displays choose “Manage Flags”.

Step 10.  In the “manage flags”  window that opens tick the “Raid” flag.  Then open the “Manage Flags” menu again and untick the “Raid” flag.  Close GParted.

Step11. From the “start menu” in the lower left hand corner of your screen choose “Accessories / Root Terminal”. 

Step 12.  When the console window opens type the following:

mount   /dev/sdc2   /media/sdc2 -oro -text2

NOTE:  change the “sdc2″ to the device name (and partition number) you saw in GParted earlier and the  “ext2″ file system type to the one you saw there as well.  It will most likely be ext2, but it might be ext3.   You have just mounted your lost data as “/media/sdc2″  – or whatever sd you entered above.

Step 13.  Click on the filing cabinet icon in the bottom left of the screen and in the address bar enter (adjusted appropriately for your device name):

/media/sdc2/

Your lost files should now be visible. 

Step 14:  Copy your lost files to your spare drive or other network store.   The Acer drive was opened in read-only mode by the mount command so you will can only copy or read them.

If you have two RAID1 volumes in your Acer EasyStore you should repeat all the steps  from step 4 for the second volume as well.

Good luck!  Feel free to drop me an email (or a comment here) if you have a question.  You can also rebuild a RAID5 array in a similar fashion – but it is a little more complex than the process for RAID1 recovery.

It is a few days after my post on the topic of getting the three open source applications – MediaWiki, WordPress and MyBB – all written in PHP using MySQL and running on Windows under IIS.   We seem to have reached an good level of stability since then – not perfect, but seemingly about 97%.  Considering the code for these three systems (and the code for the PHP and MySQL engines) is multi platform and open source, I think that is a pretty impressive outcome.  I am both surprised and impressed.

Compared to a commercial environment, open-source developers don’t have access to the same tools, nor the ability to hand pick and hone development teams over many years.  Debugging, optimisation and code tracing tools are simply not the same.  Contrast to a Delphi environment costing thousands of dollars per developer with full production grade debugging support and the ability to use commercially purchased components – the open source development team (if not necessarily the individual developer) must use a mixture of tools not necessarily purpose built to work together.  Teams self select, although inevitably filtering occurs over time, and project direction is committee based and therefore more difficult to keep focused than CEO based direction.  So the standard achieved by all of these components is remarkable.

In our case the instability seemed to be caused by insufficient memory being allocated to PHP.  We upped it to 256MB and the intermittent page failures disappeared.  I suspect we will need to go higher than that as the usage grows – or possibly lower, depending on whether the space is session based or shared. 

I am not sure how the PHP memory management interacts with the OS memory management, not whether the PHP memory space is shared between PHP applications, between sessions or specific to each session or application.  We, therefore, will adopt an approach of incremental growth/tuning until we have had a chance to read up on the PHP memory management system and had more time to understand the loads being placed on the servers.  I suspect that the PHP memory model differs between CGI and in-process implementations implementations anyway.   We have a number of large database and graphical processing systems on the dev cluster  so I don’t want to unnecessary rob memory from those systems, otherwise we would just allocate a few gigabytes to the PHP systems.  Based on the observed behaviour, I suspect it is shared between sessions and apps within the same inprocess server (which would make sense).  Whether is is assigned, or meerely available is another issue, and not clear at this point, nor whether unused memory is free to paged out of RAM or returnes to the OS when not required.

We are noting that we tend to have to specifically instruct the PHP apps to tell browsers not to store the pages in the caches (see the earlier post).  I think this is probably a good point for PHP programmers to note.  There are very few scenarios where a PHP web page should be cached – because by definition, they are generated – so it is a good idea to ensure that the header sent by the PHP page includes a ‘header(“no-store”);’ instruction by default – unless you specifically want it to be cached on the browser.

I must say: the more I look into the architecture and code of the WordPress system, the more impressed I am.  What is going on behind the scenes to display this blog to you is simply outstanding.  It is not so much the display method that is so clever (although that is clever enough), but the architecture and the administration/maintenance and update mechanisms are nothing short of brilliant.  There have been some very, very talented people working on this application.

It was a perplexing decision as to which BB software to use.  The most reliable system we have been involved in has been the team.bishopphillips.com site which uses MS Sharepoint.  The upsides to this system include incredible versatility;  that it is MS SQL  Server based and therefore a technology we know really well; and incredibly reliable – it just never goes down.  The downside is that it is a bandwidth hungry beast and every page sends a small encyclopedia of data between the browser and the server.  It works a treat on the intranet, but it is possibly less ideal for internet usage where bandwidth has to be shared among other heavy consumers.

The BPC dev team site (where this blog and the other interactive apps are located) does not have as much comms bandwidth as the main web site, so we have to be a little circumspect with the comms weight of apps we put here.  The nature of this site is that it hosts our production and experimental apps that face the internet and therefore has highly variable bandwidth consumption and is under a lot of stress.  On the plus side, it gets 24*7 human monitoring because it has to have high availability and is constantly being updated. 

We dropped a second hi-speed link into the dev site a year ago but have retained the slower link as the main comms path as it is 99% reliable, whereas the higher speed link still has emotional problems with the local exchange.  So all up, we need the apps we put here to be as light on comms overhead as possible.

It was quite an issue deciding which way to go.  The riskwiki (using media wiki) running on php and MySQL for about a year or two has proven to be very reliable and reasonably ‘tinker-able’.  As a production language PHP is not my first choice, because like most web languages (JavaScript, PHP, ASP Markup, HTML, DHTML, Java, CSS) it is a syntactic design disaster that begs coding errors.  This is not to pass judgement on these languages from the perspective of performance, versatility, or reliability.  Syntactically PHP is better than most, but still far from perfect.   By syntax we mean the grammar of the language.

On the plus side, PHP is incredibly easy to use, and lends itself to some very neat template based layering decisions and design patterns, works well with HTML/CSS/JavaScript/Ajax based clients (like browsers), gives instant feedback during development and is incredibly cross-platform portable.  So as an open source development language it has some advantages.  Debugging, however, can be a nightmare. 

So when we investigated the blogging options and came across WordPress we were comfortable with it using both PHP and MySQL on the same server as our wiki. 

The previous post discussed our experience in getting the WordPress system to work.  Except for the bug we had to deal with, it was an extremely simple exercise.  If we had not had that issue to address, it would have taken about 30 minutes – if that.   Similarly, setting up the RiskWiki a year or so ago was a fast and simple exercise.

One of the really impressive things about WordPress is the astounding range of quality themes and plug-ins available for it at no cost.  The plug-ins expand the functionality of the WordPress system to be way more than a simple blogging application.  There are plug-ins for e-commerce (turning your WordPress into an online shop, petition management (for lobbyists), content management (for web site control), and even a bulletin board/forum plug in that turns WordPress into – you guessed it – a bulletin board.

So when we turned to the question of what system to use for our forum, naturally we first considered using the WP-Forum (or rather its later cousin) for our bulletin board.  In the end, however we decided that it was best to use an application dedicated to an intended purpose rather than a plug-in that converts a blog engine into a bulletin board, a purpose for which the underlying software is possibly not optimised.  It was a close thing however, and we may yet go back and change to the Forum plug-in for WordPress.

The bulletin board we settled on was MyBB – for no particular reason other than it had a plug in for WordPress (that we aren’t actually using yet) and had a long development history, an apparently vibrant development community and seemed to have a huge range of management capabilities available for it.  Also – critically for us – it runs on PHP 5 and MySQL – so it matched the existing open-source software environments used by our RiskWiki wiki and RiskThink blog platforms.

The installation process was similarly straight forward – except that we had to make a few edits to some of the admin pages similar to the fix for WordPress to ensure the browsers were not caching some of the PHP files.  (I am starting to wonder whether IE has changed the caching model with recent security patches).

Now the tricky bit – we now have 3 systems using PHP and MySQL – all open-source.  Our apps of course use Delphi and MS SQL – which means they are all compiled (not interpreted apps). 

When we just had the two systems – RiskWiki and WordPress the shared environment was 100% reliable.  Now that we have added the third – MyBB we are getting periodic page drops (As you may have noticed). Necessitating the server to step in and restart the offending application.  This is particularly likely to occur when you jump from one to another, rather than going via a static web page.  Assuming there is nothing intrinsically wrong with any of the apps, we are thinking that it is a either a MySQL or PHP problem. 

We have turned on full logging to see if we can trace what is happening and hopefully we will have a solution in a few days. 

More on this to come…

The Problem:

The latest version of WordPress is really very good, and once it is working properly is really quite impressive; but in our environment we had some serious issues that made it non-functional initially, requiring two days of solid debugging of the code.

While the general public part of the blog worked fine, the administration screen was essentially non operational.  Various menus would not display on the browsers (IE7 IE8 and FF 2.0) and the blog authoring screen would display properly only the first time a blog was authored on any given browser.  The second and later times, the button would disappear, and the text was white (making it invisible).   The public pages could not be configured properly as the widget drag/drop mechanism stubbornly refused to drag and drop, and the accessibility menu (the alternative widget configuration mode) would not display.

Searching the WordPress forum site and documentation revealed scores of people with the same problem (but by no means universal) and no practical solutions for solving it – except it a few cases where plug-ins were involved, and where the user was using google gears.  We had not plug-ins yet, and are not prepared to install google gears.  So we were faced with either removing the software or debugging it (is is open-source after all!).  So we decided to debug it instead.

Now thinking about what was happening we noted that, firstly, we did not install a little optional tool offered by the WP installer called Google Gears which caches scripts, etc on the browser to improve performance. It emerged later that, for some users, installing google gears fixed their widget-drag drop problem.  We excluded this because our development and support computers have to be as close to yours as we can get them – and larger corporates are not in the habit of allowing modifications to browsers, which Google Gears represents.  Secondly we have been pretty much stripping out google junk from most of our dev systems after we discovered some of there libraries in unauthorised locations (like their very annoying web address redirector that takes you to a google add page when the browser fails to find the site you entered because you typed a single character wrongly).  Lastly, we are increasingly using MS’s Bing since it is proving to be a better search engine than google lately – no advertisements and more accurate searches.  We started wondering if the people without the WP problems had all installed Gears.

Further thinking about the behaviour led us to test what happened when we deleted the browsing history. Surprise, surprise – the blog authoring page worked the first time after the browsing history was deleted – then became dysfunctional on the second access.  So clearly we were looking at some kind of caching anomaly.  Caching is the mechanism whereby the browser stores stuff locally to speed browsing and, theoretically, getting fresh copies if it has changed on the server.  This works fine with static web pages and images, but is less effective when pages are dynamically generated, but seem to have the same name.

So we turned on the debuggers and looked at what was happening.  Looking at the WP page code, however suggested that everything should be fine.  The place where, and the object about which, the debugger was complaining was theoretically correct from the perspective of caching in that it was uniquely named with each access.  Yet the debugger and the browser insisted that they could not find the object bing referenced and housed in the offending script. 

So we looked around on the server code and discovered that the WP programmers had done something a little odd, that in theory should not have mattered.  The WP app generates JavaScript libraries dynamically using a list of required libraries fed to a php page called load-scripts.php.  Most web apps tend to ether have a static JavaScript library that is fed to the browser once for all pages or use some mechanism of creating a dynamic library on a page by page basis. WP uses a shared php script that receives a list of required scripts  for that purpose.  To ensure updating of the script list, the calls to the load-scripts file added a parameter with random  string in the call line – so it should be refreshed each time the calling page executes.  Now, in the \admin\load-scripts.php file they had inserted a header command that defined to the browser the caching rules for the JavaScript code dynamically constructed by the php script.  This is where it got strange – the caching rule said that the generated page should expire some time 12 months from now – but the load-scripts call lines in the hosting pages were clearly intended to cause the load-scripts page to be refreshed with each access (which makes sense for a dynamically generated JavaScript library).  So the browser was being told to keep the script indefinitely, while the access model was designed to ensure it would never be required again.

Not Right! (As my four year old son would say.)  On the face of it, this should not matter – it is just wrong, but the browser should go away and get the scripts again anyway based on the uniqueness of the calling line, but either the browsers all share a common bug, or they are smarter in the cache management than we realise, because they were clearly not getting the scripts again – they had, after all, been told that library of scripts would be current until 2010, but then the DOM in the browser could not find the JavaScript objects, because they had not been retrieved for the current instance of the browser page, and the library containing them was not the same name as the expected one (or something like that!).   Either way, the fix was straight forward – change the caching rule to match the way the load-scripts.php page is called: 

1. Locate the file “\WordPress\wp-admin\load-scripts.php”
2. Open it in a text editor visual studio or equivalent is preferred, but notepad will do. DO NOT USE MS_WORD or similar word processing tool.
3. Locate the following line:
header(“Cache-Control: public, max-age=$expires_offset”);
4. Comment this line out so it looks like this:
/* header(“Cache-Control: public, max-age=$expires_offset”);
*/
5. Add in the following line
header(“Cache-Control: public, no-store”);
6. Double check that you have written the above line EXACTLY as shown.
7. Save the file
8. Now go to your browser and delete all browser file history (sorry, but this is essential or you won’t see the fix.). On IE 7 you do this by:
a. From the menu bar: Tools/Internet Options
b. Select the General tab
c. Under browsing history choose “delete”
d. On the window that opens, choose “Delete Files”
e. Confirm and close windows
f. Close ALL open browsers and restart the browser.

The Result:

All menus in the admin panel now work, widgets can be dragged and dropped and blog editing works correctly – in IE 7, IE 8 and FireFox 2.0

Simple.

We have just turned on the new public BPC support & discussion forum at http://bpc.bishopphillips.com/forum/ .  It is new, so the cupboard looks a little bare at the moment, but we trust some of you will find it of use.  Whether we had a forum, has been a question that is frequently asked by new clients and although the team.bishopphillips.com site has been around for a few years, it is primarily used by BPC staff and is less easily accessed than the new forum.  

The reality is that our email response system has been so good and the robustness of the applications so high that there simply is not the demand for a bulletin board style help system.  Even though the team.bishopphillips.site has been available for client access, clients have almost universally opted simply to email issues and questions and get a response in a few minutes to a couple of hours.  We sweep the emails every month and add common issues, or issues we think are of public consumption to the FAQ in the riskwiki.  The fact that a traditional layer of support is really just not needed with BPC RiskManager  is difficult to explain to new clients and so we have decided to launch a site.

It is monitored continuously by the BPC Dev team so we should see stuff arriving pretty quickly.  Please remember that it is new and the software is a little experimental so it may have short outages from time to time.  If it is down when you access, try again in a few minutes and it should automatically restart in about 2 minutes.

The RiskWiki (MediaWiki), the Blog (WordPress) and the Forum (MyBB) all use the same underlying technology - PHP and MySQL.  The introduction of the Forum to that set has introduced some instability which we are still working through so periodically one causes the other to fall over.  Everything is on auto restart, so if something is down when you first access it will come back  up in a few minutes.  I am sorry about this, but hopefully over the next week we will find the culprit of the problem and get back to our normal 99.9% system reliability.