Weblog

"PHP Tools for Mobile Web Development" Published

This is just a quick heads up to say that my article, "PHP Tools for Mobile Web Development" has today been published, and is currently gracing the cover of July's php|architect magazine.

Of course, I jinxed things a little by blogging that it would be published in June, but never mind, we got there.

Big thanks must go to Ciaran for giving the initial draft the once over (on a related note, check out Ciaran's post about web development for the iPhone). Thanks also to my occasional colleague Gerard for clueing me in to the fact that the damn thing had been published.

For what it's worth, php|architect is recommended reading even when I'm not in it, so get yourself over there and get subscribed!

Ok...now to crack on with that second article...

Mobilising a Website, Part 2: Strategies

In Mobilising a Website, Part 1: The Problem I noted that this site is practically unusable when viewed using the browser on a mobile handset, and that I'd like to do something about that.

This time around, I'd like to size up some of the approaches and strategies that developers can take in order to make an existing website mobile-friendly.

Option 1: Do Nothing

It may sound flippant, but in the world of development, the option of doing nothing often has to be seriously considered. Every developer out there can recount tales of ridiculous amounts of time and expense being invested in hairbrained projects developing systems that simply were not needed.

This strategy is unparalleled in its cost-effectiveness and simplicity and is naturally very, very easy to estimate and plan.

In this particular case, we could point to the fact that, as we saw, some high-end handsets do a decent enough job of displaying the site as it stands. We can confidently expect mobile browser technology to improve immeasurably over the next couple of years, so there's a lot to be said for holding off.

Unfortunately, this strategy does not solve the original problem. I want my site to work nicely for mobile users now, and it will be a long time before browsers as capable as that of the iPhone dribble down to the majority of consumers.

Moreover, I'm as interested in the process of achieving a mobile-friendly site as I am in the finished article, and that's why we're here: this series of blog posts would be rendered rather brief and anti-climactic if I were to choose the "do nothing" route!

Option 2: Add a Mobile Stylesheet

Quite a few of the problems we saw in part one, in particular those which confounded the Nokia 6230i, were related to the site's stylesheet. For example, both the fixed width of the page and the large banner images are defined in there. The former is the result of the rule width: 790px; being applied to a number of the elements within the page; the latter is specified by applying a background-image property to a <div />.

Fortunately, HTML and XHTML provide a way of specifying that a stylesheet only applies to a certain class of device. This is all achieved by applying a media attribute to the <link /> element which calls in the stylesheet.

By way of an example, the following code extract is designed to send one stylesheet to "full" or desktop web browsers (media="screen") and an alternative stylesheet to mobile browsers (media="handheld"):

<link rel="stylesheet" type="text/css"
		media="screen" href="full.css">
<link rel="stylesheet" type="text/css"
		media="handheld" href="mobile.css">

The full list of possible values for the media attribute can be found here.

Once we're serving a different stylesheet to mobile devices, we can really start to think about customising the user experience for them. For example, a mobile stylesheet could also be used to "hide" certain elements from mobile browsers, by applying a display: none; style rule to them. The long list of links in the sidebar which made the site so difficult to use with the W880i seems like a good candidate for hiding.

This strategy has the compelling benefit of simplicity. By adding one line of HTML, plus a small additional stylesheet to the site we may be able to deliver a mobile-friendly experience. Let's knock up a prototype mobile stylesheet and see how things look.

Here's a small CSS file that "hides" some of the more spurious elements of the page. I'll also specify that links should be displayed in green, purely to make it unambiguous as to whether the new stylesheet is being loaded and applied.

.sidebar {
	display: none;
}

a {
	color: green;
}

Remember that simply by not specifying widths, colours and background images in there, we're sidestepping a lot of the decorative fluff that was causing problems for mobile browsers.

Viewed on the Sony Ericsson, things are looking up:

Fig 1: Pointbeing.net with a handheld stylesheet, viewed on a Sony Ericsson W880i

We're successfully displaying only the main navigation, an introductory heading and paragraph, 10 links to blog posts and a footer. It's actually a really clean user experience for very little effort - I like it a lot.

Unfortunately, life isn't quite so cheerful for users of the other handsets. The experience on the Nokia 6230i and the iPhone is all but unchanged, but probably for different reasons. The Nokia does occasionally display links in green, so my hunch is that it simply does not support the media attribute, and so loads and applies both stylesheets. There's no corresponding property in WURFL to confirm it, but it certainly looks that way.

Conversely, I'm certain that the iPhone fully understands the media attribute, but considers itself to be a "real" computer. The jury's out on that one [1]. My gut feeling is that screen size should be the deciding factor in these cases, and based on that, the iPhone sits squarely under the heading of a handheld device. It's only fair to point out that the iPhone is not alone in this behaviour: the Nokia N95 and my LG KU990 behave in much the same way.

So the handheld stylesheet seemed like a great idea, but we can summarise several of the drawbacks to this approach as follows:

• Some devices do not support the media attribute at all
• Many high-end handsets completely disregard the media attribute and opt for the "full web" stylesheet
• In the cases where devices both support and honour the media attribute, we're relying on them to support CSS consistently. This is not something we can safely expect from mobile devices. For example, my first attempt was to use visibility: hidden; instead of display: none; but this was disregarded by the W880i
• Even if we hide elements by using CSS, the full page still has to be downloaded, which is likely to impose both a time and cost expense on the user. Expecting the user to download reams of markup which we don't actually want them to render seems like rather poor form. Furthermore, those full web pages may be larger than the maximum deck size which the device can support
• Pages are still not tailored to the mobile experience: Cameron Moll talks a lot about contextual relevance, and it's hard to see how my enthusiastic post about The Get Up Kids, consisting of three multi-megabyte YouTube videos embedded in the page is at all relevant to the mobile context

So this strategy isn't ideal. But for so little effort I've actually managed to make the site perfectly usable on a number of handsets. With little to lose, and with Early And Regular Delivery in mind, I'm actually going to put the handheld stylesheet in place right now, while I ponder alternative stategies.

Option 3: Allow the Site Automatically to Adapt to Devices

The principle behind this strategy is one known as "adaptive rendering". In other words, device detection would be done server-side (in my case using PHP) and the client would be sent markup and content tailored specifically to the device. I can think of a couple of ways to achieve this, although I'm sure there are plenty more.

The first option is afforded to us by the fact that the site is based firmly on the MVC pattern, which is fairly common in web sites and applications these days. MVC dictates that the business logic and data (Model), display logic (View) and application flow (Controller) be arranged into discrete components, so as to be independent of each other. In this case we're primarily concerned with the display logic, so it seems feasible to take advantage of the separation and swap in a different View component for mobile devices.

I happen to be using Zend Framework, so this could be achieved by specifying the directory in which the View should look for view scripts and helpers. I imagine it would not be difficult to do this dynamically using methods such as Zend_View_Abstract's setBasePath(), setScriptPath() and setHelperPath().

Another approach would be to adopt the "two-step view" pattern. This pattern is nicely documented by Martin Fowler in his classic Patterns of Enterprise Application Architecture, so I'll quote liberally from there:

Two Step View [splits] the transformation into two stages. The first transforms the model data into a logical presentation without any specific formatting; the second converts that logical presentation with the actual formatting needed. This way...you can support multiple output looks and feels with one second stage each

That sounds rather like what I need. I guess in terms of implementation, I'd be looking at having the first stage generating some common XML format, and then perhaps using XSL Transformations server-side in order to transform the XML into the markup which the device prefers. At the same time, I can opt not to include certain elements, such as the long list of links, in the finished pages.

Again, this can all be done within the View component of the site's code, which is rather gratifying. Still, that does seem like quite a lot of work, and I'm not sure I want to get into what would effectively be writing my own templating engine.

Option 4: Build a Separate Mobile Site

One surefire way of getting myself a working mobile version of the site would be to simply build a standalone mobile site. I've already registered the pointbeing.mobi domain name, and quite honestly do not have any better ideas for what to do with it.

The benefits of this approach would be that I could tailor the mobile site exactly the way I want it, I could roll out features incrementally, and I wouldn't risk making the main Pointbeing.net site's code any more complex than it need be. Admittedly it's pretty simple stuff right now, but I'd like to keep it that way.

I could detect mobile devices as they hit the main site (perhaps using Tera-WURFL) and forward them across to the mobile version. This is a strategy used by a fair number of large sites, such as Flickr (mobile version) and Facebook (mobile version), so it seems like I'd be in good company.

The downsides would be that I would have two sites to maintain, and that I still would not have solved the original problem, that of Pointbeing.net being a bit of a dog when viewed on a mobile browser. That said, if I leave the handheld stylesheet in place, I'd be catering to most cases.

The important thing to remember will be to always provide a link back to the full site for users who feel confident that their browsers will cope with it. I'm not interested in forbidding any users from accessing any content whatsoever: that's too much of a throwback to the dark days of the desktop web when sites would block, say, Opera users, demanding that they to "upgrade" to IE5 in order to gain access.

Conclusions

Creating a standalone mobile-friendly site - either under the .mobi domain or under an "m." subdomain - and forwarding mobile devices on to it is an appealing option, and it's the one towards which I'm leaning right now. I think...

I may well change my mind and opt for the adaptive rendering path. While that route feels a little more ambitious, I certainly would be quite proud if I could pull it off, and have the site adapt itself to devices as if by magic.

Either way, I'm going to have to put some thought into which tools and libraries I'm going to use to create mobile-friendly pages. There's a few out there, some of which I've covered on this site, some in a recent piece I wrote for php|architect, and yet others with which I'm not at all familiar.

Part 3 of this series seems like a good time to start making some decisions about toolkits, and this will in turn entail making some architectural decisions about the code behind the mobile version of pointbeing.net.

Footnotes

[1] Similarly, the debate about whether to deliver "full web" content or a mobile tailored version to such devices continues. A useful piece on the subject appeared recently over at WAP Review.

Many thanks to my colleague Dan Gent, whose remarkably well-timed loan of Cameron Moll's Mobile Web Design helped to inform this post.

Previous Posts in this Series:

• Mobilising a Website, Part 1: The Problem

Response to "10 Things a Developer Should Never Ignore"

Earlier this week, I stumbled across Bill Stronge's recent 10 Things a Developer Should Never Ignore over on TechRepublic. It's recommended reading, as it's an interesting piece, filled with useful advice for developers, especially those just getting started in their programming career.

Still, a couple of the points jarred with me a little, and there were a couple which I felt could have been taken further. So here's my response to Bill's 10 Things.

#1: Clarifying User Requirements

I agree with this point in principle, but it's important not to get bogged down in the requirements gathering phase. Try to understand your customer, but don't expect them always to know what they want before they actually see a product coming together.

Furthermore, expect requirements to change over time. This is a constant source of frustration for inexperienced developers. I know because I've been there myself, but over time you will come to expect it and embrace change.

I think this is where agile processes win hands down. Agile urges you to keep it simple, roll out small, incremental changes to a system, and adapt to changing requirements over time. The result is almost always a more valuable product, happier customers, and more satisfied developers.

#2: Collaborating

Agreed. We've all worked with the "hero" programmer, who churns out thousands of lines of code, and jealously guards it from the eyes and - heaven forbid - input of his or her colleagues.

Unfortunately, the only possible outcome of that mindset is libraries that no one else wants to use, and code that no one else wants to maintain.

You don't have to go as far as pair programming (though there's a lot to recommend about it), but just don't let your pride get in the way of talking your ideas through with other developers, asking questions and listening to advice.

#3: Version Control

Agreed. Get your code, configuration files and even documentation into version control and feel the weight lift from your shoulders.

Furthermore, learn how to use version control well. Learn how to use tags and branches properly, and read up on more advanced features, such as svn externals. On this subject, I strongly recommend the Pragmatic Version Control books.

Finally, if you're only familiar with a graphical VC client, such as Tortoise, learn how to do it all from the command line. You'll find it quicker, simpler, and a great deal more powerful.

#4: Basic System Testing

I would take this point a lot further and recommend unit testing and Test-Driven Development. In short, unit testing is writing code to test your code, and Test-Driven Development (TDD) is simply writing the tests before the code.

Bill states that most developers hate testing, but I believe that they only think they do. Programmers love programming, and I like to think of unit testing/TDD as a programming technique rather than a testing technique. In fact, designing and coding a thorough and yet flexible test suite can often be as satisfying a challenge as the project you're actually delivering.

The investment of a little time up front to write some small tests typically pays back a hundredfold in terms of productivity. Time spent debugging is slashed, and you have the pride of delivering code in which you have complete confidence. Furthermore, code built using TDD invariably tends to be cleaner, terser and more flexible than testless code.

Once you become what's known as "test-infected" you're very unlikely ever to voluntarily return to the pain of untested and untestable (also known as detestable) code!

#5: Usability

It's hard to argue with this, but it's even harder to define 'usable'.

Programmers are notoriously hopeless at usability, especially the ones who think they're great at it. You only have to look as far as the clunky Ajax-heavy interfaces of trendy Web 2.0 sites such as Spoonfed [1]for evidence of that. Someone clearly had a whale of a time coding that site, but it's painful for the end user.

Being a programmer myself, I don't have a great number of pearls of wisdom to share when it comes to usability. The best advice I can give is to exercise some self-restraint, keep it simple - boring even - and your users will be far less likely to want to strangle you.

#6: System Performance

Nobody likes slow sites and applications. But please, please pay close attention to the three rules of optimization: Don't, Don't yet, and Profile Before Optimizing.

I don't know if I'd go as far as the commentator who is quoted as saying:

More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity

After all, blind stupidity is pretty prevalent. But I certainly have seen some terrible code written, and some deeply regrettable architectural decisions made in the name of optimization. Quite often those "optimizations" tend to lead to far worse performance.

In my experience, two criteria must be in place before you can justify attempting any kind of optimization: i) you must be able to prove that you have a performance problem - anything else is a waste of your time and a waste of your employer's money; and ii) you must be able to measure the problem. If you can't measure the problem, you can't prove that your whizzy optimizations haven't just made things worse.

#7: Comments in Your Code

Time and time again, we're told that comments are a Good Thing, full stop. But the situation is a little less black and white. Comments need to be appropriate.

We've all seen this kind of thing trotted out as an example of poor commenting:

That's plainly stupid, but examples like this tend to miss the real point.

As a rule of thumb, if the code you're writing is complex or obscure enough that you need comments simply to explain what it does, then that piece of code has bigger problems than can be solved just by adding a comment.

In his now classic book, Refactoring, Martin Fowler lists comments as one of his "bad smells in code" for this exact reason. It's much better to strive for shorter, clearer classes and methods. Give them expressive, meaningful names and signatures, and they can become almost self-documenting.

It is however valid to use comments to explain why code does what it does. That line of code that adds 1 to $x may do that to fix a critical bug. In this case a brief comment highlighting the fact, perhaps with a reference to a bug tracker id, will signal to the programmers who have to work with this code in future exactly why that line is there.

#8: Logging

I would urge that Bill's advice to build some helpful logging solutions into the code be followed with great caution. It may help in some specific cases, but spurious logging code can clog up the real application code and make it a great deal more difficult to read.

Furthermore, there's very little that you can do to anger your systems guy more than sending pointless logging code into production, and eating up disk space with log files that nobody ever reads.

#9: Keeping Your Skills Up-to-date

Agreed. Strive for the deepest possible understanding of the tools and technologies you're using, but also read around the subject. Pick up a new language every now and again, read about design patterns and get the hang of some Unix tools, such as sed and awk. Play with benchmarking tools and try to make it to developer conferences and local user groups.

You'll be ten times the programmer you would be otherwise, and you'll thank yourself in that next job interview, because what do you do to keep your skills up-to-date? is a question that's guaranteed to come up.

#10: Taking Pride in Your Work

Agreed. See points 1-9 for more details!

Footnotes

[1] It's only fair to point out that Spoonfed has been redesigned since I linked to it. The new version is greatly improved in some ways, but unfortunately still makes hopelessly gratuitious use of Ajax, which greatly detracts from its usability.

Benchmarking Zend Download Server

Recently I've started looking into ways that the PHP dev team in which I work can make better use of our Zend Platform installation.

For that reason, the recent Ibuildings/Zend seminar in London on the subject of "Enterprise PHP" was well timed, as it included a pretty detailed run through of a lot of what Platform has to offer.

One feature which really struck me as having the potential to bring performance benefits to one of our systems was the Zend Download Server. Back at the office, I looked into the feature, and ran a few benchmarks. Oddly though, the results don't seem to flatter Zend Download Server.

Zend Download Server

The premise behind Zend Download Server (ZDS) is that tying up valuable Apache HTTPD threads purely to serve static content is overkill, and far from efficient.

This is the same reason why lightweight webservers such as lighttpd are becoming popular. Lightweight webservers are typically run alongside a more powerful server such as Apache, and are dedicated to serving static content, leaving more Apache threads free to deal with the dynamic - for example PHP-based - requests.

ZDS follows that principle, although it works a little differently to lighttpd: it runs as a standalone process, but it hijacks a single Apache thread, thus allowing Apache to delegate the relevant requests down to ZDS.

ZDS can be utilised in a couple of ways. Firstly, there's 'transparent mode', whereby the administrator configures Platform in advance, telling it to hand specific downloads (say, all JPGs and GIFs over 128KB) off to ZDS.

The second option is 'manual mode', whereby the developer hooks directly into ZDS using a simple call to the proprietary zend_send_file() function. zend_send_file() is designed as a drop in replacement for functions such as fpassthru(), which simply read in the contents of a file and send them to output (in this case the HTTP response).

zend_download_file() seemed ideal for my needs, but not wishing to break the third rule of optimization, I decided to do a little benchmarking before I got too excited.

Benchmarking ZDS

I compared the two simplest possible scripts I could come up with. This first example uses the built-in PHP function, fpassthru():

And here's the amended version of the script, using zend_send_file()to deliver the file:

Pretty straightforward stuff, all in all. Both scripts are delivering the same file, a JPG image of slightly less than 500KB.

I threw some load at the script using the ab benchmarking tool that ships with Apache HTTPD. Here's an example of the kind of command I ran:

./ab -n 200 -c 10 http://platform/deliver_file.php

The -n argument specifies the total number of requests, while -c specifies the number of concurrent requests that ab will try to make.

Here's an abridged version of the output using fpassthru():

Time taken for tests:   8.849252 seconds

Requests per second:    22.60 [#/sec] (mean)
Time per request:       442.463 [ms] (mean)
Time per request:       44.246 [ms] (mean, across all concurrent requests)
Transfer rate:          10818.09 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2   87 514.0     14    4082
Processing:    96  337 182.3    281    1127
Waiting:        9   65  64.4     41     366
Total:        106  425 540.2    309    4364

Percentage of the requests served within a certain time (ms)
  50%    309
  66%    383
  75%    463
  80%    495
  90%    613
  95%    785
  98%   3241
  99%   4224
 100%   4364 (longest request)

And the output using zend_send_file():

Time taken for tests:   8.886843 seconds

Requests per second:    22.51 [#/sec] (mean)
Time per request:       444.342 [ms] (mean)
Time per request:       44.434 [ms] (mean, across all concurrent requests)
Transfer rate:          10721.58 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   82 530.4     15    4420
Processing:    62  338 183.1    285    1069
Waiting:       12   41  20.1     38     148
Total:         63  420 595.1    304    5022

Percentage of the requests served within a certain time (ms)
  50%    304
  66%    370
  75%    500
  80%    545
  90%    640
  95%    741
  98%   1096
  99%   4992
 100%   5022 (longest request)

I tried a few combinations of the various parameters available to ab, and quite honestly couldn't find any conclusive difference in performance between using ZDS and not using it. In fact, under low or very high load, zend_send_file() seemed to slow things down a little.

Conclusions

I'm pretty surprised - enough to doubt the validity of my tests, I admit. I don't believe for a minute that Zend would make false claims for features of their flagship product, so I must be doing something wrong. But what?

I'm aware that I'm measuring network speed as much as anything, and that the claimed benefits of ZDS centre around reduced load on the server side. But still, who cares if load is down when, at the end of the day, performance doesn't improve?

The one concrete lesson that I can offer up from all of this is that it's always valuable to follow the third rule, and Profile Before Optimizing.

Fighting Spam and Digitising Books with reCAPTCHA

When I added a comment form to this blog, I wondered how long it would be before I started getting comment spam. Then I wondered if I was flattering myself to think that spam bots would even be interested in my site.

So it's with mixed emotions that I have to admit that right now the number of spam comments I'm receiving is outstripping the number of genuine comments by a ratio of about 10:1.

The time has come to add a CAPTCHA to the comment form.

The Wikipedia article describes the CAPTCHA concept adequately, so I'll merely summarise that a CAPTCHA is a simple test that the poster of the comment is human. I show you a picture of some wonky-looking text, and you type the words you see into the box provided.

Fig 1: Some wonky-looking text

If you correctly identify the words, I'll assume that you're a real person, and not an evil bot. And your comment will get posted. Simple as that.

reCAPTCHA

I had been meaning to have a play with reCAPTCHA since it caught my eye a few months back. It's a great idea: a totally free CAPTCHA tool, developed by Carnegie Mellon University, that anyone can use on their website.

What makes reCAPTCHA special is that at the same time as you're reading that wonky text and entering the words in the box, you're playing your part in a global effort to digitise pre-computer era books, by deciphering the words that OCR software struggles with. There's a more detailed overview of the project here.

It's kind of a cool idea, so I'm going to co-opt reCAPTCHA to help me fend off those evil spammers. I won't be alone: reCAPTCHA counts sites as large as Facebook, Twitter and StumbleUpon among its users [1].

Implementation

The first step in using reCAPTCHA is to drop in at the reCAPTCHA site and get yourself an account. Of course, you'll have to fill in a reCAPTCHA to do this!

As part of the signup process, you'll be prompted to request a key for your first domain (each key is restricted for use on only one domain, apparently for security reasons). In fact, you receive both a public and a private key, and we'll see how to use those shortly. The whole process takes about two minutes.

Once you're signed up, you're free to start implementing reCAPTCHA. For us PHP users, this is delightfully simple, as the reCAPTCHA guys have thoughtfully knocked up a small library to wrap their API. You can download the library from the project's Google Code pages.

Simply download the code, and unzip it somewhere sane and accessible on the webserver. I'll refer to the installation directory as /path/to/recaptcha for the purposes of this post.

To begin using reCAPTCHA, we'll start by adding some HTML to the comment form in order to display the reCAPTCHA challenge box. The library generates all the HTML we need:

It really is that simple, and the reCAPTCHA challenge box shows up as if by magic. With its default theme, it looks like so:

Fig 2: The default reCAPTCHA challenge box

Drop that HTML into the appropriate place in whichever form you want to protect from spam. Once the form is submitted, you can check the validity of the submission as follows:

Job done, basically. You can theme the actual reCAPTCHA box - to an extent - quite easily, which is nice as the default beige and maroon jarred a little with my fetching grey and turquoise getup. To do that, add a small snippet of Javascript to the form page:

<script type="text/javascript">
var RecaptchaOptions = {
   theme : 'white',
};
</script>

There is also a 'custom' theme which gives you a lot more control over the look and feel, but for the time being I stuck with 'white'. The whistles and bells can wait!

That's really all there is to it. If you like, you can see the finished comment form, replete with reCAPTCHA, for this post. Time will tell what effect this has the amount of spam I receive.

Footnotes

[1] http://news.bbc.co.uk/1/hi/technology/7023627.stm