« Previous | Main | Next »

How we make websites

Post categories:

Michael Smethurst Michael Smethurst | 14:03 UK time, Thursday, 29 January 2009

Designing and building data driven dynamic web applications the one web, domain driven, RESTful, open, linked data way

For the past few months I've been touting a presentation around the BBC entitled 'How we make websites'. It's a compendium of everything our team has learned from long years developing /programmes, the recent work on /music and the currently in development /events.

As a warning there's very little original thinking in here. For those familiar with the concept of one web, the importance of persistent URIs, REST, Domain Driven Design and Linked Open Data it'll probably be old news. Possibly it's interesting to see all these threads tied up in one place!?! Maybe it's interesting to see them all from a user experience point of view?!? Anyway, as ever, it's built on the thinking and achievements of many clever people over many years who are too numerous to mention here. Although obviously I'll make an exception for Paul Clifford and TimBL. :)

The presentation is here and the (slightly) expanded text is below for the sake of accessibility and Google.

How We Make Websites
View more presentations or upload your own. (tags: bbc a&m)

Explore the domain

Thumbnail image for Domain Driven Design book

This should be clear from the business requirements - it might be food or music or gardening or...

Employ a domain expert. Get them to sketch their world and sketch back at them. Concentrate on modelling real (physical and metaphysical) things not web pages - try to blank from your mind all thoughts of the resulting web site. This work should never stop - you need to do this through the lifetime of the project as you refine your understanding.

Identify your domain objects and the relationships between them

Programmes domain model

As you chat and sketch with your domain expert you should build up a picture of the types of things they're concerned with. Make a list of these objects.

As your knowledge of the domain increases you'll build up a picture of how your objects interlink. You can sketch basic entity relationship diagrams with your domain expert and keep sketching until the picture clears. Bear in mind you're trying to capture the domain ontology - this isn't about sketching database schemas. The resulting domain model will inform the rest of your project and should be one of the few artifacts your project ever creates.

Check your domain model with users

Sketch with users

Run focus groups and speak to users. Get them to sketch their understanding of the domain and again sketch back at them. After several round trips you should be able to synthesise the expert model and the user model. User-centric design starts here - if you choose to model things and relationships between those things that users can't easily comprehend no amount of wireframes or personaes or storyboards will help you out.

Check to see if your website already deals with some of your domain objects

Build horizontally, not vertically

If it does then reuse this functionality by linking to these pages - you don't want to mint new URIs for existing objects. Having more than one page per thing confuses users and confuses Google. Try to think of your website as a coherent whole; not as a collection of individual products. And as ever, don't expose your internal organisational structures through your website. Users don't care about departments or reporting lines.

The glory will always come from building skyscrapers - the real challenge lies in decent town planning. It's more difficult to build new services that stitch into your site and stitch into the web than build shiny, shrink wrapped, self contained products.

Design your database

Programmes database schema

Translate your domain model into a physical database schema.

Source your data

Creative Commons logo

Check if there are business systems in your organisation able to populate your schema. Check if there are existing websites outside your organisation you can use to populate your schema. Give preferential treatment to any websites that offer their data under a liberal licencing agreement - you can buy in data to help you slice and dice your own data but if you do this you might not be able to provide an open data API without giving away the 3rd party's business model. If your organisation AND an open data website can provide the data, consider the danger in minting new identifiers for your own data - can you easily link out / can you easily get links in?

Data licensing is one of those areas that often gets ignored in project planning. If you fail to consider it or get it wrong it can severely curtail your plans further down the line.

Pipe in your data

Programmes data flow

Whether you choose to use your business data or buy data or use open data you'll need a way of piping it into your database schema. You'll probably have to reshape it to make it suitable for publishing.

Make your models

Models

In an MVC framework your models should contain all your business logic. This mean they should capture all the constraints of your database schema plus all the extra constraints implied by your domain model.

Design your URI schema

Using post-its to design URIs

Your URI schema should follow naturally from your domain model. As an example if you're dealing with books and a book can have many authors then ../:book/authors should list all the authors of that book. At Audio and Music we tend to use large walls and lots of post-its to design our URIs. Add some string to show links and journeys and there's no need to ever draw another site map.

This isn't just about designing URIs for resources you link to - sometimes your pages will be made up of other transcluded resources - all of these subsidiary resources should be addressable too. It means you can easily change your user experience layer by taking out transcluded resources and linking to them instead or removing links and transcluding.

By making every nugget of content addressable you allow other sites to link to it, improve your bookmarkability and increase your SEO - cf. an individual 'tweet'. Bear in mind that some representations (specifically mobile) will need smaller, more fragmented representations with lower page weight - designing your subsidiary resources to be addressable allows you to easily deal with this requirement - transclude the content on a desktop machine, link to it on a mobile.

This is where we begin to talk about one web and REST. Each thing should be one resource with one URI - the representation you get back (whether desktop HTML or mobile XHTML MP or RDF or YAML or JSON) should depend on what your user agent asks for via content negotiation. It means I can send a link to a friend from a desktop machine, they can click on that link from a mobile and they'll get back a representation appropriate to their device. Or vice versa. One web with no mobile ghetto.

It's important not to confuse URI design with site structure and user journeys. If you're used to working on hierarchical silo sites then the URI structure often determines the navigation. This isn't true here. Think of the individual resources as tent poles - the user journeys are the canvas that gets draped over later.

It's nice if URIs are human readable. It's also nice if they're hackable. It's an absolute prerequisite that they're persistent.

Don't sacrifice persistence for the sake of prettiness or misguided SEO. URIs are your promise to the web and your users - if you change them or change their meaning you break that promise - links break, bookmarks break, citations break and your search engine juice is lost.

Remember: Cool URIs don't change.

Make hello world pages for your primary domain objects

h1 for In Our Time

For now all they need is an h1 with the title of the object.

Make hello world pages for your primary aggregations

h1 for aggregation page

For now all they need is an h1 with the title of the aggregation and a linked list of things aggregated.

Define the data you need to build each of your pages

Some pseudo SQL

Traditional wireframes lump together data requirements (via annotations), page layout and (by implication) document design. It's best to split these out into 3 distinct tasks. The first task is to define the data requirements.

For each URI define the data needed to build all representations of the thing. Just because the HTML representation doesn't need to show the updated date doesn't mean the RSS or Atom or RDF don't need it.

Some resources will transclude others. There's no need to define the data required for these - just reference the transcluded resource.

Build up your HTML pages and other representations

More HTML

Now you know what data you need you can begin to surface this in your representations.

If you're working in HTML make sure you design your document to be semantically correct and accessible. Try not to think about page layout - that's the job of CSS not markup. Document design should be independent of page layout. In general your page should be structured into title, content, navigation - screen readers don't want to fight through calendar tables etc to get to the content.

Add caching and search sitemaps

Eggtimer

Knowing what can be cached and for how long is a vital part of designing your user experience. Cache for too long and pages go stale. Don't cache for long enough and you send unnecessary traffic across the wires and place extra strain on your application.

Cached pages will also be faster and smoother to render in a browser. And if your users are paying for data on a mobile every extra connection means bigger bills, which is definitely a user experience issue.

An example: if you're creating a schedule page for today's TV you want to cache for performance reasons but you don't want to cache it for too long since schedules are subject to change. But you can cache yesterday's schedule more aggressively and last week's schedule more aggressively still.

Creating XML search sitemaps helps search engines know which bits of your site have been updated. Which helps them to know which bits to re-index. Which helps to make your content more findable.

Apply layout CSS

Wireframe

Add layout CSS to your HTML pages. Experiment with different layouts for your markup by moving elements around the page. You're wireframing!

Test and iterate

Repeat

You should be testing with real users at every stage of development but it's particularly important to conduct usability AND accessibility tests now. It's like testing traditional wireframes but you're testing on the real application with real application behaviours and real data (no lorum ipsum nonsense).

Sometimes the results of your testing will require changes to layout CSS, sometimes to markup, sometimes to the data you need to surface and sometimes to the underlying domain / data model. Bear in mind if you're using data from existing business systems there may need to be heavy investment to make changes to that data model and employ the staff to admin those changes. Occasionally it might even mean renegotiating contracts with outside data providers. All design and usability issues are fixable - some just need more lawyers than others : )

Apply decor CSS

Decor CSS

Over the top of your wireframe application you can now start to add visual design and branding. This is exactly the same process as taking a paper wireframe and applying design treatments over the top except you're mainly working in CSS.

Experiment with different treatments - see how far you can stretch the design with the markup given. Sometimes you'll need to add additional markup to hook your CSS off.

Now's the time to add background imagery for headers, dividers, buttons, list items etc so best to open Photoshop / Illustrator to make your design assets.

And test and iterate

Repeat

Never stop testing.

Remember that personas are just abstractions of people - it's always better to use real people.

Ideally you should be able to adjust your code / markup / CSS to respond to user requests. If you can afford the recruitment / developer time there's no better way to test than with a user sitting alongside a developer - the developer can react to user requests, tweak the application and gain instant feedback without the ambiguity that sometimes comes from test reports.

Again you should accessibility test - some of the design / decor changes may affect font sizes etc - make sure your users can still read the page.

Add any JavaScript / AJAX

Ajax

By designing your browsable site first and adding in Javascript / AJAX over the top you stand a better chance of making an accessible web site - one that degrades gracefully.

As ever Google et al are your least able users - search bots don't like forms or JavaScript - sites that degrade well for accessibility also degrade well for search engines.

Making every subsidiary resource addressable and providing these resources serialised as XML or JSON makes adding AJAX relatively trivial.

You'll probably need to tweak your CSS to adjust to life with JavaScript / AJAX.

And test and iterate

Repeat

Again test your site for accessibility and usability with JavaScript turned on and off.

Continue

Desk with sketches

Follow the same steps for each development cycle. Some development cycles will just be about surfacing new views of the existing domain model; some will require expanding your domain model.

Now you know your domain model and have made each domain object addressable layering over new views and more subtle user journeys should be trivial.

And keep testing!

Comments

  • Comment number 1.

    I agree with all you have said, except for the bit about javascript/ajax. For a site like mine (www.trailbehind.com), all we have is javascript and ajax, along with some data embedded in the static pages for the search engines specifically.

    But the ajax/Javascript can't just be a tacked on thing. We have thousands of lines of Javascript. It's fundamental.

  • Comment number 2.

    @trailbehind I see what you are saying, but I think in most cases the approach outlined works - with most websites you need to consider your data and the views you create atop of that data, which can be translated into your run-of-the-mill HTML/CSS.

    If we look at your site, you have what looks like a google map with data - something which could otherwise be represented as data in a table.

    Where Ajax/JavaScript/Flash/Flex/Silverlight comes in is where you start considering how the user will be interacting with the data. Not fundamental at any way to the data structure or how the backend works.

    Same goes for video players, or any other kind of rich interface.

    In the case of the article, I'd say that whilst it is keeping with the principles of progressive enhancement, it makes sense to start thinking about the need for these rich-frontend-technologies a bit earlier, around the same time you are designing your decor/layout CSS, since the frontend-experience will make or break your website.

  • Comment number 3.

    In my days as a web-developer I remember giving a presentation at the Design Council about 7 years ago with colleagues from information architecture and design. We presented how important it is to make sure all areas are represented when starting to build a website.

    Michael's presentation, as I understand it, is from a stage further on from the beginning - as there obviously has to be input from engineering about what is possible.

    By maintaining collaboration effort between the disciplines during development - engineering, design, information architecture and the 'business' - successful delivery can be achieved. In the same way as one wouldn't wish to end up with something that looks fantastic, but performs so poorly that is unusable - we wouldn't want to have something that has all the amazing engineering to make it top-performing but the interface is unusable.

    I remember working for an agency where we worked on a key implementation of a new CMS system for a big organisation. If it hadn't been for engineering work with information architecture we wouldn't have used all the facilities that CMS had available as the engineers were able to interpret the CMS and explain those features to the Information Architects. In the other direction - the IAs pushed the engineers to make the CMS 'sing' so we could implement, what was described by the CMS company, the most innovative use of their technology at the time.

    Team work.

  • Comment number 4.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 5.

    Hi there,

    I was first introduced to domain driven software design about five or six years ago, building enterprise software. A group of us started really getting into the Fowler ideas through the PoEAA and Analysis Patterns books. It's great to see the methods applied to web application design.

    The BBC work is looking really good at the moment, with the Comet-based radio visualisation, the iPlayer and the music beta site. Can't wait to see the new events site!!

    Damien
    web developer, triple j radio (Australian Broadcasting Corporation)

  • Comment number 6.

    And not a single mention of microformats!

    Nice work, Michael.

  • Comment number 7.

    This is fascinating. Well done. Not only is the thinking lucid and logical -- the fact that coherent, conscious thought happens in this aspect is very satisfying to know.

    Question: WHO makes the decision(s) as to what content is publiched online -- and HOW (i.e. on what bases)?

    PS: E.g. particular bugbear of mine for years: Why does Moral Maze have neither a podcast nor an archive?

    PPS: From your exposition, it seems that you (info arch) go get the domain expert: would that be upon your having been commissioned for the site etc?

  • Comment number 8.

    @afrodigital - Decisions on what sites are built and what content is published are generally taken by a combined team of technical staff (who design and build things) and editorial staff (who work closely with the radio networks and audiences) based on our strategy and the requirements of the radio networks.

    As you suggest, this will happen before the process described above, but it won't necessarily be a completely different set of people.

  • Comment number 9.

    Hi there,
    I liked your clearly presented account, you make it sound so simple ! I would add that you should bear in mind rendering issues between browsers especially internet explorer because it is riddled with bugs.

    Monker
    Link Building Specialists

  • Comment number 10.

    Although I enjoy making my websites at home it would be great to be part of such a huge organisation like BBC and be part of the web design team. I'm jealous.
    Alan

  • Comment number 11.

    Interesting article but why do the BBC (and many other sites) still insist on fixed width layouts?

    Screens are getting wider and wider yet many sites still use a thin strip of content down the middle.

  • Comment number 12.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 13.

    Another good read, "If you're working in HTML make sure you design your document to be semantically correct and accessible" In regards to this section I read a couple of articles in dot net magazine one about reset browser styling to help ensure cross platform usability and another about printing out a design of the page with the most complex layout and then writing the html elements such as "img, h1, h2, a " on the page ignoring elements such as div so as to guage the best way to create a uniform site layout with tidy css pages. The articles was in dot net may 2009 entitled "slicing like a pro" and "css/reset documents".

    Regards

    James Myers
    Website Designer

  • Comment number 14.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 15.

    This is awesome stuff. I have been web building for quite a while now, and i would love to get into the nooks and crannies of the BBC site. Dynamic pages are always going to be the way forward, and learning things like ASP and PHP are a given.

    I'm just going to enjoy my Egypt holidays, until i must return to the dreary office from whence i came! :)

  • Comment number 16.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 17.

    This comment was removed because the moderators found it broke the house rules. Explain.

 

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.