« Previous | Main | Next »

New BBC Site Search

Post categories:

Matthew McDonnell | 13:21 UK time, Friday, 17 December 2010


This week, we launched our new site search product. This launch is a big step on our journey to deliver a single, unified and consistent discovery experience for the BBC.

Try it out with the search box in the top right-hand corner of most pages (or, if you can't think of something to search for, try Liz HurleyMirandaUnemploymentSpain). Let us know your thoughts using the form at the bottom of this page.

The new search is based on the Search+ platform, which I blogged about earlier in the year, and includes many refinements and enhancements.

This new search supersedes both the standard search product and the three-column Search+ product to offer a single site search experience for BBC users.

Before I go through the new features in detail, I'd like first to give you an insight into our product vision and discuss some of the issues that have shaped it.

To scope or not to scope

When I started in this job, there were hundreds of distinct sites on bbc.co.uk and most of those websites would have their own search "scoped" to their content. Not only was this inefficient and wasteful but also it led to an inconsistent experience in which search boxes in exactly the same position on, say, the Schools website and the History website would produce different results. People were often confused; indeed, I remember analysing the logs for the Gardening website search and discovering that the top result was for [eastenders] and returned no results.

We decided to reduce the number of scoped searches and make the search box in the top right-hand corner of every page return the same results, wherever you were on the site. However, this was not universally popular. People tend to think of bbc.co.uk as only that part of the website that they like to use. Many people only use the News website, or GCSE Bitesize or iPlayer and, not unnaturally, are bemused when they search for something and get results from another area. We get many emails asking for the search box on News to return results from News only. There is even an online campaign dedicated to achieving this goal!

So, on the one hand, people are finding no matching results in scoped searches, even though relevant content exists elsewhere on the website. On the other hand, people like the convenience of scoped searches, particularly when they only use one of the BBC's services.

The table below further illustrates this problem. It is a log of the top queries coming from different BBC websites on one day in June 2010. The three columns show where the searches originated (on the iPlayer, News and Sport websites). Those queries in black are the ones for which the best results will come from that website (so, for a search in iPlayer for [luther], we should expect the best content to be from iPlayer). The red ones are the queries for which the best results will come from other parts of the BBC website (so, for a search in iPlayer for [revision science gcse] we would expect the best results to come from Learning rather than iPlayer).


So, as you can see, the picture is pretty mixed, with a closer correlation visible the higher up the search logs one looks between the originating domain of the search and the provenance of the optimal result set.

Clearly, we need to have a search product which recognises that many users want to scope their searches to the domain they are currently in while simultaneously revealing great, relevant content that happens to be in another part of the website.

Product Vision

With this in mind, we created the following Product Vision.

Our aim for search is:

1. To create a scoped search for each BBC product (News, iPlayer, CBBC and so on) that "understands" the domain knowledge behind each product and uses that information to create the best possible sorting rules, display and filtering options within search. In this way, we will provide the richest search experience available anywhere on the web for each of our products.

2. To allow users to search across product scopes to see everything that the BBC has on a subject.

3. To ensure that, regardless of which scoped search someone uses, they are always exposed to the best relevant content the BBC has to offer. This will ensure that, for example, a heavy News user gets to know about iPlayer content relating to their query.

4. To offer related results from non-BBC sources.

5. To make it possible for people to discover trending topics on the BBC site and make feeds of results available to be embedded in other products (on and off the site). To allow search result pages to be recommended from within other BBC products.

We think Search v2 starts to fulfil a number of these ambitions. Let me run through some of the new features and demonstrate how they relate back to the Product Vision.

1: A scoped search for each BBC product

This example is a search for [graeme swann]. The results have been scoped to Sport. This shows the main elements of the page design, which I have highlighted with red numerals.




1. The central column will contain the main list of results. The large heading (in this case, Sport) shows that you are searching within the Sport scope.
2. The left-hand column will contain navigation to the other scopes and filtering options such as media type, date range and sorting choices.
3. The right-hand column will contain search results from other parts of the BBC (Elsewhere on the BBC) or other websites (Elsewhere on the web).

The styling of the new search results uses the BBC's Global Experience Language, which brings it in line with the rest of the website and which Bronwyn comprehensively explains here.

Category-specific filters

In the image above, you can see four sets of filters in the left-hand column. The first is the category filter. It shows you how many results there are for your query within each category. Below this are "scope-specific" filters. For the Sport scope they are:

















1. Allows you to filter your results to show only video, audio or text & pictures. The default setting is All Media.
2. Allows you to set the earliest and latest dates from which you want to find results.
3. Allows you to change the ordering of your results. As I said above, the default order is Newest First, but you can also choose to order by Most Relevant First or Oldest First.



Over time, all scopes will be given filters best suited to that particular type of content, so Learning, for example, will have filters such as Subject, Syllabus and so on.

Context-driven search results

As discussed above, there is a strong correlation between the part of the website that a user is exploring and the type of content they are searching for.

So, we have set the system to return results automatically from the category that relates to the section of the website you are using - if you are on the News site when you enter a search term, you will get results from the News category in the main, central column. Likewise, if you search from Sport, you'll get sports results.

Over time, we will extend this functionality to other sections of the site - iPlayer, Learning, Food and so on. The eventual aim is that all products on the site will scope to their own search category while staying within a single, consistent design. As this happens, we will remove the old "scoped searches" like this one.

From today, however, if you search from any part of the site other than /news or /sport, you will get combined results on our All Results page. You will know which area you are about to search in from the text inside the search box. On News pages the box looks like this:




On pages where you are searching All Results, the box looks like this:






2. Allow views across BBC products

Unscoped searches

When you hit the All Results page, the central column will contain the top results from all the matching categories. The categories are ordered by their relevance to the search query, using an adaptation of the Smart Zones process that I discussed in an earlier post. Here is the [graeme swann] query from above, run on the All Results scope.


Note that the page layout is the same as in the earlier example. However, because I haven't scoped my search, I will see results from all areas of the site in the central column. In this case, Sport is rightly deemed most relevant to the cricketer, followed by Blogs, News and then TV & Radio sites. A maximum of three results are shown from each category.

Likewise, the query [christmas cake] returns Food as the top category. (Ignore the Editor's Choice result in each of the next three examples - I deal with this feature in detail later on.)


Whereas a search for [iceland] returns News as the top category.

Iceland search


And a search for [trigonometry] will put Learning content at the top.


You can get to all results for a particular category by clicking the links in the category title (or in the navigation in the left-hand column).

Editor's Choice

Many search result pages will also include a selection of editorially chosen results that we believe are most likely to be what you are looking for. If you search for a programme title, say [miranda], you will see a link to the programme website.


If you search for a country, say [burma], we will highlight the BBC Country Profile (as well as other editorially selected features) and if you search for a British place, say [blackpool], you'll get a link to the nearest BBC Local website.



3: Ensure that related content is never hidden

Elsewhere on the BBC

As I said in the Product Vision, our aim is never to hide great, relevant content from users, regardless of the scope within which they are searching. So, whenever you search inside a scope, you will always see the Elsewhere on the BBC module showing results from other parts of the website.


4: Offer related results from non-BBC sources

Elsewhere on the web

Below the Elsewhere on the BBC module, you will see the Elsewhere on the web module showing related results from around the web. These are a mixture of editorially selected websites and algorithmically driven results.


5: Linking to search results

Linking to search results from within the BBC

In the example below, you can see that iPlayer now links to BBC search. In this case, a programme about Operation Mincemeat links to search result pages for World War II, Adolf Hitler and the show's presenter, Ben Macintyre. This is possible because search produces a controlled vocabulary of around 40,000 topics that are available as links and which I wrote about previously here.


Search feeds

These 40,000 topics are also available as RSS feeds. Click on the orange symbol in your browser's address bar to get the feed.


How we put this together

For those with a technical inclination, our senior engineer on this project, John Muth, will blog on the technical approach next week and I will add a link from this blog then.

We're hiring!

Finally, a bit of news about the BBC's search team. We are moving to Salford in the summer as part of a BBC-wide relocation. As you can imagine, such a big change has resulted in a fair amount of turnover within the team and so we are now actively hiring software engineers (predominantly Java) and other roles, including my job - Technical Product Manager, Search. You can read about the recruitment process here.


  • Comment number 1.


  • Comment number 2.

    I gave up using the search on www.bbc.co.uk some years ago, instead I've found that using Google and entering bbc.co.uk in the site box on their advanced search page (https://www.google.co.uk/advanced_search?hl=en%29, always returns good results.

    I'll give the new solution from the Beeb a try and I'll be impressed if it beats searching via Google.


  • Comment number 3.

    It's clearly a vast improvement from a visually aesthetic point of view and the filtering has certainly cracked the skinned-scoped search problem (although /iplayer doesn't seem to be playing game?) so congrats... BUT... the results themselves still seem to be quite poor for programmes that aren't currently available to watch/listen. The first example I chose to test was the wonderful BBC Four programme The Joy of Stats which despite expiring it's availability yesterday has, on it's page, a wealth of rich information, gorgeous links (Search included!) and even available video clips... So, looking at the Search results for The Joy of Stats i'm very disappointed to get only one result of an audio clip from the Today programme (that doesn't, in itself, link to the programme page).

    I don't underestimate the magnitude of the task; i do congratulate you on the improvement but to me (and my biases) there's a significant oversight.

  • Comment number 4.

    I tried it with a dozen or so recent programme titles. It failed to find any of them.

    Nul points.


  • Comment number 5.

    Jamie & Russ - thanks for your posts. We are working on improving the results for programmes which are outside their iPlayer on-demand window (typically older than 7 days). It is understandably something that many people in the BBC are very concerned about. We should see every programme being returned in search sometime in January (we had planned to release this functionality with this launch but some problems meant we had to de-couple it from the main release). I will post again once it is out.

    Russ - could you give me a few examples of the programme titles that you searched for? So I can check the reason for them not returning.

  • Comment number 6.

    Just tested out the new search, and I'm finding it a huge improvement, both in terms of results and the experience of the search itself. More functional, and better designed (design of course being about so much more than just "visual aesthetics" - it's now easier to identify links and filters and other information). Well done Matthew.

  • Comment number 7.

    Dotconnect - Thankyou for your comments, it is great to know you are seeing improvements. We feel that the platform gives us a great base upon which to develop in the future. There is still a lot to do: comprehensive programme results which I mentioned above; fleshing out the filters for scopes other than News and Sport; better support for misspellings; changes to the auto-suggest system and new advanced features.

  • Comment number 8.

    Thanks Matthew, I look forward to the update... and keep up the great work, these improvements to Search certainly help bbc.co.uk feel more like one site - those links to search results from content pages are acting like a glue!

  • Comment number 9.

    Matthew, in response to your request in #5, here are a few programme titles taken from the R4 drama and readings area that don't give any results:

    What Do You Know?
    Faith Healer
    On the Ceiling
    The Caretaker
    Achting Tan
    Severed Threads
    God's Revolution
    Tokyo Vice

    Only the BBC could launch a new search function knowing it was incapable of finding BBC programme titles.

    Nul points again for a total failure in strategic vision.


  • Comment number 10.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 11.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 12.

    I couldn't find John Muth's article about the technical approach - would be interested to know what technology you base the search on - is it open source?

  • Comment number 13.

    Since the design elements are so consistently overbearing in their eschewing of aesthetic appeal so as it would seem to purposefully diminish customer experience of the product, then insofar as BBC online is funded by the viewers without any option of exempting themselves from any part of it, then BBC online has placed itself in a rebartative stance vis a vis those who fund it. Which of course, is an inequity on more than one modish 'angry', but nevertheless, decorous battlefront.

  • Comment number 14.

    @dagi (#2) - I've been doing the same for the last couple of years - using Google and including bbc.co.uk together with my search term generally results in getting the results much faster. Also, and google's indexing technology is pretty sophisticated in working out what's news related and what's product relates so I've rarely bothered using the BBC search facility.

    I have to say though, from what's described above in terms of scoped and unscoped searches I'd probably give the new BBC search a go - think I'd get a more useful group of results now than from google

    Anna (UK blogger)

  • Comment number 15.

    I think that there should be a collaboration between old and new teams on MI High

  • Comment number 16.

    Within 24 hours of "Wonders of the Universe" I tried to find information about the first broadcast using the BBC search engine, despite long ago having taken such searches elsewhere.

    Today, if you search for "Brian Cox" you get 680 results, including relevant ones.

    If you search for "Brian Cox arrow time" you get none - not one - 0 results. In Google the top two results point to BBC pages.

    If you search for "arrow of time" on the BBC search engine, the only relevant result for this user is a link to a Daily Telegraph review of the programme.

    The simple conclusion is that specification for the search engine is so sophisticated and feature rich that it overlooks the basic fact that it is supposed to do something.

    It looks like a decision that is somewhere strategically wrong - wrong platform, or something.

    This an issue that people have raised for a long time, and for lack of any evident response, they give up mentioning it as a problem. In general, I expect the public just give up on the search engine, ignore it.

    Is there any information at all about the use of the search engine indicating whether or not it satisfies most searches, or about the extent to which links come from Google searches, for example?

    This just has to be an issue that BBC management need to be aware of and address, and tell us that they are addressing.

    What is said here has been said by many other people - just look at the comments above - and there is no evident response. Even if those responsible for it despair of improvement, it is not good enough. Where do you send a question to actually get a reply?


More from this blog...

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.