Development, Testing, Installation and Maintenance v1.23
This document details the development, testing, install and maintenance cycle for www.bbc.co.uk servers. For information on this cycle for news.bbc.co.uk servers, please contact the Editor, NM Standards & Guidelines.
1. Standard infrastructure and non-standard systems
The current bbc.co.uk infrastructure is documented in bbc.co.uk standard infrastructure – supported software.
If your project has requirements that do not fit in with the existing bbc.co.uk infrastructure you will need to discuss your options with your Technical Project Manager (technical lead). These may include:
- Hosting by third party with links from/to bbc.co.uk services (see Third Party Hosting Requirements)
- Installation of custom hardware to support your application within the bbc.co.uk architecture and with access to bbc.co.uk systems
In both cases security rules will apply and external independent 'Penetration Testing' companies will be used to check these systems in addition to the security processes below. The third-party provider will need to provide satisfactory evidence of good security/ policies and availability of their services.
For bbc.co.uk hosted solutions, all the rules in terms of access will apply. For externally hosted systems there will be more flexibility – if good practice is demonstrated.
2. Development environment
2.1 Development servers
External developers may use the extdev.bbc.co.uk development server, which resides outside the firewall, for all testing purposes.
In circumstances where external developers are allowed to use development servers inside the firewall, you may use dev0.reith.bbc.co.uk.
2.2 Use of development servers
The extdev.bbc.co.uk development server is managed by the Siemens Internet Operations team. This externally accessible development environment is available for building and testing content and applications that will eventually be published as part of bbc.co.uk.
extdev.bbc.co.uk should not be used as a general storage area for material that will not be published onto the live environment. The server is not regularly backed-up, so developers are strongly advised to keep a back-up of all material published there.
A variety of features and options are available within the development environment, including:
www(for web pages, including Barley)data(for templates used by CGI scripts)cgi-bin(for CGI scripts)results(only needed for CGI scripts that write files to the server)cgi-perl(please see mod_perl guidelines in the (Server-Side) Application Development Standards
)
2.3 Development accounts
To apply for a development account on a development server, your Technical Project Manager (technical lead) must complete a Dev Account Request Form available from Siemens Internet Operations or Red Bee Media webmasters.
An account can take up to 3 days to process and set up.
When requesting a development account, consider the following points:
- Will other developers need to work on content held within the developer's personal account?
- If Yes: It may be appropriate to set up a generic "User Group" to enable read/write privileges to other named developers.
- More information and docs on the environment can be found at http://extdev.bbc.co.uk/index.shtml
2.4 Development constraints
Ensure that you adhere to all the appropriate BBC New Media standards. Your Technical Project Manager or Technical Forum representative can assist you in identifying standards relevant to your project. These can be found at http://www.bbc.co.uk/guidelines/futuremedia/.
Ensure any database usage and schemas are approved by New Media DBA – as per the application standards
Inform the Technical Project Manager about volumetrics of your application – as per the application standards.
Ensure that your application works with the supported software versions installed on the BBC servers, or that you make a request to use unsupported software early in your development cycle – see bbc.co.uk standard infrastructure – supported software.
Ensure that file-paths will work after your project is moved to a BBC server. Typically these should be:
/home/system/www/[sitename]/(for HTML pages)/home/system/cgi-bin/[sitename]/(for mod_perl applications)/home/system/perldata/[sitename]/(for templates read by modperl applications)
You MUST test your project on a BBC development server (e.g extdev.bbc.co.uk or dev0.reith.bbc.co.uk) before submitting it for acceptance testing.
2.5 Presentation of development content
When you set up an account, you can request the setup of a symbolic link between a developer's personal development space and that of the site under development. This will allow content to behave as if it were already on the live environment.
For example, if Joe Bloggs (with login username: joeb) is working on the "disco" site, Red Bee Media webmasters can arrange for the site to be browsable via the link http://extdev.bbc.co.uk/disco instead of (for example) http://extdev.bbc.co.uk/joeb/disco.
3. Acceptance testing
3.1 Acceptance testing for new websites
All new bbc.co.uk websites MUST be tested against the appropriate bbc.co.uk technical and design standards before being published on the public web server, according to the acceptance testing procedure for website deliverables, as specified in the website's contract (see Legal & Business Considerations for further information).
Acceptance testing is supervised by Red Bee Media webmasters, who are responsible for monitoring and maintaining quality assurance by checking the consistency, continuity, and coherence of supported services. The team is responsible for testing the usability, accessibility, and cross-platform compatibility of supported services.
Sites that do not meet bbc.co.uk standards cannot be published. Website tests produce a report, available to the site's producer and developers, which outlines the reasons for a site's success or failure to meet these standards.
3.2 Acceptance (security) testing for new applications
To prevent downtime due to compromised applications, all active code destined for bbc.co.uk is vetted by the Siemens Internet Operations team to ensure it has no security flaws and passes the relevant guidelines [see (Server-Side) Application Development Standards )].
Please liaise with your Technical Project Manager to assist in the operation of this procedure, when it is time to install an application to ensure correct process.
Executable code hosted elsewhere will be subjected to independent security testing prior to public release (see Third Party Hosting Standards).
4. Uploading content and installing applications
To prevent illicit changes to bbc.co.uk site content or unauthorised use of personal data on bbc.co.uk, third parties only have access to the following:
- An FTP account on the main distribution server to upload web, streamed and downloaded content and application configuration files (not applications)
- HTTP/HTTPS to site and services with basic authentication where appropriate
Third parties have no direct shell access at all (e.g. SSH, Telnet or SQL*Net) to live applications, web or database servers.
4.1 Application installation
All applications are installed by Siemens Internet Operations – see (Server-Side) Application Development Standards .
4.2 Uploading content via the FTPBorg queue
All new and updated content to the live site e.g. web pages, AV media and config files, are FTP'ed to borg.bbc.co.uk. The BBC uses a system called FTPBorg to ensure content is distributed to the correct set of live servers.
The directory structure in your FTP account determines the servers the content goes to. See 4.2.4 for further information on this. For example, content place in the 'rmhttp' directory of your FTP account, goes to the downloads servers 'downloads.bbc.co.uk'.
4.2.1 Gaining access to the MCS (FTPBorg)
If it is agreed that you can publish your own content and scripts, you MAY request a Trusted User Account from Red Bee Media webmasters. This account lets you upload content via FTP to the Master Control Server, giving you access to the content on the site that is suitable to the status of your trusted user account.
4.2.2 FTPBorg queue upload etiquette
The FTPBorg queue is a facility which is shared by the webmasters of all websites to update their content. To ensure that no individual webmaster abuses this contended facility, you must follow these guidelines whenever you upload anything to the queue:
- always upload the minimum number of files necessary to make your content change – for example, do not upload your whole site to make a small change to a single page.
- Before uploading a large volume of files during a peak time, you must contact the Red Bee Media webmasters to request their permission.
4.2.3 Upload process
- FTP the files to the correct directory on
borg.bbc.co.uk. (NB. Always save text filetypes as UNIX (not Windows) and always upload text file-types as ASCII). - Check that they appear at http://www0.bbc.co.uk.
- The Borg system intercepts your FTP requests (such as
PUT,DEL, andMKDIR), then puts them into a transaction log file, which is applied to all servers in the content network in the order received. Find out how long your content update will take to reach all live servers by viewing http://www0.bbc.co.uk/cgi-bin/support/borgqueue. Please ensure your upload application is resilient and restarts in event of a borg restart/recovery. - Examine your changed files at http://www.bbc.co.uk.
4.2.4 Architecture of FTPBorg
The architecture of the FTPBorg system is straightforward. FTPBorg places new content in a queue for upload to each live server. Upload takes place in parallel to each server, but (depending upon a number of factors) can cause content to be delivered over approximately 15 minutes. This is especially the case for our international servers.
Not all files placed on borg.bbc.co.uk are borged to all servers. There are five distinct server types in the network.
- Web content visible to the public from
www.bbc.co.uk. - Modperl applications running on dedicated application servers, visible via
www.bbc.co.uk/cgi-perl/. - CPSs* dedicated servers to generate content for the main site.
- Audio or video Files for streaming servers (Windows and Real format
- Large file servers ( 500kbytes+ ) (
downloads.bbc.co.uk)
Your FTP account displays the following directories and their locations.
/www/– 1 – URL: http://www.bbc.co.uk/data/– 1,2 – URL: Not visible/perldata/– 2 – URL: Not visible/cps/– 3 – URL: http://cps0.mh.bbc.co.uk/.../results/– this is a special directory showing the output of scripts needing to write to files – URL: Not visible/cgi-bin/– 1 – URL: http://www.bbc.co.uk/cgi-bin/.../cgi-perl/– 2 – URL: http://www.bbc.co.uk/cgi-perl/advance/– in this folder, you can create a sub-folder where the name is the time and date at which you wish that content to become active. If you would like to use this feature (which is good for launching content to synchronise with timed events like programme broadcasts) then contact Red Bee Media webmasters to set up the advance folder in your trusted user FTP account. – URL: Not visible/realmedia/– 4 – see AV standards/rmhttp/– 5 – URL: http://downloads.bbc.co.uk/...
It is important that you place the correct content in the correct folder above. For example, streamed AV content will not work unless placed into the 'realmedia' folder and large file content (500Kbytes+) must be placed into the 'rmhttp' folder.
If any of the above folders do not exist in your account, request Red Bee to set them up for you.
5. Support and maintenance
When planning your project you MUST consider issues of support and maintenance and document and supply the relevant information to the technical contact, such as:
- What will need to be maintained?
- How much will need to be maintained?
- Who will maintain it?
For applications running on http://www.bbc.co.uk, or for developers producing content for it, the following support and maintenance arrangements are available:
- First line – Red Bee Media webmasters
- Second Line – Siemens Internet Operations, who offer the following services:
- Various RRD based monitoring and reporting tools
- A hotline number for 24/7 technical staff with direct access to these systems (the hotline can be used to request application changes or installation)
- Third Line – the following groups support their own applications:
- New Media developers within BBC divisions
- External companies, where appropriate
If you are developing applications for www.bbc.co.uk, then you MUST plan for first, second and third line support. You MUST provide those teams with documentation and possibly training so that they can carry out typical support activities. If you do not intend to provide third line (maintenance) support or ongoing development of the application, then you MUST have an agreement in place with the team that you expect to provide this for you.
6. Document history
| Date | Version | Change | Author |
|---|---|---|---|
| 09/08/2006 | v1.23 | Comments from Nick Holmes | Tred Magill/Mark Hewis |
| 24/07/2006 | v1.22 | Final edit | Tred Magill |
| 24/07/2006 | v1.21 | Review of annual update | Mark Hewis/Tred Magill |
| 30/05/2006 | v1.20 | Annual update | Mark Hewis/Andrew Macinnes |
| 16/03/2005 | v1.19 | After amendments required by the Tech Forum on 14/03/2005 | Jonathan Hassell |
| 11/03/2005 | v1.18 | Revamp of this doc with Mark Hewis | Jonathan Hassell |
| 07/09/2004 | v0.17 | Note added in section 5.3 re comparing this with Mark Hewis guidelines doc (link) | Jonathan Hassell |
| 11/08/2004 | v1.16 | Slight updates by JH pre-audit | Jonathan Hassell |
| 20/04/2004 | v1.15 | Added lines to the FTP bit (from HTML Integrity Standards) | Jonathan Hassell |
| 04/03/2003 | v1.14 | Added bookmarks to all headings for easier reference from browser | Jonathan Hassell |
| 19/09/2002 | v1.13 | updated internal link to contact instead | Jonathan Hassell |
| 19/08/2002 | v1.12 | added audience into footer | Jonathan Hassell |
| 19/08/2002 | v1.11 | moved all by-lines into comments | Jonathan Hassell |
| 17/08/2002 | v1.10 | checked all contacts are abstracted into contacts file | Jonathan Hassell |
| 16/08/2002 | v1.09 | Sorted all inter-document links | Jonathan Hassell |
| 16/08/2002 | v1.08 | Renamed as chapter 5 | Jonathan Hassell |
| 15/08/2002 | v1.07 | Done global search for MUST, SHOULD etc. and a lot of tidying up | Jonathan Hassell |
| 14/08/2002 | v1.06 | Started doing global search for MUST, SHOULD etc. | Jonathan Hassell |
| 14/08/2002 | v1.05 | Brought across info on Dev environment from App 1 | Jonathan Hassell |
| 14/08/2002 | v1.04 | Brought across testing and maintenance stuff from app1 | Jonathan Hassell |
| 14/08/2002 | v1.03 | Edited update process sections | Jonathan Hassell |
| 07/08/2002 | v1.02 | Added copyright information to footer | Jonathan Hassell |
| 05/08/2002 | v1.01 | Added info from App 2 (app installation and content modification). This still needs a good edit, to place the right things in the right places... | Jonathan Hassell |
| 05/08/2002 | v.100 | 0 created document from info taken out of App5 | Jonathan Hassell |
Document editor: Editor, Standards & Guidelines. If you have any comments, questions or requests relating to this document, please contact the Editor, Standards & Guidelines.
Like all other Future Media Standards & Guidelines, this page is updated on a regular basis, through the process described on About Standards & Guidelines.
© MMIX