The world of intelligence has grown exponentially over the last decade. This article suggests that prevailing explanation of this expansion – the spectre of ‘new terrorism’ – reflects serious misunderstandings. Much of the emergency legislation which has extended the power of the state so remarkably was already sitting in the pending trays of officials in the late 1990s. Instead, the rise of both the ‘new terrorism’ and its supposed nemesis – the secret state – both owe more to long-term structural factors. Globalisation has accelerated a wide range of sub-military transnational threats, of which the ‘new terrorism’ is but one example. Meanwhile the long-promised engines of global governance are nowhere in sight. In their absence, the underside of a globalising world is increasingly policed by ‘vigilant states’ that resort to a mixture of military power and intelligence power in an attempt to address these problems. Yet the intelligence services cannot meet the improbable demands for omniscience made by governments, nor can they square their new enforcer role with vocal demands by global civil society for improved ethical practice.

This article argues that there are five major challenges currently facing Western intelligence communities. First – ensuring skills retention for intelligence officers and analysts, while developing knowledgeable managers and customers, all in an increasingly-complex security environment. Second – instituting and inculcating knowledge and expertise in these staff – while addressing an opponent in al-Qaeda which demonstrates increasingly sophisticated use of IT, new media, etc. Third – drawing-in outside expertise from the research and business communities, as is done currently in the US and Canada but in only a very limited manner in the UK. Fourth – overcoming institutional rigidity in dividing the foreign and domestic – alongside rigid sharing and co-operation relationships. Fifth – creating truly collaborative environments that offer genuine socio-cultural incentives to collaboration rather than mere ‘IT solutions’.

The UK intelligence system is engaged in three distinct roles – producing strategic assessments in the traditional way; acting as a ‘global policeman’ by monitoring terrorist and criminal networks; and raising the capability of other countries to defeat terrorist and insurgency groups. Counter-intuitively, it is perhaps the first role that is most questionable. The use of single source intelligence reporting, drawn from individuals selected principally for their willingness to share secrets, may not be the best way to analyse emerging issues such as climate change, energy security and financial stability. The Joint Intelligence Committee (JIC) may be drawing on too narrow a range of reporting to compete with increasingly sophisticated assessments from the private sector, academia and NGOs. In any event, the JIC has less impact on policy than is often imagined. The second task of ‘global networker’ is better-suited to the intelligence community's ability to combine human intelligence with communications intelligence and bulk data gathering, and is producing results. The third task of helping other countries to enforce the law and resist insurgency is proceeding on an ad hoc basis with occasional successes, but requires co-ordination across Whitehall so that improvements in the capabilities of other countries' intelligence services are accompanied by improved police and justice systems and enhanced oversight. Joint Intelligence Committee perhaps ought to be a Joint Action Committee.

A major feature of current security is the development of intelligence networks between agencies within and between nations and across public and private sectors. The performance of the UK Intelligence and Security Committee (ISC) is examined within this context, especially regarding allegations of collusion by UK agencies in rendition and torture. Proposed changes to the ISC are discussed and it is argued that the ISC must seek to leverage its limited powers by helping to construct an oversight network.

This article argues that there is a need to modernise the law governing accountability of the UK security and intelligence agencies following changes in their work in the last decade. Since 9/11 the agencies have come increasingly into the spotlight, especially because of the adoption of controversial counter-terrorism policies by the government (in particular forms of executive detention) and by its international partners, notably the US. The article discusses the options for reform in three specific areas: the use in legal proceedings of evidence obtained by interception of communications; with regard to the increased importance and scle of collaboration with overseas agencies; and to safeguard the political independence of the agencies in the light of their substantially higher public profile. In each it is argued that protection of human rights and the need for public accountability requires a new balance to be struck with the imperatives of national security.

This article examines the status, role and development of imagery intelligence in the UK government. It is argued that imagery intelligence occupies a subordinate and marginalised position compared to other forms of intelligence, chiefly from human sources and the interception of communications. The origins of that position are recounted, and the problems arising from internal struggles over control of imagery examined. It is concluded that the existing approach to imagery represents a serious problem and that a substantial restructuring and upgrading of imagery intelligence is essential if UK foreign policy decision-making is to be properly informed in the 21st Century.

The use of force in international relations by the West is increasingly witnessing a greater reliance on Special Forces. This trend has profound implications for state action because Special Forces represent a very different kind of soldier and they possess the inherent ability to transgress traditional boundaries in peace and war. The development and participation of UK Special Forces in the Global War on Terror provides a microcosm of the positive and negative dimensions of using secret military units as the force of choice against insurgents and terrorists in Afghanistan, Iraq and indeed on the streets of London.

The UK's domestic intelligence agency, MI5, has become increasingly involved in the realm of law enforcement over the last decade. This article puts the British experience in perspective by comparing it with France's main domestic intelligence agency, which has pushed deep into the law enforcement arena in recent years. A similar perception of Islamist terrorism underpins these parallel developments in the two countries. However, differences relating to accountability, legal systems and conceptions of the state mean that the French intelligence agency has expanded its role considerably more than its British counterpart. The analysis indicates that MI5's move into law enforcement is likely to remain a relatively conservative one.

The 7/7 terrorist attacks demonstrated that there were some clear deficiencies with the organisation of the United Kingdom's counter-terrorism intelligence community. In the aftermath of the attacks, there were moves to develop a more robust ‘counter-terrorism network’ in the United Kingdom that would facilitate better communication and intelligence sharing. While recent developments are to be welcomed, the reforms have not addressed some of the fundamental cultural, institutional and technological issues at the heart of the problem. The creation of an effective counter-terrorism network demands that information flows more freely through the intelligence community and that institutional boundaries are broken down. Until these obstacles have been overcome, the new counter-terrorism network will continue to be hampered by the same old problems of intelligence sharing.