Open-source intelligence

Open source intelligence is an intelligence gathering discipline that involves collecting information from open sources and analyzing it to produce usable intelligence for government decision-makers. The field uses open sources to mean information generally available to the public, rather than information produced directly by the government for its own use. For example, the U.S. government's Open Source Center was established to collect information available from "the Internet, databases, press, radio, television, video, geospatial data, photos and commercial imagery." In the style of similar abbreviations in the intelligence community, open source intelligence is sometimes abbreviated OSINT.

Context

According to the Collective Intelligence Portal Page at OSS.Net, "The fundamental premise of OSINT, as opposed to research, is that no single government and no single organization can master "ground truth" on any topic with either in-house resources or with those "at hand." Only an agile network capable of harnessing the distributed intelligence of the Whole Earth--a network able to identify, integrate, exploit, *and disband* key sources, software, and services on a "just enough, just in time" basis, can be successful in both output and financial terms. OSINT brings together eight intelligence tribes not to be confused with the secret intelligence bureaucracies: government, military, law enforcement, business, academia, non-governmental organizations, media, and civil societies including advocacy groups, labor unions, and religions. Aided by the Internet as a vehicle for sharing information OSINT seeks to create what Thomas Stewart calls "the wealth of knowledge" by applying the proven process of secret intelligence to legal and ethical sources of information in order to create decision support at all levels of society from neighborhood to planet. OSINT is especially applicable to the threats facing mankind, from poverty to transnational crime, and could help the public demand improved public policies on the basis of public intelligence."

Public Value

According to the Co-Intelligence Institute public value is derived from collective intelligence that ensures that public policies are consistent with the needs, concerns, and morals of the people who comprise the policy. According to OSS.Net, OSINT is the process by which the public can create public intelligence that holds governments, corporations, and other organizations accountable for operating in keeping with established truths. OSINT is an antidote to the abuse of power based on secret intelligence or the manipulation of the actual truth. At its best, OSINT should allow citizens to have access to structured knowledge about specific global to local threats and opportunities that is superior to that now produced for a handful of key decision-makers in Washington, D.C., at a cost to the taxpayer of $60 billion a year--information that is not shared with the public because it is secret.

Term

The term OSINT is related to the term open source as it applies to the five "opens" (open source software, open source intelligence, open spectrum, open access copyright, and Open Society) but in this context, is distinct from the much better organized and much more reliable free/open source software (F/OSS) social network that shares source code which is publicly available (and modifiable). OSINT should also not be generally confused with OSIF (Open Source Information) on which OSINT is based. OSIF is any information that is publicly available; OSINT is an analytically-tailored intelligence product composed of OSIF that is designed to answer a specific tasking or to support decision-making rather than general research.

OSINT is distinguished from research in that is applies the process of intelligence to created tailored knowledge supportive of a specific decision by a specific individual or group.

Information collection in OSINT is generally a different problem from collection in other intelligence disciplines where obtaining the raw information to be analyzed may be a major difficultly, particularly if it is to be obtained from non-cooperative targets. In OSINT, the chief difficulty is in identifying relevant, reliable sources from the vast amount of publicly available information. However, this is not as great a challenge for those who know how to access local knowledge and how to leverage human experts who can create new tailored knowledge on the fly.

OSINT is now a subordinate part of modern Information Operations (IO), which consists of Strategic Communication (the message), OSINT (the reality), and Joint Intelligence or Inter-Agency Collaboration or Coordination Centers or Commands (JIOC, the technology). The U.S. military is heavily invested in OSINT, with the U.S. Strategic Command having the lead for IO, the U.S. Special Operations Command having the lead for the best operationally-oriented OSINT capability as it applies to the Global War on Terror, and the U.S. Central Command being the most important consumer of tailored operationally-oriented OSINT in support of the Multinational force in Iraq.

Current standard guides to OSINT are:

• NATO Open Source Intelligence Handbook
• NATO Open Source Intelligence Reader
• Intelligence Exploitation of the Internet.

OSINT applications

The High Level Threat Panel of the United Nations, and as stated there are ten threats to public security and prosperity, in this rank order: poverty, infectuous disease, environmental degradation, inter-state war, civil war, genocide, other atrocities (e.g. trade in women and children or kidnapping for body parts), proliferation of weapons of mass destruction, terrorism, and transnational crime. According to one expert on OSINT, 80% or more of the information needed to address those ten threats is openly available, but not readily accessible to the secret intelligence organizations. This same individual, writing in Forbes.com, has suggested that the secret intelligence world is "inside out and upside down" because it spends over 99% of its money on the 5-10% of the informaiton it can steal, and virtually no money at all on the other 90-95%. The recent emergence of Google.org (the charitable spin-off of Google as a foundation focused on using information to address threats like poverty, and the growing possibility of Amazon migrating to the sale of individual paragraphs of books for micro-cash, or direct access to authors by the hour, suggest that the most important developments in this discipline will be outside the secret federal world, and in the public domain.

The secret intelligence world, which has resisted any significant expenditures on OSINT for the past fifty years, is finally beginning to slowly adapt to the modern world. According to The Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction report submitted in March 2005, OSINT must be included in the all-source SECRET intelligence process for the following reasons (as stated in the report):

1. The ever-shifting nature of our intelligence needs compels the IC to quickly and easily understand a wide range of foreign countries and cultures. - … today’s threats are rapidly changing and geographically diffuse; it is a fact of life that an intelligence analyst may be forced to shift rapidly from one topic to the next. Increasingly, IC professionals need to quickly assimilate social, economic, and cultural information about a country—information often detailed in open sources.
2. Open source information provides a base for understanding classified materials. Despite large quantities of classified material produced by the IC, the amount of classified information produced on any one topic can be quite limited, and may be taken out of context if viewed only from a classified-source perspective. Perhaps the most important example today relates to terrorism, where open source information can fill gaps and create links that allow analysts to better understand fragmented intelligence, rumored terrorist plans, possible means of attack, and potential targets.
3. Open source materials can protect sources and methods. Sometimes an intelligence judgment that is actually informed with sensitive, classified information can be defended on the basis of open source reporting. This can prove useful when policymakers need to explain policy decisions or communicate with foreign officials without compromising classified sources.
4. Only open source can “store history.” A robust open source program can, in effect, gather data to monitor the world’s cultures and how they change with time. This is difficult, if not impossible, using the “snapshots” provided by classified collection methods. (The Commission on the Intelligence Capabilities, 378-379).

History of OSINT

The earliest known modern reference to the need for open source intelligence in the public service is by Quincy Wright. It is noteworthy that Carnegie Endowment for International Peace was his home at the time, and remains today one of the foremost producers of open source information. Other important institutions in this vein include the U.S. Institute for Peace, and the U.S. Army's Peacekeeping and Stability Operations Institute as well as its Strategic Studies Institute, both located within the Army War College. More recently, the Swedish Folke Bernadotte Academy had created, and opened to the public, the first structured course in information and intelligence sharing for multinational peacekeeping operations.

The current modern history of OSINT began in 1988, when the Marine Corps Intelligence Command (then Center) discovered that 80% of what it needed to create intelligence for policy, acquisition, and operations was not secret, not in English, not online, and not known to anyone in the Washington, D.C. area. The Marines attempted, from 1988-1992, to obtain official recognition of what the Aspin-Brown Commission would later call a "severe deficiency" in access to OSINT, to no avail. The Commandant of the Marine Corps, Al Gray, published his views, including a call for more attention to OSINT. Subsequently the Marines allowed Robert David Steele, the founding Special Assistant of the Marine Corps Intelligence Command, to organize a conference on OSINT in December 1992.

1990's

In the fall of 1992, Senator David Boren, then Chairman of the Senate Select Committee on Intelligence, sponsored the National Security Act of 1992, attempting to achieve modest reform in the U.S. Intelligence Community. His counterpart on the House Permanent Select Committee on Intelligence was Congressman McCurdy. The House version of the legislation included a separate Open Source Office, at the suggestion of Larry Prior, a Marine Reservist familiar with the MCIC experience and then serving on the House Permanent Select Committee on Intelligence staff. This legislation was defeated by a combination of opposition from Senator John Warner of Virginia, who feared that reform would reduce intelligence jobs in Virginia, and a letter from then Secretary of Defense Dick Cheney. Within the past several years, Lt. Gen Dr. Brent Scowcroft studied this issue again and repeated the recommendation that the three national agencies be removed from the Department of Defense, only to be dismissed by now Vice President Dick Cheney.

In August 1995 the Aspin-Brown Commission sponsored a benchmark exercise on Burundi in which a private sector party, with six telehone calls, overnight, producing the following:

• From the Institute of Scientific Information (ISI), the top 100 international experts on Burundi available for immediate debriefing;
• From LEXIS-NEXIS, the top international reporters on Burundi, available for immediate debriefing.
• From Oxford Analytica, twenty two-page reports on the political-military significance of Burundi at the Presidential, Prime Ministerial, and CEO levels;
• From Jane's Information Group, one paragraph summaries of every article ever published about Burundi, and new, tailor-made tribal orders of battle created overnight for this need
• From East View Cartographic, 1:50,000 meter combat charts with contour lines from Russia (the Americans do not have combat charts for 90% of the world--they rely too heavily on secret imagery satellites.)
• From SPOT Image in France, commercial imagery for Burundi, 100% coverage, cloud-free, less than three years old, in the archive and inexpensively available.

The CIA (which sponsors the Open Source Center today) had a small map of Africa and a regional economic study with flawed (Western) premises. There was no clandestine human intelligence, no imagery intelligence, and no signals intelligence, and of course no open source intelligence either.

Several DCI's in succession refused to implement the Aspin-Brown Commission recommendations, including the recommendation in 1996 that stated that US access to open sources was "severely deficient" and that this should be a "top priority" for both funding and DCI attention. Consequently, the US entered the new century completely ignorant of all of the open source information in all languages relevant to identifying and containing Al Qaeda, which has been active, publicly, since 1988.

Concurrent with the Commission's effort, the HPSCI commissioned its own review. Staff Director Mark Lowenthal has also published useful books on the intelligence process and the literature of intelligence.

In July 1997 then DCI George Tenet received a report conducted by Senior Intelligence Service officer Boyd Sutton. After interviewing virtually all of the Assistant Secretaries of Defense and State, and the heads of the varied intelligence organizations, the report recommended that the U.S. Intelligence Community, which continues to focus on seven "hard targets" and continues to ignore the Third World and all but two of the ten threats identified by the High Level Threat Panel of the United Nations, should spend $1.5 billion a year (a fraction of 1% of the total intelligence community budget) to provide a $10 million dollar a year "insurance policy" for each of 150 lesser countries and topics associated with instability and non-traditional threats. In July 1997 DCI George Tenet told the author of the report that CIA was in the business of doing secrets for the President, the report was to be archived, and none were to speak of this again.

Post-9/11

In issuing its July 2004 report, the 9/11 Commission recommended the creation of an open-source intelligence agency, but without further detail or comment. Subsequently, the WMD Commission (also known as the Robb-Silberman Commission) report in March 2005 recommended the creation of an Open Source Directorate at the CIA.

Following these recommendations, in November 2005 the Director of National Intelligence announced the creation of the Open Source Center. The Center’s functions were to include collecting openly available information and training analysts to make better use of this information. It absorbed the CIA's previously existing Foreign Broadcast Information Service (FBIS), originally established in 1941. FBIS head Douglas Naquin was named director of the Center.

In December 2005, the Director of National Intelligence appointed Eliot A. Jardines as the Assistant Deputy Director of National Intelligence for Open Source to serve as the Intelligence Community's senior intelligence officer for open source and to provide strategy, guidance and oversight for the Community's open source activities. However, Jardines does not have Program Authority, he does not have a permanent staff, and he is limited in his discretionary authority to $5M a year--the other $20M a year in new money is funding the Large Scale Internet Exploition contract that the Open Source Center has recently awarded.

The Director of National Intelligence's Chief Information Officer, MG Dale Meyerrose, USAF, has broken with the insular traditions of the past and sponsored both an open forum on open standards for information sharing, and a major conference open to foreigners, to discuss all aspects of the U.S. Intelligence Community's Intelink program for sharing and making sense of all sources of information. This is a major positive change, and could conceivably lead to the establishment of a Multinational Information Sharing System that is open to all legitimate governments and organizations. Note: the original Open Source Information System (OSIS) has been re-named as the unclassified subordinate element of Intelink, and to avoid its being confused with any network associated with the Open Source Center.

In January 2004 Secretary Rumsfeld's Undersecretary of Defense for Intelligence, Dr. Stephen Cambone told the Security Affairs Support Association (SASA) that he required nothing less than universal coverage in all languages, 24/7, a requirement that can only be addressed by creating a properly funded and structured OSINT network.

In February 2006 speech at the Council on Foreign Relations, Secretary of Defense Donald Rumsfeld seems to have acknowledged the importance of open media as a component of national security in the information age.

Librarians, Researchers, and Open Source Officers

Librarians play a role in the OSINT world. Each of the eight communities of interest below has its own librarians. While Librarians come together at the Special Libraries Association, and independent information researchers at the Association of Independent Information Progressionals, and Open Source Officers at IOP or in secret gatherings within their respective secret communities. As of early 2006, no global conference or network exists.

Traditional librarians have begun to migrate away from the hard copy acquisition and internal classification roles, and emerged as very talented "scouts" able to discover, disciminate, distill, and disseminate essential information for their respective constituencies.

Open Sources, Software, and Services

Open sources of information include the surface Internet, the deep Web, and electronic mail, as well as traditional media sources inclusive of niche media or industry-specific newsletters and online discussion groups. They include gray literature, subject-matter experts, and individuals who have direct observation or life experience knowledge.

Open source intelligence software cover eighteen distinct functionalities associated with data acqusition, analytic tradecraft, and intelligence production.

Open source intelligence services are as diverse and tailored as one can imagine. The specific services listed in the NATO reference include data data conversion; database construction and stuffing; document acquisition, human intervention; imagery interpretation & annotation, indexing & abstracting; international studies analysis; modeling & simulation; online collection; open source intelligence portals; private investigation; scientific & technical analysis; signals processing; and telephone surveys (also known as primary research).

Sources and notes

• One-Page Link Guide to OSINT References - Covers history, context, process, practice, and tools
• Forbes "Reinventing Intelligence"
• TIME "The New Craft of Intelligence"
• IO: Putting the I Back Into Dime - Monograph from the Strategic Studies Institute of the US Army War College
• The New Craft of Intelligence - Monograph from the Strategic Studies Institute of the US Army War College

Further reading

• Open Source Center - U.S. government arm focusing on open source intelligence under the DNI
• OSS.Net.