A Research Guide
for Identifying Indicators of the
Effectiveness of International Environmental Agreements

© Ronald B. Mitchell, 2000

Overview

This guide provides a strategy for conducting research into the effectiveness of international environmental treaties. The fundamental goal of the strategy delineated in the following pages is to guide the researcher through a systematic set of steps for identifying indicators relevant to evaluating the effectiveness of an environmental treaty and determining whether those indicators are available. The guide seeks to allow the researcher to identify systematic answers to the following questions:

• What are the environmental goals and behavioral requirements delineated in the treaty?
• What are useful operational indicators (or proxies) corresponding to these goals and requirements that, if available, would provide the basic data needed by an analyst to evaluate a treaty’s effectiveness?
• Of these "relevant" indicators, which are currently available in a specified (and extensive if not exhaustive) set of likely sources?
• Are there other indicators of environmental quality or environmentally-related behavior that, although not initially identified as relevant indicators, might nonetheless be useful in evaluating the treaty’s effectiveness?

In short, the guide provides a systematic way of identifying environmental and behavioral indicators that are both available and relevant to evaluating an environmental treaty’s effectiveness.

The Nature of the Problem

Evaluation of environmental treaties has, to date, been stymied by structural obstacles that hinder, even if they do not prevent, identification of indicators that are both relevant to and available for conducting such evaluations.

On the one hand, a wide variety of groups have taken on the task of identifying and often making available data relevant to a particular treaty. Thus, for example, many secretariats provide tables of data that could be used in evaluating the treaty or treaties for which they are responsible. Many governments, non-governmental organizations, and private corporations have also begun providing similar data, sometimes covering a wide range of different treaties. Rarely, however, do these groups have incentives to undertake in-depth and unbiased assessments of the effectiveness of these treaties.

On the other hand, those interested in evaluating the effectiveness of environmental treaties are often unaware of and unskilled at identifying the range of available data relevant to a particular treaty. No systematic and comprehensive catalog of these data sources is available, particularly not one which provides an overview of the parameters that would help an analyst interested in such questions to determine whether a dataset with particular characteristics is available or which would help them recognize the availability of datasets with certain characteristics that make them advantageous for use in evaluating a particular treaty. Given the increasing and frequently changing nature of what data is available, as well as the variety of places in which one might be able to locate such information, data quite valuable to an analysis is frequently overlooked. Most scholars and practitioners lack the skills, the time, or both to undertake a systematic search for currently available data on a particular treaty. This often leads either to a discouraged decision not to evaluate a particular treaty because "the data simply isn’t there," or the use of indicators that are not particularly good proxies of the effectiveness of a treaty. In the latter case, the relief at eventually finding an indicator that bears the slightest resemblance to the indicator that the analyst would like to have data on often creates the temptation to build an argument that uses less-than-fully-adequate data justified on the basis that it was available rather than that it was particularly relevant to the question at hand.

In short, those providing the data that would be relevant to evaluating the effectiveness of a treaty usually do not have the incentives or resources to undertake unbiased analyses. At the same time, the distributed, non-systematic, and often random approaches to making such datasets available mean that it is at best difficult and time-consuming, and at worst, impossible, for those with incentives to undertake independent analyses to easily identify what data relevant to analyzing a particularly treaty or set of treaties is available.

This research guide seeks to facilitate efforts by scholars and students to conduct rigorous, high-quality, positivist research into the effectiveness of international environmental treaties despite these contextual obstacles.

Summary of Procedures

Conducting systematic research on the effectiveness of international environmental treaties requires several distinct steps. Mitchell and Bernauer lay out one approach to these problems in their "Empirical Research on International Environmental Policy: Designing Qualitative Case Studies" (Journal of Environment and Development 7:1 (March 1998), 4-31).

1. Identify environmental goals and behavioral requirements in the treaty text. Brainstorm as many observable, and measurable, indicators as possible for each goal or requirement. These are described more extensively under "Identifying Goals, Requirements, and Relevant Indicators" below.
2. Identify the secretariat for the relevant treaty and determine whether data on these relevant indicators are available from the secretariat, using their website or direct contact.
3. Identify whether relevant indicators are available in a range of other likely sources.
4. Catalog information regarding your data source.

When complete, the researcher should have identified one or more indicators of treaty effectiveness that are both relevant and available or, alternatively, be able to claim that such relevant indicators were not available at the time and in the sources used in the above procedures.

Identifying Goals, Requirements, and Relevant Indicators

This stage seeks to identify treaty goals and requirements so they can be used to generate a list of indicators relevant to evaluating a treaty’s effectiveness. These procedures are crucial to identifying the explicit goals and rules that those negotiating the treaty set for themselves as standards of success. In many cases, "common wisdom" about what a treaty seeks to accomplish (and hence how its success should be judged) can stray significantly from the goals states actually set for themselves and the operational requirements they placed on themselves.

Procedures:

1. Read the treaty through completely but quickly to gain an overall sense of the treaty’s goals and requirements. Then, re-read the treaty more slowly, and identify the explicit goals that motivated its creation, usually identified in the preamble but also elsewhere, as well as the explicit behavioral requirements laid out in the substantive articles. Make sure that you have a sense of what the treaty intends to do, in the words and definitions of that treaty. Seek to answer the following questions:
• What were the treaty negotiators seeking to accomplish?
• What is the problem the treaty seeks to solve?
• What are the specific substantive rules established by the treaty?
• What are the behavioral requirements placed on the signatories to the treaty?
• What behaviors are parties to the treaty required to report on?
2. Distinguish, as needed, between what the treaty seems to want to accomplish and what it actually requires states to do.
• For example, the Framework Convention on Climate Change seeks "to protect the climate system for present and future generations" (Preamble) which is made more specific in Art. 2 as the "stabilization of greenhouse gas concentrations in the atmosphere at a level that would prevent dangerous anthropogenic interference with the climate system." However, the convention then requires only that developed states "adopt national policies and take corresponding measures on the mitigation of climate change, by limiting its anthropogenic emissions of greenhouse gases and protecting and enhancing its greenhouse gas sinks and reservoirs" (Art. 4).
3. Note that a single treaty often has several goals and numerous requirements, some quite different from each other and others close but not quite the same. Identify as many distinct goals and requirements as possible since each can provide one or more possible indicators of the treaty effectiveness.
4. "Brainstorm" a list of potential indicators for each goal and each requirement.
• Begin by identifying specified indicators which the treaty delineates as standards against which it wants to hold itself.
• Now, identify "related indicators" which are not specified in the treaty but would serve as a good "proxy" of achievement of the goals that are specified. To give an example, the amount of oil pollution in the ocean would not be a relevant indicator for the Framework Convention on Climate Change, but "greenhouse gas concentrations in the atmosphere" and "sea-level rise" would be relevant specified indicators and "average global temperature" could be a relevant related indicator even though it is not mentioned anywhere in the convention.
5. Using the secondary literature on the treaty, identify other indicators of effectiveness by identifying those used by other scholars, making use of and updating indicators used in those studies and avoiding unnecessary "re-inventing of the wheel."
6. Taken together, this should produce a list of relevant indicators for a treaty that may be available.

Identifying If Relevant Indicators Are Available

Armed with a brainstormed list of relevant indicators, you can now proceed to determining whether such data is available. The following procedures seek to allow you to systematically determine whether an indicator that has been deemed relevant to evaluating a particular treaty is available, and if not, to use a consistent search method that can document, and hence provide confidence, that the indicator is not available in an extensive, although by definition not comprehensive, set of likely sources.

Procedures:

1. For each of your brainstormed relevant indicators, take each of the following steps until you have found adequate data.
2. Go to the secretariat’s web site, if there is one, and look for links to data sources.
3. For each indicator, develop a list of search terms and search phrases to use in searching for the availability of an indicator in the databases listed below and any others you can identify.
4. If you find a data source, make sure to record the source AND the search term that uncovered it so that you can re-find the data yourself and provide the information to others.

Starting databases to search:

• All Secretariat web pages http://www.uoregon.edu/~iea/ciesin/secretariats.html
• Statistical Universe (Lexis/Nexis) http://web.lexis-nexis.com/statuniv and follow link for Statistical Searches (electronic version of Index to International Statistics)
• AccessUN http://infoweb.newsbank.com (index to UN documents, useful for reports)
• EU’s European Environment Agency’s Data Warehouse http://badger.eea.eu.int/warehouse
• Infoterra http://unep.unep.org/infoterra
• EU’s Statistical Office EUROSTAT (environmental section only) http://europa.eu.int/en/comm/eurostat/index.htm
• NASA’s Global Change Master Directory http://gcmd.gsfc.nasa.gov
• WHO Statistical Information Service (environmental section only) http://www.who.int/whosis
• UNEP’s Global Environmental Outlook (1997) tables http://www.unep.org/unep/eia/geo1/ch/toc.htm
• UN Food and Agriculture Organization http://apps.fao.org/default.htm
• UN Statistics Division (environmental and social indicators sections only) http://www.un.org/Depts/unsd/statdiv.htm
• UN University’s Environmental Information Sources list http://www.inweh.unu.edu/unuinweh/enviro.htm
• IISD’s Information Center http://iisd1.iisd.ca/ic/
• National Environmental Data Index http://www.nedi.gov/
• UNEP Global Resource Information Database (GRID): http://www.grid.unep.ch/
• World Resources Institute: http://www.wri.org/datainfo.html especially the World Resources data link
• World Conservation Monitoring Centre: http://www.unep-wcmc.org/
• OECD Environmental Documents http://www.oecd.org/env/lists4.htm, particularly hard copy of OECD’s Environmental Indicators
• University of Michigan’s Statistical Resources on the Web Environmental Statistics List http://www.lib.umich.edu/libhome/Documents.center/stenv.html (this is a list of other web sites, not a list of sources)
• UNCED CD-ROM of country reports
• World Bank’s World Development Indicators CD-ROM and Social Indicators of Development Dataset from CIESIN
• UN Official Document System – Official Documents of the UN http://www.ods.un.org
• UN’s Infonation for non-environmental statistical information http://www.un.org/Pubs/CyberSchoolBus/infonation/e_infonation.htm
• Inter-university Consortium for Political and Social Research (ICPSR) http://www.icpsr.umich.edu/ [follow links for Table of Contents and Access Holdings and then search using Abstract as well as Title options]
• UN Statistical Yearbook
• General "environmental data" and "environmental indicators" and "environmental statistics" search on the web, in StatUniverse, and in other databases, both electronic and otherwise
• World Cat and Library of Congress for books on the subject
• Article databases:
• Environmental Sciences and Pollution Management
• GeoBase
• PAIS
• Article1st
• Uncover
• Lexis-Nexis Academic Universe - Law Reviews http://web.lexis-nexis.com/universe and follow link to Legal Research and then Law Reviews

Identifying Treaty Secretariats and Data Sources

This stage seeks to identify contacts for a treaty secretariats that can assist the scholar in identifying both relevant indicators and available data sources. Secretariat staff are very likely to have extensive knowledge regarding which behavioral and environmental data relevant to a treaty is available and which is not. Because most scholars working on a treaty will have been in contact with the secretariat, and because a wide range of data may be available only to the secretariat and groups and contractors working for the secretariat, there may be far more information available than would be easily identified through other sources. In many cases, secretariats will provide access to closely-held data if the scholar promises to keep the information confidential and/or provides the completed analysis in exchange.

Procedures:

1. Read the treaty to identify the name of the organization responsible for secretariat duties.
2. Start with the Secretariat List established by Ronald Mitchell at http://www.uoregon.edu/~iea/ciesin/secretariats.html.
3. If you cannot find the secretariat for a treaty there, search the Union of International Associations (http://www.uia.org/webintt/aawebndt.htm) web site. This site lists web (if available), fax, phone, and snail-mail contacts for the vast majority of international organizations and treaty secretariats.
4. Examine other collections of international relations websites, such as those of the International Studies Association (http://www.isanet.org) or the International Studies Network (http://www.newisn.ethz.ch/linkslib).
5. If you still fail to identify secretariat contacts, search the web using different search engines, using popular and official names for the treaty and secretariat to identify the main web site for the treaty in question.
6. Once identified, always ensure that you have the official secretariat home page for the site and make sure to get the exact address for contact information for the secretariat.
7. For those for which web information does not appear available, look at the hard copy of the Yearbook of International Organization, since this provides a more comprehensive listing of information than on the UIA web page.
8. Once identified, spend time perusing the secretariat’s website, or contact the secretariat by phone, fax, or snail-mail to request information on issues of treaty effectiveness. Given the dearth of resources available to most secretariats, an offer to undertake an analysis and provide the results to the secretariat free of charge upon completion can often open many doors that would not otherwise be available.

If you identify a secretariat not already on CIESIN’s website, please send CIESIN the following information:

• Relevant treaty name
• Secretariat official name and acronym
• URL for the secretariat home page, and treaty home page, if different

OR

• Mailing address
• Phone number
• Fax number
• Email address
• Information officer, if any – preferably the name for the position, not the person

Cataloguing Data Source Information

Since it is often difficult to resurrect where you found data after the fact, it is important that you catalog the following information at the time that you first find the data. It will be helpful to the scholarly community more generally if you can also provide this information to CIESIN so that CIESIN can develop a larger meta-database of information relevant to studying the effectiveness of environmental treaties.

Information to collect and catalog:

• Brief name of data: Provide a brief title that will allow quick identification
• Official title for data: Provide the fullest title possible, based on the original source document
• Source of data: Provide the name of the organization or individual that has produced the information.   Provide most complete citation available. Provide the URL or the hard copy citation, including exact pages, for the data.   For URLs that are identified through data compilations, provide exact sequence of choices to get to data set and date of website access.
• Units of indicator: Provide the metric used to measure the indicator, e.g., tonnes of SO2, or violations
• Time period of metric for indicator: For example, annual, every five years, monthly
• Spatial scale of indicator: For example, global, regional, national, subnational, individual, continent, region, country, substate aggregations, municipality, river basin, wetlands, national parks, hectare, etc.
• Actual, estimated or projected data: Delineate whether data is actual, estimated, or projected.  Where necessary, provide separate entries for different year spans, even if for same data set
• Years: Either separated by commas or by a hyphen, e.g., 1980-1996 or 1980, 1985, 1990, 1995
• Countries involved: Provide a list of all countries for which at least some data is available; e.g., Austria, Czechoslovakia, Denmark, Germany, Finland, Hungary, Norway, Sweden, Switzerland, USSR.
• Other clarifications of data: E.g., (not legends or footnotes available from chart itself, but more general clarifications of what data represents that are not captured in foregoing parameters)
• Treaty goal or requirement to which indicator is related: Specify the goal to which the indicator is related.
• Location of explanatory notes: Identify where clarifying notes such as footnotes are available providing further clarification on the data.
• Contact person for data: Identify a contact person at the secretariat or elsewhere, if one is provided.