New Approaches and Methods in
Assessing Networked Information Services:
Issues and Lessons

Charles R. McClure
<cmcclure@mailbox.syr.edu>
Distinguished Professor
School of Information Studies
Syracuse University, Syracuse, NY 13244

Coalition for Networked Information Conference
Crystal City, VA April 1-2, 1997

Abstract

The notion of the "networked environment" encompasses a range of electronic networked activities and services. Minimally, the networked environment includes information services, products, hardware and software, telecommunications infrastructure, and resources that users and providers of the network receive via the various electronic networks. Increasingly, policy making officials, as well as users, are asking questions -- not about whether their organizations should invest in electronic networks -- but about ways to maximize the benefits and impacts of existing and/or evolving networking initiatives.

The concept of networked information services is an evolving one. Such services can be offered by individuals, libraries, computer centers, publishers, networks, government agencies, or a host of other organizations and groups with access to the Internet and the evolving National (and Global) Information Infrastructure. Networked information services comprise bulletin boards; email; listservs; remote access to distant databases, video, software, and high speed computing; and, can support collaborative efforts among geographically dispersed individuals--to name but a few possible services. Indeed, technological advances create an extremely fluid and rapidly changing networked environment.

Assessment techniques, however, related to network use by specific audiences such as government officials, teachers, business persons, researchers, students, librarians, etc. are just in the development process. Thus, some key questions concerning methods and approaches are:

• How does one insure the quality of data in these new assessment tools, e.g., what are we really measuring in web log analysis?

• How can these assessment techniques be "quick and clean" tools that encourage organizations to use them?

• Do different types of users (or users with different skill levels) require different types of assessment techniques?

• What are the costs for various kinds of assessment tools?

• How does the structure of the networked service (e.g., a construction of the website) affect the appropriate use of assessment tools?

• What types of performance measures are appropriate to determine the impact of networking initiatives on various stakeholder groups

• What types of evaluation tools can assist policy makers and researchers measure those identified performance measures?

To date, there has been little practical guidance offered to assess the impact of networking initiatives along the above dimensions. Techniques such as scripted user assessments, web log analysis, and web mapping (among others) need to be developed and tested in addition to traditional techniques such as focus groups, surveys, policy analysis, etc.

This presentation critiques techniques and issues related to assessing networked services from two studies recently completed:

• Assessing the Academic Networked Environment: Strategies and Options; ¹ funded by the U.S. Department of Education

• Sailor Network Assessment Final Report: Findings and Future Sailor Network Development,² a study funded by the Maryland Division of Library

It also offers new assessment perspectives from work in process from four additional studies:

• An Evaluation of the Federal Government's Implementation of the Government Information Locator Service (GILS),³ funded by the US General Services Administration;

• Quality Criteria for Evaluating Information Resources and Services Available from Federal Websites Based on User Feedback,⁴ funded by OCLC; and

• Analysis and Development of Model Quality Guidelines for Electronic RecordsManagement on State and Federal Websites,⁵ funded by the US National Historical Publications and Records Commission.

• Web Usage Statistics: Measurement Issues,⁶ funded , inpart, by the US General Services Administration.

These efforts suggest that developing assessment techniques in a range of networking environments is a multi-dimensional process due to the complex technological and behavioral nature of electronic networks. The presentation explores several methods to evaluate electronic networks, and identifies and discusses key issues and findings that affect the successful evaluation of networked services.

---

^1. Charles R. McClure and Cynthia Lopata, Assessing the Academic Networked Environment: Strategies and Options. Washington, D.C. Coalition for Networked Information, 1996. [Available, $15.00 <pubs@cni.org> 202-296-5098.] [Also appears on author's homepage: <http://istweb.syr.edu/Project/Faculty/McClure.html>

^2. John Carlo Bertot and Charles R. McClure, Sailor Network Assessment Final Report: Findings and Future Sailor Network Development, and Sailor Network Assessment Final Report Compendium. Baltimore, MD: Division of Library Development and Services, 1996. [Available through ERIC Clearinghouse and Division of Library and Development Services, 200 W. Baltimore Street, Baltimore, MD 21201, 410-767-0443.]

^3. William E. Moen and Charles R. McClure, Multi-Method Approach for Evaluating Complex Networked Information Services: Report from an Evaluation Study of the Government Information Locator Service (GILS), Denton, TX: School of Library and Information Sciences, January, 1997. Available at: <http://www-lan.unt.edu/slis/research/gilseval/acmdlib.htm>.

^4. Kristin R. Eschenfelder, John C. Beachboard, Charles R. McClure, and Steven K Wyman, Assessing U.S. Federal Government Websites. Government Information Quarterly, 14 (1997), in press.

^5. Abstract appears on author's homepage: <<http://istweb.syr.edu/Project/Faculty/McClure.html>>

^6. John Carol Bertot, Charles R. McClure, Jeff Rubin, and William E. Moen. Web Usage Statistics: Measurement Issues and Analytical Techniques. Syracuse, NY: Syracuse University, School of Information Studies, February 1997. Paper submitted to 1997 American Society for Information Science Annual Conference.

webmgr@cni.org