Assessing the Academic Networked Environment: Proposals

RATIONALE

The Johns Hopkins University is a highly decentralized institution, both geographically and administratively. Especially given this institutional structure, a ubiquitous and effectively managed network environment is a fundamental requirement for the advancement of our mission of excellence in research, teaching, and patient care. In recognition of this, the report of the JHU Committee for the 21st Century--a comprehensive and far-reaching self-analysis intended to assure Hopkins' continued leadership in higher education--included strong recommendations to expand, enhance, and effectively manage our electronic information resources (http://www.jhu.edu/news_info/c21/). Several major steps have already been taken in this direction, and the university is currently poised to make significant enhancements in our network environment on the technical, functional, and organizational levels. We recognize that in doing so it is absolutely essential that we have accurate and reliable measures with which to assess where we are, and ultimately to judge the success of our efforts in realizing the vision we share for Hopkins' technological environment.

PRIMARY PARTICIPANTS

Hopkins' participation in the CNI/AANE project will be lead by the two largest centralized academic information services organizations at the university: the Milton S. Eisenhower Library (http://milton.mse.jhu.edu/) and Homewood Academic Computing (http://www.jhu.edu/~hac/). Both organizations serve primarily the Homewood campus (Schools of Arts & Sciences, Continuing Studies, Engineering, and the Administration), but also have responsibilities and provide services which span the breadth of the university. We work closely together on many information technology-related activities. We believe that we have much to offer the CNI/AANE project and that we have much to gain by participating. The leadership of the university and of our divisions is committed to an on-going technology assessment process and to providing the necessary resources.

Also participating as part of the institutional team are the School of Medicine, Office of Medical Informatics Education (OMIE, http://infonet.welch.jhu.edu/~omie/), and the School of Continuing Studies, Information Technologies office. The educational activities of OMIE depend almost entirely on a robust and comprehensive network, and they actively conduct evaluative research in the use of these technologies to enhance medical education. The School of Continuing Studies is the most physically distributed of Hopkins' divisions and has the largest enrollment. Much of this student body is comprised of professionals who have high expectations of technology and specialized needs for time and location-independent access to information resources. Teaching and learning activities within SCS increasingly involve asynchronous and/or "distance" components, thus the effective use and management of the academic network becomes of paramount importance.

RELEVANT ACTIVITIES

A number of specific projects and recent developments at Hopkins have directly influenced our desire to participate in this project, as described below.

METROPOLITAN-AREA NETWORK

Johns Hopkins is in the process of implementing a state-of-the-art SONET network linking all of our primary academic and health campuses and soon our satellite campuses. Justification for the tremendous expenditures required for the establishment of this network was primarily based on perceived competitive pressures, rather than on objective analysis. We believe that this network infrastructure is essential to accomplish our goals, however we are unable to answer the hard questions about the qualitative and quantitative impact of this infrastructure on academic life.

FULL-TEXT JOURNALS

The rapid proliferation of digital information resources continues to intensify the demand for a robust network environment. Within the first quarter of 1997, the Eisenhower Library, the main research library of the University, will begin to provide access to both current and retrospective full-text electronic journals. Project Muse, the library's joint project with the University Press, has just completed the digitization of all 43 of the Press's journals. In a slightly different type of digital initiative, the library is also developing a production center dedicated to digitizing existing print materials for innovative applications in teaching and research.

NEW LMS

Despite the highly decentralized nature of our university, many of Hopkins' digital initiatives are creating the opportunity for a more collaborative environment among the library staff, faculty, other Hopkins libraries, and University departments involved in information technology, such as Academic Computing. This summer, four Hopkins libraries will begin the implementation of a shared library management system (LMS), marking the first time that Hopkins libraries have collaborated on a project of this scale. The distributed network environment at Hopkins contributes to the challenge of providing "quality" access to the diverse digital resources which continue to proliferate. Assessment of network resources and the quality of network service and support will be invaluable in assuring the success of these digital initiatives.

REMOTE NETWORK ACCESS

Hopkins has two rather disparate services providing remote access to the campus network. The Welch Medical Library provides "free" PPP service with an account on their shared Unix system. Homewood Academic Computing recently instituted a fee-based PPP service after conducting a series of pilot projects and user studies. The salient characteristics of these services (subscriber/modem ratio, software & documentation, etc.) are quite different. Neither serves the entire university population, and some sectors are without such service altogether. A university-wide committee also continues to investigate outsourcing of remote network access for all of Hopkins, perhaps including affiliates, alumni, and others. Decisions in these matters (fee vs. "free", in-house vs. outsourced, institution-wide or campus-specific) must be based on an objective assessment of the real needs of our user community, expected levels of access and service, current actual and projected utilization, and so forth. In many cases we don't have the tools to make such assessments or they are nearly impossible to implement in practice.

CLIENT/SERVER COMPUTING

Within the last two years we have decommissioned our academic IBM mainframe and established a system of network-based servers which far exceeds the functions previously provided. Examples include an email hub, news server, accounting server, e-list processor, webserver, and so on. Our computer labs and similar facilities are tied together via file, print, backup and other services into a highly interdependent whole. Desktop applications such as scheduling programs (e.g. Synchronize) typically rely on servers which may not even be known to the end user. In such a distributed computing environment, it is accurate to assert that "the network is the computer". The management of this network--and the responsive delivery of services to our users--requires a set of measures that are almost completely novel.

SPECIFIC INTERESTS

We are interested in "field testing" those assessment approaches and measures that we have not previously tried and are feasible in our environment, or that we have previously conducted but which would nonetheless be instructive to repeat and/or extend. This would include the following of those listed in the McClure/Lopata manual:

NETWORK TRAFFIC

We are especially interested in these measures because of configuration changes and capacity increases in our inter-campus network, the deployment of the new multi-site LMS, developments in remote network access, and the rapidly increasing usage of networked resources for instructional and research activities. In addition, Hopkins is now a charter member of the Internet II consortium, which will make proper network traffic analysis even more important.

USE

E-mail: universal email access, the new email hub, email software inconsistencies, a lack of standards for use of email lists and aliases, all require that we get a better handle on email usage. Public Sites: continuing competition for access to public technology resources, the expansion of the network to all student residences and to off-campus locations via PPP, consideration of required computer purchase policies for students, all these make public network access a critical metric to quantify.

SUPPORT SERVICES

Help Desk: a recently formed, cross-divisional "HelpDesk Alliance" will be using these measures to do comparative assessments of their services and then working to improve their complementarity. Network Repair and Services: the new head of Network Services is committed to assessing and monitoring these measures in order to achieve the highest possible service standards; the planned use of the Action Request System (see below) to track all network service requests will greatly facilitate this effort. Availability of Networked Resources: all four of the participating departments support and/or maintain such networked resources, but coordinated efforts and strategic planning have been missing due at least partly to a lack of clear definitions and reliable inventories.

USER SURVEY

We are particularly interested in quantifying how students are making use of network access in their residences (and faculty from their homes!), the impact of distributed instructional efforts (such as in the context of web-based curricular material for medical students rotating through clinical clerkships), and the effectiveness of network-based pedagogical activities in continuing education (conferencing, "groupware" tools, Web environments, etc.).

We are, of course, open to discussion and revision of the eventual distribution of tasks among the participating institutions, particularly as regards some of the measures which we routinely perform and for which we could provide advice and assistance to others interested in conducting them (e.g. CWIS usage statistics).

PREVIOUS ASSESSMENT EXPERIENCE

The participating Hopkins offices and individuals have a wide variety of related experience that is specifically applicable to this project. A sampling of such activities are described below.

ACTION REQUEST SYSTEM

ARS is a client-server application that functions as a "trouble ticket" system used to follow all service requests through to completion and as a "knowledgebase" of problem solutions. It also provides the ability to do detailed analyses, enabling us to gauge performance by groups and individuals, to identify patterns of support needs overall and within specific domains, and to spot emerging trends for which we can adjust our support and training activities. Simple monthly reports of who is getting support calls and how they are being handled (e.g. http://www.jhu.edu/~hcic/checkbits/v12/sept96.html#report) provide invaluable feedback. This system can be useful in many kinds of support-oriented assessment activities.

USER SURVEYS

Academic Computing has undertaken a comprehensive evaluation of service quality and access levels through a variety of survey methods. These include: [1] a Microcomputer Support Group "Satisfaction Survey" -- a postcard left with the client on each and every service call which addresses the technician's performance, resolution of the problem, and timeliness of the service provided, which is then tied to detailed problem and demographic data through the assigned ARS tracking number (see above); [2] an annual Computer Lab usage survey (http://www.jhu.edu/~hcic/checkbits/v11/jan96.html#LabSurvey) -- given on-line in the computer labs (a web form automatically loads when Netscape starts up) and focused on usage, access, and support issues; [3] a SupportNet survey (http://www.jhu.edu/~hcic/checkbits/v11/jan96.html#SupportSurvey) -- a paper survey form, mailed out to clients and focused on "quality of service" issues for our "virtual help desk" program; and [4] an annual Faculty Survey -- now done both on paper and electronically (web form or diskette), used to identify faculty technology needs, concerns, directions, and to gather feedback on our services, programs, and proposals.

FOCUS GROUPS

Focus groups have been conducted by MSE Library to develop a better understanding of the challenges and satisfaction level of users of electronic resources in the library. A series of structured discussions were held with students and faculty to assess the "barriers" they encounter when using electronic resources, and to determine their preferences in terms of getting assistance, e.g. context sensitive help within the resource, printed supplemental documentation, one-on-one assistance with librarians. These data are now being used to improve in-house assistance, and will also be useful in developing "user assistance" strategies for remote users.

SPECTRUM Network Management System

In conjunction with uniform installation of "manageable" network devices, we now use this system for network traffic load monitoring and analysis. As we phase in monthly charges for network connectivity, it has been critical in assessing number, location, and types of devices attached to the network (reality does not necessarily correspond with what's in our database), and in providing the data to support the anecdotal belief that our connection to the Internet was inadequate (it has since been upgraded to 4MB/sec). SPECTRUM can be programmed to perform many such tasks automatically.

THE TEAM

[LIAISON] Lee Watkins Jr, MA, is Assistant Director of Academic Computing for the Information Center. He is Project Manager for JHUniverse (http://www.jhu.edu/), for the Residential Computing program, and for the School of Arts & Sciences' computer classrooms. He serves on the university-wide Subcommittee on Electronic and Distance Education (http:/www.jhu.edu/~wse1/distance/), established as a result of the C21 report. He recently completed a top-to-bottom information technology assessment and planning exercise for the Student Affairs division. Previously he was a Systems Analyst and Manager of the Statistical Support Group at the University of Texas at Austin Computation Center. Mr. Watkins has conducted numerous large-scale, data-intensive statistical analyses and has done contractual work with Survey Research Associates in Baltimore.

Pamela L. Higgins, MLS, MA, is Head, Library Systems at the Milton S. Eisenhower Library, where she has been since 1986. Her previous assessment experience includes usage assessment of electronic resources, e.g. Online Public Catalog, citation and full-text databases as well as analysis of library network requirements prior to library-wide infrastructure upgrade. The library is currently implementing its "second generation" infrastructure upgrade in the context of a major library renovation.

Ronald S. Hudson, MBA, is Director of Information Technologies for the School of Continuing Studies. Previously he was Assistant Director of Academic Computing for Network Services, in which position he was responsible for the complete overhaul of the campus network & upgrade to switched Ethernet and ATM (under a grant from the Pew Trusts), establishment of a network management system, and extension of the network to all student housing. He was also instrumental in the planning and implementation of the institutional SONET network.

Harold P. Lehmann, MD, PhD, is Assistant Professor of Pediatrics and Director of the Office of Medical Informatics Education. Prof. Lehmann has conducted extensive assessment research involving (among many other things) effective use of educational technologies (and is an active practitioner of these in his own courses) and clinical decision support systems. His office has created and developed a number of innovative curricular programs and tools.

---