Virginia Polytechnic Institute and State University

 Virginia Tech’s Preliminary Plan: Assessing the Academic Environment

Introduction

In recent times most institutions of higher education have had to reconsider policies and procedures in the face of close scrutiny by the public at large. Traditions such as tenure, shared governance, and the focus on conventional degree programs are being challenged, and modified often as a result of pressure from governing boards and state legislators. Rising costs and declining budgets demand increased efficiency while changing demographics require programs that are responsive to a more diverse population of learners The result is that academia is being compelled to operate more like industry. Trustees and governors mandate greater efficiency, clearer goals, and oversight procedures, such as post tenure review processes and proficiency standards. Reductions in state support at a number of public universities combined with additional regulation are leading a renegotiation of institutional roles as state agencies. Both the benefits and constraints that state support imposes are being reconsidered in light of new economic realities. And, while technology is not the cause of these phenomena, computing and telecommunications technologies (the network) is perceived by many to be the vehicle that will enable higher education to respond to its critics and garner sufficient revenue to remain viable in a new century.

Challenged not only by participation in the CNI Assessing the Academic Environment Project but also by to demonstrate accountability in the environment described above, Virginia Tech’s Information Systems project participants submit the following preliminary plan for evaluating various aspects of our network activities and services. The benefits that we expect to gain from this process are the:

• development of systematic, regular assessment techniques
• collection of baseline data
• evolution of longitudinal data collection
• establishment of a collaborative relationship among disparate IS units
• improvement in planning and decision-making ability
• opportunity to work with colleagues at other institutions.

---

Communication Network Services

Clark Gaylord

Narrative

One aspect of network assessment is how much users use the network and what their experience with the network is. It is important when providing network resources that these resources be sufficient for the users’ needs. In today’s global Internet environment, this is not always feasible, since users depend on connectivity to sites outside the control of the campus network provider. By comparing the ways users connect to the network, and the different network resources they utilize, their needs can be better met.

Questions to be answered: We wish to understand better:

1. How users connect to the network
2. Where users are connecting to on the network

Methodology and Performance Indicators

The appropriate measures of network utilization are dependent on how users access the network. The two principle access methods are via modem through the Virginia Tech inbound modem pool and via a dedicated Ethernet connection.While the modem pool is a dedicated resource while the user is connected, other users can use this resource at other times, so contention is a function of how many concurrent users the modem pool is servicing. Hence, our performance indicator for modem usage is how many users are being serviced relative to the number of available modems. This measurement can be collected using various network management tools from the terminal servers that control the modems.On the other hand, Ethernet contention does not occur at the connection itself but rather at points where this connection enters other components of the network. This is dependent on the network topology relative to this Ethernet connection. Most Ethernet connections at Virginia Tech are part of a shared building Ethernet backbone which then connects via an Ethernet switch in one of the major switchrooms to the campus ATM backbone. Some buildings are undergoing upgrades where the users’ Ethernet connections are made to an Ethernet switch in their building, and these switches have high speed (e.g., FastEthernet) uplinks to the campus backbone. Due to the varied nature of the Ethernet topology, a single measurement is not likely to be appropriate for assessing an individual’s network performance. An especially interesting aspect of desktop Ethernet access, however, is the impact of the building upgrades to switched Ethernet technologies. To assess this, we can compare network performance before and after a building upgrade. Since the usual measures of network utilization do not compare well between shared and switched Ethernet, this measurement should be made at a higher level, for example the time to complete a file transfer. Therefore, we propose assessing network performance, both within a building and across campus, with the time to transfer a file, for example via FTP. This transfer time can be compared before and after a building upgrade to compare these technologies.Another major point of network congestion is assessing access to the Internet. While this is largely beyond the control of the network management staff, it may be of some value to better understand which sites are used most by Virginia Tech, and how responsive these sites are. As the sites on the Internet are quite varied, there is little that can be measured with respect to comparable performance indicators. It is possible, however, to know which Internet sites are used most frequently, and these data can be collected from network devices.

Value

The principle objective of this aspect of network assessment is to better optimize the allocation of network resources. The two most common access methods, modem and desktop Ethernet, are quite different in their capacity planning issues, but each can be assessed individually to determine if current resources are adequate. Additionally, the assessment of current building upgrade plans will provide valuable feedback to network planning. Finally, by better understanding Internet utilization, we can assess whether the existing connection to the Internet is sufficient, and whether steps can be taken to improve this access.

---

University Libraries

Eileen Hitchingham

Narrative

We are living in a time when change in the higher education information environment is one of the few constants we will experience. This is likely to be true for several decades. Most information wrappings or content holders –things like journals, books, web pages, newspapers, media and reports -are changing, and will continue to change. The mechanisms for delivery of information content are changing also.

One mission of a library in a college or university is to provide the information resources and related support services that students need for learning, and that faculty need for teaching and engaging in research. In our current evolutionary information environment the concept of a five or ten year plan to carry out this mission should be rejected. For example, a pre-1994 plan which ignored the web as an elemental factor in information delivery, would have major deficiencies today. We need plans for the short term – one or two years-so that evolutionary developments, — new technologies, new user expectations, new information services — can be factored into continual operational changes. This means that libraries will be making more decisions more quickly, and will need information measures that are themselves evolving rather than static. There is little relevant baseline data for decisions, and baselines that are established will need constant examination to understand how changes in contributory elements affect the baseline’s validity.For this reason, we are proposing to focus on measures that detect changes in the information habits and needs of our users as the users choose to work with different sources of information. Specifically we are examining when, how, and why students access the library web, and for the entire community we are looking at the use of specific resources on the library web which have cost and processing implications.

Questions to be answered

1. Library Web

The University Libraries web is a gateway for the Libraries’ networked resources, and gives information about operations and services. We don’t know its utility for students or why they use it. Questions we will examine include:
• How many new or returning students have ever connected to the library web page?
• For students who have connected to the web page, what first level information path(s) do they say they followed?
• At the end of the term, in a class that links to the library from the class web, how many times will the library link be used in relation to the other links on the page?
• At the end of the term what will students say they have used on the library web page?
• At the end of the term what will students say about the frequency of their use of the library web page?
• Is there a relationship between the student’s attitude toward computer and the frequency of use of the library web page?
• Is there a relationship between frequency of use of the library web page and a term assignment that requires information for a project, presentation, or paper?
• What percent of network traffic originating on the campus network, connects to the Libraries?
• What percent of network traffic from outside the campus network connects to the Libraries?

2. Electronic Resources

We subscribe to many searchable indexes, abstracts and full content journals that are linked on the Libraries’ web. Questions we will examine here include:
• By title, what is the frequency of use of these information resources on the library web?
• How does the frequency of use of web resources compare with the frequency of use of paper indexes, abstracts and journals in the library?
• How does the frequency of use of all paper resources in the library compare with hardcopy use from the previous year?
• Over the five most recent years, can an up or down trend in the use of the library catalog system, VTLS, be detected?
• Where would students want to find an electronic journal or database that the library subscribes to -from a web listing or in the library catalog system?

Methodology

We will participate in the surveys that are being conducted by Educational Technologies in selected Biology and Philosophy classes that make use of networked resources. Specific details regarding the methodology are evolving.

1. Library Web

• How many new or returning students have ever connected to the library web page?
  Demographic question determines whether student is first semester at VT or beyond.
  Screen capture picture shows library web page.
• For students who have connected to the web page, what first level information path(s) do they say they followed?
  Students are asked to circle, on the picture, functions that they have used.
• At the end of the term, in a class that links to the library from the class web, how many times will the library link be used in relation to the other links on the page?
  From web traffic analysis
• At the end of the term what will students say they have used on the library web page?
  Students will be asked to circle items on screen capture picture.
• At the end of the term what will students say about the frequency of their use of the library web page?
  Question on the survey regarding use of the library web page during the term
  Never
  1-5
  6-10
  More than 10
• Is there a relationship between the student’s attitude toward computers and the frequency of use of the library web page?
  Correlation of computer attitude value and frequency of use data.
• Is there a relationship between frequency of use of the library web page and a term assignment that requires information for a project, presentation, or paper?
  Question on survey that asks whether student had any assignments, in any classes that term which needed information for a project, paper, presentation etc.
• What percent of network traffic originating on the campus network, connects to the Libraries?
  Data available from VT networking services
• What percent of network traffic from outside the campus network connects to the Libraries?
  Data available from VT networking services

2. Electronic resources

• By title, what is the frequency of use of these information resources on the library web?
  We will use counting software to determine the frequency of use of those resources on the library web that connect to the vendor via a cgi script. We will not be able to count the use of resources that are recognized at the vendor’s site by IP address. Many of our resources are scripted.
• How does the frequency of use of web resources compare with the frequency of use of paper indexes, abstracts and journals in the library?
  We will count items picked up in the library after use.
• How does the frequency of use of all paper resources in the library compare with hardcopy use from the previous year?
  We will compare materials picked up in the library in 97/98 to the number of items noted in 96/97.
• Over the five most recent years, can an up or down trend in the use of the library catalog system, VTLS, be detected?
  We will count logins to the VTLS system.
• Where would students want to find an electronic journal or database that the library subscribes to -from a web listing or in the library catalog system?
  We will develop a question for the student survey.

Value

The biology and philosophy sections are introductory courses required of most VT students. This means that the sample of students could be considered as representative of new students at VT. We have more than 10,000 students in their first or second year of work at Virginia Tech. It is difficult to reach all of these students with any consistency. Where we can we want to use this sample group as a stand-in for students in their early years at VT.

1. Library Web

We believe that the library web can be a good vehicle for information communication with student users. We want to know whether we need to tell new students that a library web page exists, or whether it is likely that they will already have tried it. If they have tried it we want to know what they use, and why they use it. We think that most library use of library resources, whether those on the web or paper resources, is motivated by assignments in class. We want to look at this connection and frequency of use of the web. Much library assistance is still one-to-one. Some understanding of the computerphobic student’s frequency of use of the library web by the end of the term, can give us some insight into whether the one-to-one assistance needs to also continue to be face-to- face, or whether we can explore other technologies to provide similar assistance, but in a different way – perhaps with technologies similar to CU-SEE-Me The magnitude of the network traffic to the library is useful for considering priorities in staffing resources, i.e., is it big, small, or somewhere in-between when considered against overall network use.

2. Electronic resources

Whether through consortia or individually libraries are spending big dollars for electronic resources. Many believe that there will be a major shift to electronic resources within the next few years, particularly in the scientific/technical/medical literatures. While electronic resources offer no cost savings right now, they do make it easy to use materials when and where needed. It is clear that the maintenance of parallel systems, keeping the item in paper and having it electronically available, is not a good investment for an extended period. This is true even if the electronic resource is offered with the paper resource for the same cost – processing and housing costs continue for the paper items.The Tulip study suggested that many users have an emotional attachment to paper library resources. That study examined user behavior with materials that were narrowly focused and fairly difficult to access. We want to examine some base information regarding the use of electronic resources that are easier to find, use and to print. With the availability of more electronic publications we might expect to see a decline in the use of print resources of a similar nature. It’s sensible to assume that the archiving of electronic resources will be addressed at a more global level and the solution will not be improved by every local library maintaining electronic and paper versions of resources for just in case situations. The questions we are posing here are designed to give us some sense of whether the emotional attachment to paper resources is an issue of magnitude that should be factored into considerations of migration to electronic resources.How to make users aware of electronic resources is another question to be examined. Should electronic resources be cataloged in the online catalog, or are they too ephemeral to merit this more costly level of processing. Should electronic resources be included as live links on a web page, but lose the richness of detail which is associated with traditional cataloging. Many libraries are doing some variation of both. This is not scaleable. We might consider that the web is also changing where people expect to find information. Even if the catalog system has live links to the resource will users even consider going there? We want to explore what student participants in this study expect to find on the web, and what they will go into the catalog to find.

References:

Odlyzko, Andrew, (, May 8, 1997). Silicon dreams and silicon bricks: the continuing evolution of libraries -Preliminary draft. [Online]. Available: http://www.research.att.com/~amo/doc/silicon.dreams.txt[1997, May 23].Tulip, Final Report,(July 18, 1996). [Online]. Available: http://www1.elsevier.nl/homepage/about/resproj/trmenu.htm [1997, May 23].

---

User Services

Kim Homer

Virginia Tech’s home town, Blacksburg, Virginia, has been called “the most wired town in America.” (Reader’s Digest article on the Blacksburg Electronic Village) But until computers are as simple to use as telephones, computer users will need support. Our mission is to enhance the teaching and learning environment through the use of networked computers. We plan to use both objective and subjective measures to assess our effectiveness in providing computer support.Virginia Tech’s computing environment is distributed; all resident students and most faculty and staff have access to a data connection, and off-campus students can access the network through the campus modem pool. Most students, faculty and staff have personal computers which they use for e-mail, WWW, word processing, and computation. There are public computer labs on campus, and some departments have computer labs for special applications, but most computing is done from the user’s home or office computer. There are no “shell” accounts or computer clusters, so each user needs to have a TCP/IP connection at his desktop. About half of the residence halls and most of the faculty and staff offices are wired for 10BaseT Ethernet. The rest use SLIP or PPP to access e-mail, Web pages, and other networked resources. Each student, faculty and staff member is assigned a personal user ID, called a PID, which he uses for network authentication and e-mail. E-mail is provided through a single POP server, which averages about 100,000 POPmail message deliveries per day.While we have moved computing from the mainframes to the desktop, support remains centralized. User Services’ help desk (4HELP) strives to be the single point of contact for all computing problems. Until three years ago, 4HELP did not support undergraduates, and most of the questions were about use of the mainframe. But the use of the network has increased dramatically since then, and the number and types of questions have changed as well. For the past five years, we have collected monthly statistics on the volume of questions, the type of user, the mode of question (call, e-mail, or walk-in), and the type of computer systems used, e.g.,VM/CMS, UNIX, Macintosh, PC. For the last two years, we have also tracked the category of question and the response time. We plan to correlate these numbers with increased use of the network, which we will do by looking at POPmail deliveries, DropAdd sessions, and the number of activated PIDs. So far, our statistics show that undergraduates using PC’s generate most of our questions, and that most of their questions are about network access. In spite of an increased call volume, our response time has actually gone down in the last year.

Methodology:

User Satisfaction Survey

 

Assessing user satisfaction is more subjective, and therefore more difficult. The Educational Technologies group will be administering surveys to select Biology and Philosophy classes which make use of networked resources. User Services will add the following questions to the survey:

Network Accessibility:

1. Do you have your own computer? If so:
   How would you rate the difficulty in getting connected to the network for e-mail and WWW access?
   1-Easy 2-Somewhat easy 3-Somewhat difficult 4-Very difficult 5-Still not connected
2. Is your network connection through:
   1-CBX 2-Modem 3-Campus Ethernet 4-Off-campus Ethernet
3. If your network connection is through a modem or CBX:
   Have you experienced busy signals, dropped sessions, or time-outs when accessing mail or Web pages?
   1-Never 2-Only in the afternoons 3-Frequently 4-Still not connected

Help Desk Quality:

1. Have you called the 4HELP desk this semester? If so:
2. Did a consultant respond quickly?
   1-Yes 2-Fairly quickly 3-I had to wait a long time
3. Was the consultant able to resolve your problem?
   1-Yes 2-No
4. If not: Did the consultant offer any suggestions, or refer the problem to another consultant?
   1-Yes 2-No
5. Were the help desk staff courteous?
   1-Yes 2-No

User Services Outreach Effectiveness:

1. Were you aware of Virginia Tech’s computing resources before you came to campus?
2. Were you aware of any specific requirements your department has?
3. Have you used the Computing Center’s WWW pages (http://www.cc.vt.edu), which contain FAQs (frequently asked questions), software, reference materials, and the PIDtool?
4. How do you access DropAdd (online course registration)?
   1-from my home computer 2-from a friend’s computer 3-from a campus lab computer 4-through my department’s computer or terminal

Other Measures for Improvement Some of the other measures we are taking to make computer usage more satisfactory for our users:

1. We publish a list of recommended communications hardware, and make sure local computer vendors have a copy. Poor quality modems and network cards are responsible for most of our help desk calls during the first week of Fall classes.
2. We attend summer orientation for new students, so that they will have the information they need about computer hardware, software, and services.
3. We are training two resident computer consultants who will assist students living in their hall with computer problems. If this pilot program succeeds, it may be expanded to other residence halls.
4. We have assigned two full-time staff members to assess our support for faculty and research computing, and to respond to their specific needs.
5. We are setting up a distributed support model to provide special assistance and training for technical support professionals in the departments. They, in turn, will provide support for the faculty and staff in their departments.

Value

By looking at help desk statistics and user survey results, we hope to get a better assessment of our effectiveness in supporting the computing needs of our faculty, staff, and students. In a time of austere budgets, our support staff has actually gotten smaller as our support load has grown. We have made do by using student consultants and by providing online help wherever possible. This assessment should tell us whether we have been using our resources well, and if not, the steps we need to take for improvement.

---

Media Services/Educational Technologies

David Taylor

Narrative

Public universities have typically used large lecture classes as a response to increasing enrollments. However, large lecture classes have been blamed for a host of problems: alienation, cynicism, a focus on grades as the sole outcome of a course, and a passive attitude toward learning. Students are cut off from their professors as well as from each other. When classes cannot be made smaller, administrators and other decision-makers have turned to technology as a way to dissolve the distance between students and faculty. Technology has also been touted as a way to boost personal productivity by increasing access to course materials, and to enhance the quality of lecture presentations.

Students’ perception and attitude toward the way technology is used in their courses will doubtless determine their overall response to the course and, presumably, how much effort they put into a subject and how much they learn. Do they feel that various parts are useful? Do they think the technology contributes to their success? In what ways?Significant resources are required and expended to re-design a course, to convert class materials to online and to provide universal online access to these materials. Creating and maintaining a campus network infrastructure is also expensive especially when all students are given access to the network and its use is required in their classes. For these reasons, it is important to monitor how much and how well the online parts of a course are used. Do students use the materials in the way they were intended? Which parts are used the most? The least?

Questions to be answered

What are students perceptions of the usefulness of various technological innovations:

• multimedia presentations
• synchronous and asynchronous discussion sessions
• e-mail access to professor
• e-mail communication with other students
• electronic access to course materials

If the class uses a web site, how useful do students think various parts of the web site are?How frequently do students use the class web site?
Is there a relationship between the frequency with which students access the web site and their grade in the course?
Is there a relationship between student’s computer ownership and grade in the course?
Is there a relationship between student’s attitude toward computers and their success in the course?

Methodology

We will administer either two or three surveys during the two courses. As a baseline, the surveys will gather demographic data and identify computer ownership and the method by which students go online. The surveys will also administer attitude scales and a standardized “Computer Anxiety/Alienation scale” (Ray & Minch, 1990), which we have used for two years in a variety of classes at Virginia Tech. This and other scales will be administered at the beginning and the end of the course to determine changes during the course.We will also collect and analyze the extensive and detailed log file that is created by the class web server. We have adopted and developed software that can determine detailed, individualized patterns of usage of the web site. These patterns can then be linked with an individual’s grades and other performance indicators to determine if there is a relationship.

Performance Indicators

• Score on the Computer Anxiety/Alienation Scale
• Traffic on the course web site (from analysis of the web server log file)
• Score on several attitude scales developed and administered in a variety of courses at Virginia Tech over the past two years.

Value

Faculty and administrators need to know if student usage and practice in online courses matches the intentions and instructional objectives of faculty and instructional designers. The information we are gathering will help them monitor both student usage and students’ attitude toward the way that instructional technologies are being used.They should also know if online aspects unfairly discriminate against certain subgroups within the student population; for example, do gender, ethnicity, class standing, or ownership of a computer with network access affect instructional outcomes?Educational researchers know that motivation and attitude play a key role in learning; a related question is whether a student’s attitude toward computers plays a role in his or her performance in an online course. Equally important, does attitude toward computers change as a result of exposure and practice, i.e., as a result of having taken such a course? If “computerphobic” students tend to do worse in such a course, then possibly the technology is interfering with learning and steps should be taken to alleviate their stress or provide this group with an alternative method of instruction.Prior data at Virginia Tech show that taking a Cyberschool course tends to increase students’ confidence in computers and online learning and reduces computer anxiety and alienation. We would like to extend these findings by continuing this exploration across different types of courses and student populations.

Reference:

Ray, N.M. and Minch, R.P (1990). Computer anxiety and alienation: Toward a definitive and Parsimonious Measure. Human Factors, 32 (4), 477-491.