Assessing the Academic Networked Environment

BROWN UNIVERSITY CIS TRAINING

Assessing the Academic Networked Environment

Training Assessment

ASSESSING THE ACADEMIC NETWORKED ENVIRONMENT





Brown University CIS Training

115 Waterman Street Box 1885

Providence RI 02912-1885

Phone 401-863-7299 • Fax 401-863-7329





Table of Contents

IN CLASS EVALUATION	1

EXTENSIVENESS	1

EFFICIENCY	1

EFFECTIVENESS	2

SERVICE QUALITY	2

IMPACT	2

USEFULNESS	3

INSTRUCTOR EVALUATION	4

EXTENSIVENESS	4

EFFICIENCY	4

EFFECTIVENESS	5

SERVICE QUALITY	5

IMPACT	5

USEFULNESS	5

POST COURSE EVALUATION	6

EXTENSIVENESS	6

EFFICIENCY	6

EFFECTIVENESS	7

SERVICE QUALITY-	7

IMPACT	7

USEFULNESS	7

SIGN-IN SHEETS	8

EXTENSIVENESS	8

EFFICIENCY	8

EFFECTIVENESS	8

SERVICE QUALITY	9

IMPACT	9

USEFULNESS	9

COURSE REGISTRATION	10

EXTENSIVENESS	10

EFFICIENCY	10

EFFECTIVENESS	11

SERVICE QUALITY	11

IMPACT	11

USEFULNESS	11



Chapter

1



In-class Evaluations

In-class Evaluations provide real time feedback on topic value and instructor style.

T

ICON KEY

* Extensiveness

$ Efficiency

* Effectiveness

* Service Quality

* Impact

* Usefulness

oo often, people doing computer training have extreme opinions of in-class evaluation sheets.

 Some trainers consider them nearly worthless, often referring to them as "Smiley Sheets",

feeling that they only serve to measure how personable the instructor was.

 Others trainers err on the side of putting too much value on the in-class evaluations,

 using them as the sole, or major assessment tool.

Either attitude, carried to the extreme,

loses sight of the value and the limitations of an in-class evaluation.



ü

Extensiveness:   D+

The in-class evaluations are poor measurements of extensiveness

of the training offerings. These evaluations are optional,

 and usually unsigned. Fewer than 20% of the attendees

normally fill out the critiques, but use varies depending

 on how much the instructor mentions the form.





$

Efficiency:   F-

In class critiques have no value in calculating the efficiency

of the training program.







:

Effectiveness:  B+

In-class critiques are often used to try to measure

 the effectiveness of a training program.

Users attending training classes (especially

undergraduate students) are not shy about expressing

 their displeasure if a class fails to meet their expectations.

 This can happen if a course description is poorly worded or unclear,

 attendees arrive expecting training in one area and receive

training in a different form on or a different topic.

An instructor with poor instructor skills will also be

flagged in the in-course critique, subject mater

expertise is important, but you have to be able to explain the material to new users.



:)

Service Quality:  A+

Service Quality is the area where in-class critiques shine.

Instructor talent is the area where this is most often commented on.

 Positive comments are usually filled out in class and left for the

instructor to read, i.e. "Best computer class I have ever taken!".

 Attendees less happy with the class will often take the critique

sheet with them when they leave class. The form can be completed

and dropped into the campus mail system. This saves the user the

cost of a stamp, and increases the likelihood of the critique being returned.

 Users may comment on the usefulness of the material,

 but often it is too early to tell.

They will comment about how wonderful the application sounds

(especially after a demo only class), but they want to go back

 and try it at their own computer before they could state how

 much they have learned. Comments about facilities (room temperature),

 registration, and confirmation would be included in this category as well.



M

Impact:  C+

The ability to measure the impact of training varies a great deal

between demo classes and hands-on classes.

 The demos expose users to new software, upgraded applications,

 and tips and tricks of using the software. A demo attendee sees

 and hears things that sound useful, but it is often difficult to measure

the impact until they have had a chance to try the material.

 Hands-on training classes provide in-class opportunities to apply

the material covered, so in-class critiques can and do include

some indication of the real world usefulness (or the lack there of)

of the material covered.



&

Usefulness:  B-

Without a period of actually applying the material,

 users often are unclear as to how useful the material

will be to them outside of the classroom.

 Negative comments on the in-class critique form

will most likely be accurate, the user will normally

know when the material has no value to them.

Positive, or undecided users need time outside of class

to try and apply the material in the real world.





Chapter

2

Instructor Evaluation

Instructor Evaluations provide real time feedback

on the instructor's opinion of topic value

and various course support issues.

T

ICON KEY

* Extensiveness

$ Efficiency

* Effectiveness

* Service Quality

* Impact

* Usefulness

he instructor for a class can provide valuable,

first hand information on course content,

unexpected questions regarding content,

difficulties experienced by the attendees

with the registration process, and a host of other issues.

 These are NOT critiques of the instructor, rather,

they are the instructor's opinions of how the class went

 and how it could be improved.



ü

Extensiveness:   B

The instructor evaluations provide an opportunity

for the instructor to comment on who he or she has observed

attending the training session. Demos, in particular,

 have no pre-registration requirement, so often

the instructor's notes as to who attended are

very helpful indications of the extensiveness

of the training offerings.



$

Efficiency:   F-

Instructor critiques have only a limited value in calculating

the efficiency of the training program. Often times,

 a large percentage of the people attending the training

may arrive late or leave early, and often neglect to sign

 the Attendance Sheet.  The instructor's critique form can

 indicate an estimate of the number or percentage of

 the attendees that did not sign in.







:

Effectiveness:  B+

Instructor critiques can indicated where material in

the lesson plan was ineffective with a particular audience,

 or subset of the classroom population. The examples used

 in class often can be tailored to the needs of the attendees,

and the wrong examples can hurt the class effectiveness.

The instructor, noting this on his or her critique,

can give the curriculum developed for the course valuable feedback.



:)

Service Quality:  B-

Service Quality is harder to measure with instructor critiques.

Instructors may not always know how well a class went,

 but they usually know when a class goes poorly.

Since the instructor critiques may be seen by

the instructor's supervisor, and feedback is provided

on a semestral basis from the Training Coordinator

to all instructor supervisors, most instructors may

be reluctant to "place themselves on report".



M

Impact:  D+

The instructor can rarely measure the impact of

the material being taught. If the instructor has

a class with limited attendance, it is possible

to survey the attendees to see how they hope to use

the material, but the critique is not designed

to collect this information. This form of feedback

is almost always passed by word of mouth to the Training Group.



&

Usefulness:  B-

Only in cases where the instructor is given verbal feedback

 by someone unwilling to fill out an in-class critique

would the instructor have second hand information

on the usefulness of the material.





Chapter

3

Post Course Evaluation

Post-course evaluations provide real world feedback

on the usefulness of the course content.

T

ICON KEY

* Extensiveness

$ Efficiency

* Effectiveness

* Service Quality

* Impact

* Usefulness

he newest form of evaluation used by CIS Training,

 the post-course evaluation is a form sent out via electronic mail

two to four weeks after a hands-on training class to measure

 the trainees' evaluation of the course after he or she

 has had an opportunity to apply the material learned.

Limitations include the fact that not all attendees use email,

 and not everyone will bother to reply to the survey.



ü

Extensiveness:   F

The post-course evaluations depend on data collected

during course registration. It adds no new information

as to the extent of users of the training services



$

Efficiency:   F+

Post-course critiques have only a limited value

in calculating the efficiency of the training program.

Post-course evaluations can show when people are

 attending training that they are not using in their

jobs (staff learning how to design web pages that

don't need those skills for their jobs).







:

Effectiveness:  A+

Post-course critiques can indicate whether

 the material being taught is in fact of value

to people back at their own computers.

By looking at the Usefulness to the different audiences

(staff, faculty, undergraduate students and graduate students),

 CIS Training can determine how effective

 the training programs are in meeting the

needs of the University.



:)

Service Quality:  B-

Service Quality is can also be measured

 with post-course critiques. Surveying for Service Quality

 is not the primary goal of the post-course survey form,

and has intentionally been minimized on the form.

 However, the difference in the design format makes

 some people feel that they are personally

 being asked their opinion, so comments

sometimes are received from attendees that did

not choose to fill out a in-class critique.



M

Impact:  A+

The post-course survey was created to measure

impact and usefulness. Training classes are attended,

knowledge leaves the room, but what benefit is that

 knowledge to the people attending training?

This is what the post-course survey is designed to measure.

 One area that is not presently covered that we would

like to add is a supervisor's section of the post-course

 survey to collect the supervisor's opinion of

the training received. This might introduced slowly,

 using it only for Departmental Training classes at first.



&

Usefulness:  A+

Post-course surveys measure usefulness. "Nuff said.



Chapter

4

Sign-in Sheets

Sign-ins provide information on who attends the training classes.

T

ICON KEY

* Extensiveness

$ Efficiency

* Effectiveness

* Service Quality

* Impact

* Usefulness

he sign-in sheet asks for the attendees name, department (as applicable),

 status: staff, faculty, undergraduate or graduate.

It provides a count of the numbers of people attending class,

 and a rough breakout as to which of the major audiences they are from.







ü

Extensiveness:   B+

The sign-in sheets collect data on who is attending classes,

 and which community they are from. This is particularly useful for demos,

 which require no pre-registration.



$

Efficiency:   C-

Sign-ins have limited value in calculating the efficiency

of the training program. They provide us a count of

the number of people attending classes.



:

Effectiveness:  F

Sign-ins provide no information on effectiveness.

:)

Service Quality:  F

Sign-ins provide no information on the quality of the service provided.



M

Impact:  F

Sign-ins provide no information on the impact of the training provided.



&

Usefulness:  F

Sign-ins provide no information on the usefulness of the training classes.



Chapter

5

Course Registration

Course Registration provides information on who wishes

to attend the training classes, and why.

T

ICON KEY

* Extensiveness

$ Efficiency

* Effectiveness

* Service Quality

* Impact

* Usefulness

he registration process is used only for the hands-on training

classes, but collects a lot of valuable information.

We learn which classes are most popular, which portion

of the campus community is interested in the training

(the status, as we call it), the time of day, day of week,

and time of year that is most popular, the platform

(operating system) of choice, and the version(s)

of software being used.







ü

Extensiveness:   A+

The registration information collected provides data

 on who is attending classes, and which community they are from.



$

Efficiency:   B+

We try to schedule enough classes to meet the demand,

without having too many. The registration process

allows us to see where the demand is, and to schedule

 overflow classes as needed..



:

Effectiveness:  B

The speed that classes fill up is a fairly good indicator

 of which classes are popular, but doesn't tell us what

 we should be offering that we are not, or how people

 are using the training after class..

:)

Service Quality:  C+

Registration allows us to track who is taking training,

either at the individual level (repeat customers) or the

status level (how many grad students took the Windows 3.1 modem class).



M

Impact:  C-

The registration process is not designed to collect

 this type of information, but it is noted when it

is mentioned, and passed onto the Training Coordinator..



&

Usefulness:  C+

The registration process guides the users to the right

class for their needs, so it helps improve the usefulness

 more than it measures it..





3





3























BROWN UNIVERSITY   CIS TRAINING



1	12/16/97



BROWN UNIVERSITY   CIS TRAINING









3





4





11









12

---