Contact Us | Members Only | Site Map

Association of Research Libraries (ARL®)

  Statistics & Assessment Contact:
Martha Kyrillidou
StatsQUAL®

New Measures

Share Share   Print

An investigative activity by the Statistics and Measurement and the Research Library Leadership and Management Committees that took place in 1999 and shaped a rich research agenda in the next decade.

Background

For several years, the ARL Statistics and Measurement Committee has discussed issues related to performance measures. Invited guests have presented information on data and techniques (e.g., electronic resource expenditures, SERVQUAL) and committee members have considered what additional data and activities ARL might undertake to address these issues. The April 1998 issue of ARL provides some of the background and activities surrounding this topic. In 1998, committee members suggested that time in a focused session could result in more substantive action by the Association to develop new measures. Committee members from the Research Library Leadership and Management Committee were also interested in participating in this activity, and a retreat for members of the two committees and other interested ARL member leaders was planned. January 1999 Retreat

At the facilitated retreat held in Tucson, Arizona, in January 1999, several members of the Statistics and Measurement Committee, the Leadership and Management Committee, and other interested member leaders, gathered to discuss what ARL can do to assist members in developing new measures that better describe research libraries and their services. Those attending the retreat addressed a set of questions regarding the data needed to describe research libraries in today's environment, the need for new measures, and the means by which useful data and measurement tools could be developed.

The retreat began with an overview of the issues of new measures and a review of past ARL programmatic activities to look at or develop new measures, setting the context for why the traditional measures are not enough. Several exercises were then conducted with participants identifying areas of interest, importance, and action. The many ideas generated were discussed, clarified, enhanced, or combined to create a set of eight categories in which further work could be conducted.

Action Plan

The retreat participants developed an action plan for Spring 1999 that included timelines for developing a set of papers that addressed the eight areas of interest. They also suggested holding events at the ARL May Membership Meeting to discuss the topics with the full membership to determine whether a continuing investigation of these ideas would be supported. Some of the topic areas may be more appropriate for the agendas of other ARL committees, some topics may be best explored with visiting program officers, or by ARL staff.

Development of Issue Papers

For each of the categories of investigation, one or two lead persons were identified who could prepare a short paper on the topic that would: a) include an outcomes based definition; b) identify the need for investigating the topic; c) provide some initial thoughts on needed data elements; d) identify what data is already collected; e) consider whether new measures should be a part of the metrics of ARL or addressed in another way (e.g., are the data private or public, should they be collected annually or episodically); f) provide a recommendation for additional research; g) note whether topic is long-term or short-term; h) recommend possible experts or provocative thinkers on the topic; and i) prepare a call for pilot institutions to investigate the topic.

White papers are provided below and readers are encouraged to provide comments on the topics, the substance of the papers, or the new measures effort in general.

Ease and Breadth of Access

This topic addresses tomorrow's library from the user perspective and includes measures for electronic resources and the effect of perpetual licenses. A range of topics will need to be considered and actions taken to understand how users search for information and what their success rates might be. Some of these topics may include: 1) examining electronic data on searches; 2) considering how a reduction of reference questions can be a measure of success; and 3) collecting data on the percent of need that can be met or resources delivered a) immediately, b) within one hour, c) within 6 hours, d) within 24 hours, e) within 48 hours, or f) forget it, we failed. Actions to obtain information about user behavior could include the use of focus groups and surveys.

"Ease and Breadth of Access," Shirley Baker [PDF]

Send comments to jblix@arl.org.

User Satisfaction

Can we validly measure the degree to which our "services" meet the needs and expectations of our clients and would a "standard" approach give us insight to best practice? Addressing the issues of number of satisfied users and the percentage of satisfactions will require answering many questions: 1) what do we mean by satisfaction? 2) can we define our potential user community and can we solicit input from that entire community or defined subsets? 3) can we define a standard set of "services" to study (e.g., service, collections, facilities), 4) what services are desired but not available? 5) do we want our studies to be voluntary and variable or standard and longitudinal? 6) who is doing this already? are there experts? 7) what instruments can be used to gather satisfaction data? surveys? focus groups? 8) what satisfaction levels are associated with selected services? 9) what services consistently show up as needing improvement? 10) with what intensity does the community utilize selected services? 11) what satisfaction levels would the community like to see (e.g., SERVQUAL)? 12) how do we move away from anecdotal information on satisfaction? 13) can we use standard practice on satisfaction to identify best practice? Data will need to be generated.

"User Satisfaction White Paper," Brinley Franklin & Danuta Nitecki [PDF]

Send comments to jblix@arl.org.

Library Impact on Teaching and Learning

This area covers the impact on learning and education, learning outcomes, involvement in the educational enterprise. It includes the entire learning environment, including such things as: contribution to diversity as a community value, activities that build community, diverse workforce, educational impact and effect, availability of bibliographic instruction, reference librarians, hours the library is open, access to other collections (reciprocal relationships), authentication of off-campus users, and reserve systems. It can address penetration into the co-creation of the curriculum, provision of opportunities for students to engage experientially with information, development of new knowledge and access tools, involvement in honors programs, electronic classrooms to enhance instructional sessions, involvement in distance education (the number of students that take advantage, seats filled), 7x24 user support, support for student cohorts, group study areas, library connectivity, bandwidth, outreach with academic committees, food and drink policies. Data can be gathered through market studies, user satisfaction surveys, etc. One measure could be contact hours. Data on students can be gathered to determine the library influence. Have they used the library, what difference did it make? Grades? Survey employers to identify successful new employees. Are students better able to solve information problems? Do graduates pursue life-long learning? apply skills? Is there a correlation of library information skills with graduate success, faculty productivity and teaching effectiveness? In the preparation of future scholars and teachers, does size of collection and ease of access result in effective faculty?

Library Impact on Research

What specific library activities have the most impact on the success of the research enterprise locally and globally? An assumption is that the library does indeed have a role. At the global level, measures could be obtained by evidence of use of library resources by non-primary clientele and their output (e.g., Title IIC). At the local level, measures could come from evidence of use by entry-level researchers, graduate students, and senior scholars.

"Libraries Impact on Research: A Preliminary Sketch," Carolynne Presser [PDF]

Send comments to jblix@arl.org.

Cost Effectiveness of Library Operations and Services

This category includes all services, including public services. One question to be addressed is to what degree do service offerings facilitate use to meet expressed and actual needs. Data questions include (1) was the need met, (2) was the need met efficiently? for the user? for the library? (3) was value added (e.g., increased self-sufficiency)? Data could come from cost and cost savings of individual services, transaction assessments, staff/user ratios. Measurement activities could include:

  1. one-time study with possible replication (ILL model)
    2.provide for local (weighted) definition of level of quality and annualized costs as well as unit costs for operations; may need to consider census or sampling methodologies
    3.for services, assume that highest quality is desired, no weighted definitions

Note: public services are also included in teaching and learning area

Space and Facilities

This area addresses how space can be best used to permit us to meet user needs. It makes an assumption that the library as place has value and that the library is a variety of formats. This area addresses infrastructure. Data can come from:

Collections -- capacity (how full), shifting frequency, storage facility, compact shelving/public access

Patrons -- workstations, reader stations, network ports, group study, service points

Building use -- gate counts, traffic patterns

Overall utilization -- percent space allocated to patrons, collections, staff, unassigned, restrooms, elevators, etc.

"The Measurement of Facilities Utilization in ARL Libraries," Cliff Haka, Joan Giesecke, and Jennifer Cargill [PDF]

Send comments to jblix@arl.org.

Market Penetration

What portion of the academic community uses the range of services, information resources, and facilities available? What percent of the target population is reached? What difference does the library make to the academic enterprise? Data answers will come from inventories of services, information resources, and facilities. Market segments will be drawn from such groups as faculty by discipline, student by class, alumni, etc. Methods might include a matrix that incorporates what? how many? and provides ratios of actual to potential use showing differences between and among institutions.

"Market Penetration in Research Libraries," Paul Kobulnicky & Carla Stoffle [PDF]

Send comments to jblix@arl.org.

Organizational Capacity

This area includes measures that indicate an investment in the capacity to change, a re-allocation of resources to future opportunities; opportunity costs. Organizational capacity assessment can include (a) performance measures for staff, (b) staff morale, (c) skills/abilities needs assessment based on outcomes we want to achieve, (d) salary information based on competencies we need, (e) competency based job descriptions and salaries that support outcomes, (f) risk-taking, innovation, (g) individual investment in research and development, learning, (h) alignment of human resource support systems to leverage investments.

"Organizational Capacity White Paper," Kathryn J. Deiss [PDF]

Send comments to jblix@arl.org.