Eric O’Hanlon
Head of Development, Infrastructure, and Applications
Columbia University
Benjamin Zweig
Digital Projects Manager
Columbia University
Deseree Stukes
Software Quality Assurance Specialist
University of North Carolina at Chapel Hill
Benjamin Pennell
Applications Analyst
University of North Carolina at Chapel Hill
Jason Casden
Head, Software Development
University of North Carolina at Chapel Hill
Exploring AI Self-Assessment Capabilities for Metadata and Image Description Creation at Columbia University Libraries
Columbia University Libraries (CUL) is currently initiating a year-long exploration and critical evaluation using artificial intelligence (AI) to enhance description, discovery, and transcription of its collections. As part of this multi-pronged effort, CUL is specifically testing how AI might enhance cataloging processes by generating bibliographic metadata and image descriptions for open access architectural journals using the subject classifications from the Avery Index authorities. Central to this effort is a process CUL is testing that involves developing an AI self-assessment mechanism that grades the quality of its own output based upon well-defined cataloging practices, and which can be checked both by human reviewers and inform an automated review process executed by secondary models or agents. CUL is further evaluating this work with regard to the cost and quality of open-weight and closed-weight models, utilizing Amazon Web Services solutions like Bedrock and SageMaker to run these models. This work is one of several CUL efforts to build AI solutions that improve the processing of collections that will enhance user discovery and research use.
Scaling Accessible Image Description: Combining LLMs and Risk-Based Human Review
This presentation examines University of North Carolina at Chapel Hill (UNC) Libraries’ use of artificial intelligence (AI) to generate accessible image descriptions at scale while maintaining accuracy, context, and responsible description practices. A framework informed by Web Content Accessibility Guidelines (WCAG) 2.2, cultural heritage description practices, and UNC Libraries’ Conscious Editing Guidelines was developed to evaluate several large language models (LLMs) for description quality, factual accuracy, bias, cost, transcription quality, and scalability. That work has evolved into a hybrid workflow using ChatGPT for full-text and alt-text image descriptions and Gemini for more accurate transcription, coupled with structured AI-generated risk assessments and derived risk scores to help identify descriptions that need human review. In a pilot with more than 700 images selected for their range and difficulty, the research team’s review found that an LLM can reliably assess risk using criteria they defined, and they are increasingly confident in using the LLM’s risk scores to focus human review where it is most needed. The presenters will report how different score thresholds perform, including what each could miss, and how the team is weighing that against the review effort required for a collection of more than 2.5 million digital images. The workflow also aims to reduce environmental impact through selective model use and targeted processing. This presentation demonstrates how AI-assisted description, risk-based human review, and strategic model use can responsibly make digital collections more accessible and discoverable at scale.
