ALA Redux - Ready Reference in Chat - Not Dead Yet! PLUS 1 Foolproof Way to Increase Accuracy in Chat Reference

"I'm not dead yet!"

I wanted to update the LG faithful on the 5 (5!) very different presentations I gave at ALA in Washington DC, based on the research we are doing on the Seeking Synchronicity IMLS grant studying live chat virtual reference at Rutgers University, SCILS and OCLC.

It was wonderful to speak to so many people at ALA who were eager (yes, actually eager) to hear about research in Virtual Reference! Feedback was amazingly positive and I plan to blog about each of these presentations, just to give some highlights, and to point you to the PowerPoint presentation and handouts if you want to get more information.

For the 13th RUSA New Reference Research Forum at ALA, Lynn Silipigni Connaway of OCLC and I presented “Not Dead Yet! Ready Reference in Live Chat Reference.” We had pulled a random sample from almost 500,000 QuestionPoint transcripts from 2004-2006 and done an analysis of the types of questions and an evaluation of accuracy in ready reference. We found that early reports of the death of ready reference (due to user’s ability to search Google and Wikipedia independently for factual type questions) are premature and that these types of questions surprisingly still comprise nearly 30% of chat questions. Here are some of our findings as a tease to invite you to go to the Seeking Synchronicity site for more detail:

What Types of Questions are Being Asked in Chat?

Of 915 total chat reference questions:

• 243 (27%) were Ready Reference (short, factual, questions)
  
• 293 (32%) were Subject Searches (largest group)
  
• 10 (1%) were Inappropriate (comforting in such a low number)

What Types of Libraries/Consortia are Getting more Ready Reference Questions?

Of 162 chat transcripts:

• 59% of the Ready Reference questions came from Public Libraries
  
• 45% from Public & Academic Consortia
  
• 35% from Adacemic and Law Libraries

How about Accuracy?

How accurate are chat librarians in answering ready reference questions? We used the Arnold & Kaske (2005) method to classify chat ready reference answers and found 78% (141 of 180) to be correct. 69% (125 of 180) meeting the gold standard of correct with citation.

Increase Accuracy! Our Most Important Finding!

For the 20% (35 of 180) incorrect questions, the most frequent reason was that the chat librarian sent only general information (usually a website on the topic). The general website did not contain the specific information requested.

So the “One Foolproof Way" to increase accuracy for ready reference chat questions is as follows: before you push a general information page, make sure it has the specific, exact answer to the user’s question.

Clidk here for the Not Dead Yet! PowerPoint presentation and here for the Not Dead Yet! handout that details coding procedures for type of question and accuracy.

Cited Reference: Arnold, J. & Kaske, N. (2005). Evaluating the quality of a chat cervice, portal: Libraries and the Academy, 5(2), 177-193.

Special thanks to Janet Torsney, an MLIS student at Rutgers University, SCILS, for her help in doing the accuracy analysis and Susanne Sabolcsi-Boros for intercoder check. Also thanks to Nick Belkin of SCILS for suggesting the accuracy analysis.