Friday, November 26, 2010

TREC 2010 Roundup

Back from another successful TREC conference on the NIST campus. 2010 is a transition year, with the end of old tracks and the proposition of new ones. Indeed, TREC is moving with the times, looking at new data sources and test collections, as well as new evaluation strategies.

Outwith the old . . .

For example, TREC 2010 marks the end of the Relevance Feedback and Blog tracks. While TREC 2010 will be the last year of the Relevance Feedback track, the Blog track, which has been running for the last 5 years, is now morphing into a new Microblog track, investigating real-time and social search tasks in Twitter. A brand new test collection possibly containing 2 months of tweets is planned, with linked web-pages and a partial follower graph. Join the Microblog track googlegroup to obtain the latest updates and follow the Microblog track on Twitter.

TREC 2011 will also witness the initiation of the new Medical Records track, dedicated to investigating approaches to access free-text fields of electronic medical records.

On the test collection front, the Web track is also forward planning a new large-scale dataset to replace ClueWeb09. Indications are that this new dataset will be about the same scale as ClueWeb09 but might provide more temporal information (multiple versions of a page or site over time). Moreover, we have suggested that this might be the heart of a larger dataset comprised of multiple parallel/aligned corpora, for example blogs and news feeds covering the same timeframe.

TREC Assessors, Relevant?

In terms of evaluation, 2010 marks the first year where evaluation judgments were crowdsourced using an online worker marketplace, as opposed to relying on TREC assessors, the participants themselves, or a select group of experts. Indeed, both the Blog track and the Relevance Feedback track crowdsourced some of their evaluation (although the Relevance Feedback track suffered many setbacks and its crowdsourcing process is still incomplete). Furthermore, to investigate the challenges in this new field of crowdsourcing, a specific Crowdsourcing track has been created and will run in 2011. More details can be found here.


As usual, themes emerged within the various tracks. Learned approaches were far more prevalent this year, now that training data was available for the ClueWeb09 dataset. Indeed, the Web track was dominated by trained models mostly based on link and proximity search features. Diversification, on the other hand, remains a challenging task, with the top groups leaving their initial rankings as is. An outstanding exception is our own approach using the xQuAD framework under a selective diversification regime, which further improves our strongly performing adhoc baseline. Craig Macdonald presented our work in the Web track plenary session.

In the Blog track, voting model-based and language modeling approaches proved popular for blog distillation. For faceted blog ranking, participants employed variants of facet dictionaries to either train a classifier or as features for learning. For the top news task, participants deployed a wide variety of methods to rank news stories in a real-time setting, from probabilistic modeling to blog post voting with historical evidence. Richard Mccreadie presented our work on the blog track as a poster during TREC 2010, which attracted very interesting discussions.

During the TREC conference, Iadh Ounis, Richard Mccreadie and others have done a fair amount of tweeting. You can follow some bits of the TREC conference through the #trec2010 hashtag.

Wednesday, November 3, 2010

CIKM 2010 in Toronto, ON, Canada

I'm back from Toronto, where a few of us attended the CIKM 2010 conference last week. On Friday, I presented our paper on "Selectively diversifying Web search results", a joint work with Craig Macdonald and Iadh Ounis. This work extends our successful participation in the diversity task of the TREC 2009 Web track, by investigating the need for search result diversification in the first place. In particular, we proposed a novel supervised learning approach to predict not only whether promoting diversity is beneficial, but also how much diversification should be applied to attain an effective retrieval performance on a per-query basis. After thorough, large-scale experiments with over 900 query features, we found that our selective approach can substantially improve existing diversification approaches, including our state-of-the-art xQuAD framework. Nonetheless, we believe the significance of our contribution goes beyond these successful results. Indeed, it was with great pleasure that we heard from the NTCIR organisers that NTCIR-9 will run an Intent task, aimed---among other things---at selectively diversifying search results, an area where we are proud to be pioneers.
Besides our own paper, a few other papers caught my attention:
  • Web Search Solved? All Result Rankings the Same? by Hugo Zaragoza, B. Barla Cambazoglu and Ricardo Baeza-Yates
  • Reverted Indexing for Feedback and Expansion, by Jeremy Pickens, Matthew Cooper and Gene Golovchinsky
  • Rank Learning for Factoid Question Answering with Linguistic and Semantic Constraints, by Matthew Bilotti, Jonathan Elsas, Jaime Carbonell and Eric Nyberg
  • Organizing Query Completions for Web Search, by Alpa Jain and Gilad Mishne
  • Clickthrough-Based Translation Models for Web Search: from Word Models to Phrase Models, by Jianfeng Gao, Xiaodong He and Jian-Yun Nie
The conference also featured great keynotes, of which those by Jamie Callan and Susan Dumais deserve a particular mention. Jamie talked about his view for the future of search, in which search engines capable of fully leveraging the structure of queries and documents would enable more sophisticated applications built on top of them. Susan addressed the temporal evolution of Web content, how it impacts the way users access this content, and how test collections should account for it. For more details, have a look at the excellent posts by Gene Golovchinsky on Jamie and Susan's talks.
Last but not least, many of us were involved in promoting the next edition of CIKM, to be held here in Glasgow. There was a lot of excitement from the several people that visited our booth, and also during the hand-over talk at the end of the conference. Well done Jon, Mary, Craig, and Iadh for the hard work! The arrangements for CIKM 2011 are well advanced, and the call for papers is now online. You can also follow the latest news about CIKM 2011 on Twitter, Facebook, LinkedIn, and Lanyrd. We look forward to welcoming you all to Glasgow next year!