TREC 2010 Roundup

Back from another successful TREC conference on the NIST campus. 2010 is a transition year, with the end of old tracks and the proposition of new ones. Indeed, TREC is moving with the times, looking at new data sources and test collections, as well as new evaluation strategies.

Outwith the old . . .

For example, TREC 2010 marks the end of the Relevance Feedback and Blog tracks. While TREC 2010 will be the last year of the Relevance Feedback track, the Blog track, which has been running for the last 5 years, is now morphing into a new Microblog track, investigating real-time and social search tasks in Twitter. A brand new test collection possibly containing 2 months of tweets is planned, with linked web-pages and a partial follower graph. Join the Microblog track googlegroup to obtain the latest updates and follow the Microblog track on Twitter.

TREC 2011 will also witness the initiation of the new Medical Records track, dedicated to investigating approaches to access free-text fields of electronic medical records.

On the test collection front, the Web track is also forward planning a new large-scale dataset to replace ClueWeb09. Indications are that this new dataset will be about the same scale as ClueWeb09 but might provide more temporal information (multiple versions of a page or site over time). Moreover, we have suggested that this might be the heart of a larger dataset comprised of multiple parallel/aligned corpora, for example blogs and news feeds covering the same timeframe.

TREC Assessors, Relevant?

In terms of evaluation, 2010 marks the first year where evaluation judgments were crowdsourced using an online worker marketplace, as opposed to relying on TREC assessors, the participants themselves, or a select group of experts. Indeed, both the Blog track and the Relevance Feedback track crowdsourced some of their evaluation (although the Relevance Feedback track suffered many setbacks and its crowdsourcing process is still incomplete). Furthermore, to investigate the challenges in this new field of crowdsourcing, a specific Crowdsourcing track has been created and will run in 2011. More details can be found here.

Themes

As usual, themes emerged within the various tracks. Learned approaches were far more prevalent this year, now that training data was available for the ClueWeb09 dataset. Indeed, the Web track was dominated by trained models mostly based on link and proximity search features. Diversification, on the other hand, remains a challenging task, with the top groups leaving their initial rankings as is. An outstanding exception is our own approach using the xQuAD framework under a selective diversification regime, which further improves our strongly performing adhoc baseline. Craig Macdonald presented our work in the Web track plenary session.

In the Blog track, voting model-based and language modeling approaches proved popular for blog distillation. For faceted blog ranking, participants employed variants of facet dictionaries to either train a classifier or as features for learning. For the top news task, participants deployed a wide variety of methods to rank news stories in a real-time setting, from probabilistic modeling to blog post voting with historical evidence. Richard Mccreadie presented our work on the blog track as a poster during TREC 2010, which attracted very interesting discussions.

During the TREC conference, Iadh Ounis, Richard Mccreadie and others have done a fair amount of tweeting. You can follow some bits of the TREC conference through the #trec2010 hashtag.

CIKM 2010 in Toronto, ON, Canada

I'm back from Toronto, where a few of us attended the CIKM 2010 conference last week. On Friday, I presented our paper on "Selectively diversifying Web search results", a joint work with Craig Macdonald and Iadh Ounis. This work extends our successful participation in the diversity task of the TREC 2009 Web track, by investigating the need for search result diversification in the first place. In particular, we proposed a novel supervised learning approach to predict not only whether promoting diversity is beneficial, but also how much diversification should be applied to attain an effective retrieval performance on a per-query basis. After thorough, large-scale experiments with over 900 query features, we found that our selective approach can substantially improve existing diversification approaches, including our state-of-the-art xQuAD framework. Nonetheless, we believe the significance of our contribution goes beyond these successful results. Indeed, it was with great pleasure that we heard from the NTCIR organisers that NTCIR-9 will run an Intent task, aimed---among other things---at selectively diversifying search results, an area where we are proud to be pioneers.

Besides our own paper, a few other papers caught my attention:

• Web Search Solved? All Result Rankings the Same? by Hugo Zaragoza, B. Barla Cambazoglu and Ricardo Baeza-Yates
• Reverted Indexing for Feedback and Expansion, by Jeremy Pickens, Matthew Cooper and Gene Golovchinsky
• Rank Learning for Factoid Question Answering with Linguistic and Semantic Constraints, by Matthew Bilotti, Jonathan Elsas, Jaime Carbonell and Eric Nyberg
• Organizing Query Completions for Web Search, by Alpa Jain and Gilad Mishne
• Clickthrough-Based Translation Models for Web Search: from Word Models to Phrase Models, by Jianfeng Gao, Xiaodong He and Jian-Yun Nie

The conference also featured great keynotes, of which those by Jamie Callan and Susan Dumais deserve a particular mention. Jamie talked about his view for the future of search, in which search engines capable of fully leveraging the structure of queries and documents would enable more sophisticated applications built on top of them. Susan addressed the temporal evolution of Web content, how it impacts the way users access this content, and how test collections should account for it. For more details, have a look at the excellent posts by Gene Golovchinsky on Jamie and Susan's talks.

Last but not least, many of us were involved in promoting the next edition of CIKM, to be held here in Glasgow. There was a lot of excitement from the several people that visited our booth, and also during the hand-over talk at the end of the conference. Well done Jon, Mary, Craig, and Iadh for the hard work! The arrangements for CIKM 2011 are well advanced, and the call for papers is now online. You can also follow the latest news about CIKM 2011 on Twitter, Facebook, LinkedIn, and Lanyrd. We look forward to welcoming you all to Glasgow next year!

Terrier Team at SIGIR 2010 in Geneva

SIGIR 2010 has just started in Geneva. From the TerrierTeam, Richard and myself are attending.

On Monday, Richard presented his PhD topic, Leveraging User-generated Content for News Search at the doctoral consortium.

Later, at the Web Ngram workshop, I'll be presenting a paper on Global Statistics in Proximity Weighting Models.

About the same time, Richard will be presenting at the Crowdsourcing for Search Evaluation workshop. His paper on Crowdsourcing a News Query Classification Dataset examines the effectiveness of different interfaces for having Mechanical Turkers classify queries as news-related or not.

Last but not least, and continuing on our proximity theme, Nicola Tonellotto from CNR is presenting our joint work titled Efficient Dynamic Pruning with Proximity Support at the Large Scale & Distributed Systems workshop.

Meanwhile, please say hello if you see us at the conference, or stay up to date by following #sigir2010. And remember, if you are near the registration desk, please pick up flyers for Terrier and CIKM 2011.

Top Authors in Information Retrieval

Thanks to Sérgio Nunes who alerted us to this ranking by Microsoft Academic Search of the Top Authors in Information Retrieval, in the past 5 years.

According to this recent ranking, two members of the TerrierTeam, namely Iadh Ounis and Craig Macdonald, are in the top 5 authors in Information Retrieval in the past 5 years (position #1 and #4, respectively). The ranking is based on in-domain citations.

This good news comes just at the start of the SIGIR 2010 Conference, which will be held in Geneva, Switzerland this week (19-23 July 2010). Several members of the team will be in attendance.

WWW 2010 in Raleigh, NC, USA

I am back from the sunny Raleigh, NC, USA. Besides the nice weather, I had a great time last week attending the 19th International World Wide Web Conference (WWW 2010), where I presented our paper on Exploiting query reformulations for Web search result diversification, a joint work with Craig Macdonald and Iadh Ounis. The paper introduces a probabilistic formulation of our xQuAD framework for search result diversification, and analyses the effectiveness of query reformulations provided by three commercial search engines for the diversification task. My talk was very well received, with lots of questions from the audience, and subsequent chatting with many people from both academia and industry.

The blend academia-industry was indeed a signature of WWW. I was also impressed with the multidisciplinary nature of the conference—with up to five parallel sessions, there was always something for everyone! In particular, from the sessions I attended, a few papers caught my attention:

• Clustering query refinements by user intent, by Eldar Sadikov et al. (Stanford University and Google)
• Optimal rare query suggestion with implicit user feedback, by Yang Song and Li-wei He (Microsoft Research)
• Building taxonomy of Web search intents for name entity queries, by Xiaoxin Yin and Sarthak Shah (Microsoft Research)
• Exploring Web scale language models for search query processing, by Jian Huang et al. (Microsoft Research Asia, Facebook, and Penn State University)
• Classification-enhanced ranking, by Paul N. Bennett et al. (Microsoft Research)
• Ranking specialization for Web search: A divide-and-conquer approach by using topical RankSVM, by Jiang Bian et al. (Georgia Tech and Yahoo! Labs)
• Generalized distances between rankings, by Ravi Kumar and Sergei Vassilvitskii (Yahoo! Research)
• Relational duality: Unsupervised extraction of semantic relations between entities on the Web, by Danushka T. Bollegala et al. (University of Tokyo)

The conference also featured three passionate keynotes:

• Vint Cerf discussed a broad range of topics of interest on today's Web, where everything is connected: 1.8 billion users, around a billion Web-enabled mobile devices, and still a large room for growth in developing countries. Touched points included the implications of the explosion of data production on mobility, accessibility, security and privacy, intellectual property, digital preservation, as well as new technologies (e.g., cloud computing).
• dannah boyd discussed privacy implications of the availability of "big data". Her keynote revolved around common misconceptions associated with the analysis of data produced by online social activities, as well as ethical concerns related to using this data in the first place, "just because it is accessible".
• Carl Malamud from public.resource.org described his experiences trying to convince seven bureaucratic institutions to make public data publicly accessible. His keynote was organised around "10 rules for radicals", a guide on how to break the barriers towards negotiating with bureaucrats.

On Thursday night, the conference banquet featured an exciting performance by the North Carolina string band Carolina Chocolate Drops. Check out Snowden's Jig (Genuine Negro Jig) and Don't get trouble in your mind for a taste.

Friday held the closing ceremony, with the announcement of the award winners.
Best Paper:

• Factorizing personalized Markov chains for next-basket recommendation, by Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme (Osaka University and University of Hildesheim)

Best Student Paper:

• Privacy wizards for social networking sites, by Lujun Fang and Kristen LeFevre (University of Michigan)

Best Posters:

• How much is your personal recommendation worth, by Paul Dütting, Monika Henzinger and Ingmar Weber (EPFL Lausanne, University of Vienna, and Yahoo! Research)
• SourceRank: Relevance and trust assessment for deep Web sources based on inter-source agreement, by Raju Balakrishnan and Subbarao Kambhampati (Arizona State University)

The closing ceremony also featured a short presentation of WWW 2011, to be held in Hyderabad, India. WWW 2012 will take place in Lyon, France.

Finally, on Saturday, the IW3C2 announced the Brazilian bid as the winner to host WWW 2013, which I was very glad to hear about!

RIAO 2010 in Paris, France.

The 9th International RIAO Conference has started in Paris, France (28-30 April, 2010). It is unfortunate that it is being held concurrently with WWW 2010 in Raleigh.

The first RIAO conference was held in Grenoble in 1985. RIAO is currently a triennial conference, addressing Information Retrieval research topics of interest to both Academia and Industry. This year, the conference focuses on Adaptivity, Personalization and Fusion of Heterogeneous Information.

The following papers have caught my eyes, while browsing the RIAO 2010 program:

• Boiling down information retrieval test collections. T. Sakai et al. (Microsoft Research Asia, CMU)
• Improving tag recommendation using social networks. A. Rae et al. (The Open University, Yahoo! Research Barcelona).
• Analysis of robustness in trust-based recommender systems. Z. Cheng and N. Hurley (UCD)
• Opinion-finding in blogs: A passage-based language modelling approach. M. Saad Missen et al (IRIT)
• Predicting query performance using query, result, and user interaction features. Q. Guo et al. (Emory University/Microsoft Research)
• Towards a collection-based results diversification. J.A. Akinyemi et al. (University of Waterloo)

In addition, the TerrierTeam has two full papers, which are being presented today at the conference (hopefully, the slides will follow shortly):

• Voting for Related Entities by R.L.T. Santos, C. Macdonald and I. Ounis. The paper addresses the problem of entity search, where the goal is to rank not documents, but entities in response to a given query. The paper proposes to tackle this problem as a voting process, by considering the occurrence of an entity among the top ranked documents for a given query as a vote for the existence of a relationship between this and the entity in the query. The approach led to high precision and unparalleled recall compared to TREC 2009 systems.
• News Article Ranking: Leveraging the Wisdom of Bloggers by R. McCreadie, C.Macdonald and I. Ounis. The paper investigates how news article ranking can be performed automatically, so as to assist editors in selecting the articles, which should make the front page of their newspaper. In particular, the paper investigates the blogosphere as a prime source of evidence, on the intuition that bloggers, and by extension their blog posts, can indicate interest in one news article or another. The paper proposes to model the automatic news article ranking task as a voting process, where each relevant blog post acts as a vote for one or more news articles. The approach led to the best TREC 2009 retrieval performance in the Blog track.

Craig Macdonald is tweeting the conference, pending an appropriate wireless signal. You can follow some bits of the RIAO conference through the #riao2010 hashtag.

ECIR 2010 in Milton Keynes: A Report

Last week, five of us attended the ECIR 2010 conference in Milton Keynes. The conference was fairly well-organised, although it markedly lacked the lustre of the previous three editions of the conference. In terms of attendance, only about 170 delegates have registered, much less than Glasgow 2008 (210+), and Toulouse 2009 (180+). Perhaps, the exotic town of Milton Keynes was not deemed to be a very attractive venue for a conference. In fact, apart from attending the conference, there was not much else to do -- e.g. the nearest proper pub was at about 2 miles from the conference venue.

The ECIR 2010 conference has suffered from a new and previously unseen problem: several authors and presenters did not make it to the conference, preferring to give their presentation by proxy or using a pre-recorded talk. No less than 5 no-shows were recorded during the conference. Even the keynote speaker and winner of the first BCS IRSG Karen Sparck Jones award, Mirella Lapata, did not show up and gave her presentation through a pre-recorded video. While Lapata certainly had a valid reason (as probably did the other speakers) not to show up, it is clear that ECIR should concretely deal with such a problem, e.g., by making it compulsory that at least one author of each accepted paper be present during the conference.

In addition, the organisers decided not to have parallel sessions (because of lack of facilities?) during ECIR 2010. Therefore, several full papers were turned into poster presentations, which were held during the short lunch period. This was a very bad move, as because of the setting, these papers received much less attention and credit, even compared to the actual posters, the session of which was rather successful. Some delegates argued that some of the full-papers-turned-posters should have been given a full presentation slot, in lieu of those full papers with a no-show author.

Other than the problems mentioned above, the conference program was generally of a very good quality. In the first day, we enjoyed an excellent tutorial by two MSR researchers on Machine Learning for IR. The tutorial was given by Paul Bennett and Kevyn Collins-Thompson. We also enjoyed an equally excellent tutorial on Crowdsourcing by Omar Alonso from Bing.

In the next days, there were also several good papers that are worth reading:

• A language modeling approach for temporal information needs (from Max-Planck)
• The role of query sessions in extracting instance attributes from web search queries (from Google)
• Interpreting user inactivity on search results (from Univ. of Washington, Univ. of Patras)
• Learning to distribute queries onto Web search nodes (from Yahoo!)
• Temporal shingling for version identification in Web archives (from Max-Planck)
• Evaluation and user preference study on spatial diversity (University of Sheffield)
  

The best paper award was jointly awarded to:

• Promoting ranking diversity for biomedical information retrieval using Wikipedia. Jimmy Huang and Xiaoshi Yin (York University)
• Evaluation of an adaptive search suggestion system. Sascha Kriewel and Norbert Fuhr (University of Duisburg-Essen, Germany)

We have also had the chance to present our two full-papers on search result diversification, and learning to select:

• Explicit search result diversification through sub-queries by Rodrygo L. T. Santos, Jie Peng, Craig Macdonald, and Iadh Ounis. Rodrygo presented our xQuAD search results diversification framework, and the talk was very well received by the delegates, leading to several questions, and many comments that this was arguably the best presentation of the conference.
• Learning to select a ranking function by Jie Peng, Craig Macdonald and Iadh Ounis. This was one of the full-paper-turned-poster presentations. Jie presented the poster, which attracted a lot of attention and led to some very interesting discussions.

Finally, during the posters/demos session, two good contributions particularly caught our attention:

• An Empirical Study of Query Specificity (Poster) - Avi Arampatzis and Jaap Kamps
• NEAT :News Exploration Along Time (Demo) - Omar Alonso, Klaus Berberich, Srikanta Bedathur and Gerhard Weikum

The conference had also an Industry day, which we missed. You can see a report on the Industry day in the following blog post. During the conference, a few of us actively twittered the conference sessions. You can look at the archived ecir2010 hashtag for more details.

One of the most exciting moments of the conference was our visit to the Bletchley Park as part of the ECIR 2010 social dinner. This was an excellent venue with a lot of history, and the food was also good! During the dinner, we were given an impossible quiz to answer. Despite the wine, and a long day, some delegates did manage to find the answers.

Usually, when ECIR is held in the UK, the last day of the conference is the venue for Annual General Meeting of the BCS IRSG - the umbrella group for ECIR. However, in 2010, there was no AGM. We can only suppose that this was because the 2009 AGM was only held in October, co-located with Search Solutions 2009 at BCS HQ. We say suppose, because at the time of writing, the 2009 AGM minutes are not yet available!

Finally, we would like to thank the organisers for their hard work during the conference, for the idea of the ball-bouncer game during the session breaks, which was really cool/fun and for an overall reasonably organised conference. We look forward to ECIR 2011 in Dublin!

Terrier 3.0 released

Firstly, we have a new website for Terrier: http://terrier.org

Also, we have just released Terrier 3.0!

This is a major update to Terrier, including:

• support for indexing WARC collections (such as ClueWeb09)
• improved MapReduce mode indexing
• improved and more scalable index structures
• added field-based and proximity term dependence models, such as BM25F, PL2F and Markov Random Fields
• new Web-based retrieval interface

Fuller changelog at http://terrier.org/docs/current/whats_new.html

If your looking for our team publications, etc., please see our new team website: http://terrierteam.dcs.gla.ac.uk/

Thanks are due to everyone in the Terrier Team for their hard work to make this release, as well as the contributions and feedback about Terrier from our users and collaborators.

TREC Blog Track 2010

The TREC Blog track will be continuing in 2010. In
 2009, 
the
 Blog 
track 
has
 been
 markedly 
revamped
, addressing 
more
 refined
 Blog 
search 
scenarios
 using 
the new Blogs08 collection, a
 large
 sample 
of
 the 
blogosphere covering the period of 14th January 2008 to 10th February 2009.

A summary of the TREC Blog track 2009 edition has been presented by Iadh Ounis at the main TREC conference (Slides). The Blog track 2009 overview paper will be available on the TREC website shortly, once it is updated and reviewed.

The details of the TREC 2010 Blog track are still being finalised by the organisers. However, following the discussions at the TREC 2009 Blog track workshop, here are some salient details (see also the TREC 2009 Wrap-up Slides):

1. Faceted blog search task will run again in 2010: The task addresses
 the 
quality aspect
 of
 the
 retrieved blogs
. It is a feed search task.

• We will adopt a two-stage submission procedure: (1) a participating group submits "topically-relevant"blogs for each query; (2) a few standard baselines will be distributed to participants, so that they can re-rank them with respect to various facet inclinations (e.g. opinionated, in-depth, personal).
• Groups can participate in stage 2 without stage 1, and vice-versa. Stage 1 is akin to an adhoc blog search task.
• More topics for various facet inclinations.

2. Top news story identification task will run again in 2010: The task addresses the 
news‐related 
dimension
 of 
the 
blogosphere. In particular, it investigates whether the blogosphere can be used to identify the most important news stories of the day.


• Real-time news search task rather than retrospective.
• Much larger and a more comprehensive headlines sample, provided by a major news organisation.
• A two-stage submission procedure: (1) Groups submit a ranking of top stories for some days per-category (e.g. sport, politics, business, etc.) (2) We will then select some top relevant stories, for which we will ask the participating groups to identify the related blog posts, in a manner that covers the various/diverse aspects of each story.
• Groups can participate in stage 2 without stage 1. In the latter case, its is an adhoc diversity blog post search task, where the headline is the query.

We welcome any feedback and comments on the tasks above to trecblog-organisers (at) dcs.gla.ac.uk

Finally, note that if you wish to participate in TREC 2010, you should answer the TREC 2010 call for participation. We will update the Blog track wiki as things become more refined - keep following the Blog track developments as they happen on our dedicated Wiki web site.