TerrierTeam

Terrier 3.5 released

2011-06-16T17:41:00.009+01:00

Today, we are proud to announce a brand new release of Terrier, our state-of-the-art open source information retrieval platform. Terrier 3.5 represents a significant update over its previous version (Terrier 3.0), including:

Document-at-a-time (DAAT) retrieval for large indices
Refactored tokenisation for enhanced multi-language support
Upgraded Hadoop support to version 0.20
Synonym support in query language and retrieval
Out-of-the box indexing support for query-biased summaries and improved example web-based interface
New, 2nd generation DFR models as well as other recent effective information-theoretic models
Fully revised and improved documentation
Many more JUnit tests (now 300+)

Check out the full change log for this release and upgrade to Terrier 3.5!

Many thanks to everyone at the TerrierTeam and all Terrier contributors for their hard work making this release possible!

ECIR 2011 + DDR 2011 in Dublin

2011-04-27T12:42:00.010+01:00

Last week, a few of us attended ECIR 2011 in Dublin. The conference was a resounding success both in terms of its program and organisation. Compared to last year, the event was very well attended with about 250 delegates registered to the conference and/or its satellite events. The majority of delegates were from Ireland and the United Kingdom.

Workshops

The kick-off was on Monday, with a selection of workshops and tutorials at the fabulous Guinness Storehouse. We attended the Diversity in Document Retrieval (DDR 2011) workshop, jointly organised by Craig Macdonald, Jun Wang, and Charlie Clarke.

The DDR workshop was sometimes a standing-room only event and appeared to be the largest workshop of the conference. It was structured around three broad themes: evaluation, modelling, and applications. Besides good keynotes by Tetsuya Sakai and Alessandro Moschitti, the workshop featured technical and position paper presentations, as well as a poster session and a breakout group discussion on all three workshop themes. While there was no agreement on a possible "killer application" for diversity, there was a consensus that diversity is best described or seen as the lack of context. In addition, a few key points arose across the boundaries of the tackled themes:

Representing diversity
How to best represent the possible multiple information needs underlying a query? Should this representation reflect the interests of the user population, or should it be itself diverse?
Measuring diversity
What does diversity mean and how should it be promoted in different scenarios? The workshop featured some ideas for applications, including expert search, geographical IR, and graph summarisation.
Unifying diversity
How to diversify across multiple search scenarios (e.g., multiple verticals of a search engine)? How to convey a summary relevant to multiple information needs in a single page of results?

Some of these ideas are currently being investigated as part of the NTCIR-9 Intent task. Charlie was also keen to consider these questions in future incarnations of the diversity task in the TREC Web track. During the workshop, Rodrygo presented our position paper entitled "Diversifying for multiple information needs". The full DDR workshop proceedings are available online.

While we haven't attended it, it was of note that the Information Retrieval Over Query Sessions workshop, which was held at the same time as DDR, also received very good and positive feedback from its attendees.

The workshops were followed by an excellent welcome reception where the least we could say is that Guinness was not in shortage.

Conference

On Tuesday, the main conference took over with a diverse (no pun intended) program. The conference started with a thoughtful keynote by Kalervo Järvelin who urged the information retrieval community to see beyond the [search] box. The keynote led to some very interesting discussions about whether IR is a science or a technology (i.e. mostly about engineering). We would like to believe that it is science, although some delegates argued (sadly) for the opposite.

The second keynote was given by Evgeniy Gabrilovich, winner of this year's KSJ Award. Evgeniy provided a very comprehensive overview of the fascinating computational advertising field, highlighting the current state-of-the-art and possible future research directions. We were encouraged to hear about the Yahoo! Faculty Research and Engagement Program (FREP), which might allow academics to access the necessary datasets to conduct research in a field that has been thus far the sole territory of researchers based in industry.

The last keynote talk was superbly given by Thorsten Joachims about the value of user feedback. Thorsten convincingly argued for the importance of collecting user feedback as an intrinsic part of both the retrieval and learning processes. The talk highlighted how user feedback could improve the quality of retrieval and by how much. We wish that the slides will be made publicly available at some point.

As for the rest of the program, there were two types of papers/presentations: full papers were presented in 30 min, while short papers had only 15 min. As usual, the quality of papers (or at least the presentations) varied from the outstanding to the less good. One suggestion for future ECIR conferences is to limit all the talks to at most 20 min, encouraging conciseness and pushing the speakers to focus on the "message out of the bottle". Indeed, some talks appeared to be exceedingly long with respect to their informative content. While we see the value of giving a 30 min slot to a 10-pages long ACM-style paper, there does not seem to be a valid reason for giving that much time for a (comparatively much shorter) 12-pages LNCS-style paper.

It was interesting to see several Twitter-related papers in the program, suggesting that the community will find the upcoming new TREC 2011 Microblog track and its corresponding shared dataset particularly useful/helpful. The theme of crowdsourcing was also highly featured in the conference, with several papers showing how cheap and reliable relevance assessments could be obtained through the Amazon Mechanical Turk or similar services. Finally, we were very pleased to see many presented papers using our open source Terrier software in their experiments.

Overall, a few papers caught our attention and were particularly interesting:

On the contributions of topics to system evaluation
Steve Robertson
Caching for realtime search - in our opinion by far the best paper/presentation of the conference
Edward Bortnikov, Ronny Lempel and Kolman Vornovitsky
Are semantically related links effective for retrieval?
Marijn Koolen and Jaap Kamps
A methodology for evaluating aggregated search results - Excellent paper/presentation that was also awarded the best student paper award
Jaime Arguello, Fernando Diaz, Jamie Callan and Ben Carterette
Design and implementation of relevance assessments using crowdsourcing
Omar Alonso and Ricardo Baeza-Yates
The power of peers
Nick Craswell, Dennis Fetterly and Marc Najork
Automatic people tagging for expertise profiling in the enterprise
Pavel Serdyukov, Mike Taylor, Vishwa Vinary, Matthew Richardson and Ryen W. White
What makes re-finding information difficult? A study of email re-finding
David Elsweiler, Mark Baillie and Ian Ruthven

Of course, we also recommend our own paper, which was nominated for best paper award, and for which we received excellent feedback:

Learning models for ranking aggregates
Craig Macdonald and Iadh Ounis

The program also featured a busy poster and demo session. We liked the work of Gerani Keikha, Carman and Crestani concerning identifying personal blogs using the TREC Blog track, and that of Perego, Silvestri and Tonellotto, which suggests that document length can be quantized from docids without loss of retrieval effectiveness. There were also several interesting demos that caught our eye:

ARES - A retrieval engine based on sentiments: Sentiment-based search result annotation and diversification - which used our xQuAD framework for diversifying sentiments
Gianluca Demartini
Conversation Retrieval from Twitter
Matteo Magnani, Danilo Montesi, Gabriele Nnziante and Luca Rossi
Finding Useful Users on Twitter: Twittomender the Followee Recommender - addressed the Who to Follow (WTF?) task on Twitter
John Hannon, Kevin McCarthy and Barry Smyth

The ECIR organisers hosted a particularly sumptuous conference banquet at the impressive, unique and beautiful venue of The Village at Lyons Demesne in County Kildare. The journey to the village was a welcome break from the hotel setting of the conference and its technical program.

On the last day of the conference, and concurrently to the technical research sessions, an Industry Day event was under way. However, we only had the chance to go and see the excellent talk by Flavio Junqueira on the practical aspects of caching in search engine deployments. There is a comprehensive summary of the whole Industry program in this blog post. We believe that the planning of the Industry Day event in parallel to the technical sessions was detrimental to attendance. Next year, the Industry Day will be held after the conference ends.

Finally, we would like to thank the organisers of ECIR 2011 for a very enjoyable conference, and a great stay in Dublin. ECIR 2012 will be held in Barcelona, Spain, between 1st and 5th April 2012. We hope to see you all there.

TREC 2010 Roundup

2010-11-26T12:39:00.026Z

Back from another successful TREC conference on the NIST campus. 2010 is a transition year, with the end of old tracks and the proposition of new ones. Indeed, TREC is moving with the times, looking at new data sources and test collections, as well as new evaluation strategies.

Outwith the old . . .

For example, TREC 2010 marks the end of the Relevance Feedback and Blog tracks. While TREC 2010 will be the last year of the Relevance Feedback track, the Blog track, which has been running for the last 5 years, is now morphing into a new Microblog track, investigating real-time and social search tasks in Twitter. A brand new test collection possibly containing 2 months of tweets is planned, with linked web-pages and a partial follower graph. Join the Microblog track googlegroup to obtain the latest updates and follow the Microblog track on Twitter.

TREC 2011 will also witness the initiation of the new Medical Records track, dedicated to investigating approaches to access free-text fields of electronic medical records.

On the test collection front, the Web track is also forward planning a new large-scale dataset to replace ClueWeb09. Indications are that this new dataset will be about the same scale as ClueWeb09 but might provide more temporal information (multiple versions of a page or site over time). Moreover, we have suggested that this might be the heart of a larger dataset comprised of multiple parallel/aligned corpora, for example blogs and news feeds covering the same timeframe.

TREC Assessors, Relevant?

In terms of evaluation, 2010 marks the first year where evaluation judgments were crowdsourced using an online worker marketplace, as opposed to relying on TREC assessors, the participants themselves, or a select group of experts. Indeed, both the Blog track and the Relevance Feedback track crowdsourced some of their evaluation (although the Relevance Feedback track suffered many setbacks and its crowdsourcing process is still incomplete). Furthermore, to investigate the challenges in this new field of crowdsourcing, a specific Crowdsourcing track has been created and will run in 2011. More details can be found here.

Themes

As usual, themes emerged within the various tracks. Learned approaches were far more prevalent this year, now that training data was available for the ClueWeb09 dataset. Indeed, the Web track was dominated by trained models mostly based on link and proximity search features. Diversification, on the other hand, remains a challenging task, with the top groups leaving their initial rankings as is. An outstanding exception is our own approach using the xQuAD framework under a selective diversification regime, which further improves our strongly performing adhoc baseline. Craig Macdonald presented our work in the Web track plenary session.

In the Blog track, voting model-based and language modeling approaches proved popular for blog distillation. For faceted blog ranking, participants employed variants of facet dictionaries to either train a classifier or as features for learning. For the top news task, participants deployed a wide variety of methods to rank news stories in a real-time setting, from probabilistic modeling to blog post voting with historical evidence. Richard Mccreadie presented our work on the blog track as a poster during TREC 2010, which attracted very interesting discussions.

During the TREC conference, Iadh Ounis, Richard Mccreadie and others have done a fair amount of tweeting. You can follow some bits of the TREC conference through the #trec2010 hashtag.

CIKM 2010 in Toronto, ON, Canada

2010-11-03T18:03:00.006Z

I'm back from Toronto, where a few of us attended the CIKM 2010 conference last week. On Friday, I presented our paper on "Selectively diversifying Web search results", a joint work with Craig Macdonald and Iadh Ounis. This work extends our successful participation in the diversity task of the TREC 2009 Web track, by investigating the need for search result diversification in the first place. In particular, we proposed a novel supervised learning approach to predict not only whether promoting diversity is beneficial, but also how much diversification should be applied to attain an effective retrieval performance on a per-query basis. After thorough, large-scale experiments with over 900 query features, we found that our selective approach can substantially improve existing diversification approaches, including our state-of-the-art xQuAD framework. Nonetheless, we believe the significance of our contribution goes beyond these successful results. Indeed, it was with great pleasure that we heard from the NTCIR organisers that NTCIR-9 will run an Intent task, aimed---among other things---at selectively diversifying search results, an area where we are proud to be pioneers.

Besides our own paper, a few other papers caught my attention:

Web Search Solved? All Result Rankings the Same? by Hugo Zaragoza, B. Barla Cambazoglu and Ricardo Baeza-Yates
Reverted Indexing for Feedback and Expansion, by Jeremy Pickens, Matthew Cooper and Gene Golovchinsky
Rank Learning for Factoid Question Answering with Linguistic and Semantic Constraints, by Matthew Bilotti, Jonathan Elsas, Jaime Carbonell and Eric Nyberg
Organizing Query Completions for Web Search, by Alpa Jain and Gilad Mishne
Clickthrough-Based Translation Models for Web Search: from Word Models to Phrase Models, by Jianfeng Gao, Xiaodong He and Jian-Yun Nie

The conference also featured great keynotes, of which those by Jamie Callan and Susan Dumais deserve a particular mention. Jamie talked about his view for the future of search, in which search engines capable of fully leveraging the structure of queries and documents would enable more sophisticated applications built on top of them. Susan addressed the temporal evolution of Web content, how it impacts the way users access this content, and how test collections should account for it. For more details, have a look at the excellent posts by Gene Golovchinsky on Jamie and Susan's talks.

Last but not least, many of us were involved in promoting the next edition of CIKM, to be held here in Glasgow. There was a lot of excitement from the several people that visited our booth, and also during the hand-over talk at the end of the conference. Well done Jon, Mary, Craig, and Iadh for the hard work! The arrangements for CIKM 2011 are well advanced, and the call for papers is now online. You can also follow the latest news about CIKM 2011 on Twitter, Facebook, LinkedIn, and Lanyrd. We look forward to welcoming you all to Glasgow next year!

Terrier Team at SIGIR 2010 in Geneva

2010-07-20T08:49:00.006+01:00

SIGIR 2010 has just started in Geneva. From the TerrierTeam, Richard and myself are attending.

On Monday, Richard presented his PhD topic, Leveraging User-generated Content for News Search at the doctoral consortium.

Later, at the Web Ngram workshop, I'll be presenting a paper on Global Statistics in Proximity Weighting Models.

About the same time, Richard will be presenting at the Crowdsourcing for Search Evaluation workshop. His paper on Crowdsourcing a News Query Classification Dataset examines the effectiveness of different interfaces for having Mechanical Turkers classify queries as news-related or not.

Last but not least, and continuing on our proximity theme, Nicola Tonellotto from CNR is presenting our joint work titled Efficient Dynamic Pruning with Proximity Support at the Large Scale & Distributed Systems workshop.

Meanwhile, please say hello if you see us at the conference, or stay up to date by following #sigir2010. And remember, if you are near the registration desk, please pick up flyers for Terrier and CIKM 2011.

Top Authors in Information Retrieval

2010-07-19T13:04:00.017+01:00

Thanks to Sérgio Nunes who alerted us to this ranking by Microsoft Academic Search of the Top Authors in Information Retrieval, in the past 5 years.

According to this recent ranking, two members of the TerrierTeam, namely Iadh Ounis and Craig Macdonald, are in the top 5 authors in Information Retrieval in the past 5 years (position #1 and #4, respectively). The ranking is based on in-domain citations.

This good news comes just at the start of the SIGIR 2010 Conference, which will be held in Geneva, Switzerland this week (19-23 July 2010). Several members of the team will be in attendance.

WWW 2010 in Raleigh, NC, USA

2010-05-04T01:48:00.014+01:00

I am back from the sunny Raleigh, NC, USA. Besides the nice weather, I had a great time last week attending the 19th International World Wide Web Conference (WWW 2010), where I presented our paper on Exploiting query reformulations for Web search result diversification, a joint work with Craig Macdonald and Iadh Ounis. The paper introduces a probabilistic formulation of our xQuAD framework for search result diversification, and analyses the effectiveness of query reformulations provided by three commercial search engines for the diversification task. My talk was very well received, with lots of questions from the audience, and subsequent chatting with many people from both academia and industry.

The blend academia-industry was indeed a signature of WWW. I was also impressed with the multidisciplinary nature of the conference—with up to five parallel sessions, there was always something for everyone! In particular, from the sessions I attended, a few papers caught my attention:

Clustering query refinements by user intent, by Eldar Sadikov et al. (Stanford University and Google)
Optimal rare query suggestion with implicit user feedback, by Yang Song and Li-wei He (Microsoft Research)
Building taxonomy of Web search intents for name entity queries, by Xiaoxin Yin and Sarthak Shah (Microsoft Research)
Exploring Web scale language models for search query processing, by Jian Huang et al. (Microsoft Research Asia, Facebook, and Penn State University)
Classification-enhanced ranking, by Paul N. Bennett et al. (Microsoft Research)
Ranking specialization for Web search: A divide-and-conquer approach by using topical RankSVM, by Jiang Bian et al. (Georgia Tech and Yahoo! Labs)
Generalized distances between rankings, by Ravi Kumar and Sergei Vassilvitskii (Yahoo! Research)
Relational duality: Unsupervised extraction of semantic relations between entities on the Web, by Danushka T. Bollegala et al. (University of Tokyo)

The conference also featured three passionate keynotes:

Vint Cerf discussed a broad range of topics of interest on today's Web, where everything is connected: 1.8 billion users, around a billion Web-enabled mobile devices, and still a large room for growth in developing countries. Touched points included the implications of the explosion of data production on mobility, accessibility, security and privacy, intellectual property, digital preservation, as well as new technologies (e.g., cloud computing).
dannah boyd discussed privacy implications of the availability of "big data". Her keynote revolved around common misconceptions associated with the analysis of data produced by online social activities, as well as ethical concerns related to using this data in the first place, "just because it is accessible".
Carl Malamud from public.resource.org described his experiences trying to convince seven bureaucratic institutions to make public data publicly accessible. His keynote was organised around "10 rules for radicals", a guide on how to break the barriers towards negotiating with bureaucrats.

On Thursday night, the conference banquet featured an exciting performance by the North Carolina string band Carolina Chocolate Drops. Check out Snowden's Jig (Genuine Negro Jig) and Don't get trouble in your mind for a taste.

Friday held the closing ceremony, with the announcement of the award winners.

Best Paper:

Factorizing personalized Markov chains for next-basket recommendation, by Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme (Osaka University and University of Hildesheim)

Best Student Paper:

Privacy wizards for social networking sites, by Lujun Fang and Kristen LeFevre (University of Michigan)

Best Posters:

How much is your personal recommendation worth, by Paul Dütting, Monika Henzinger and Ingmar Weber (EPFL Lausanne, University of Vienna, and Yahoo! Research)
SourceRank: Relevance and trust assessment for deep Web sources based on inter-source agreement, by Raju Balakrishnan and Subbarao Kambhampati (Arizona State University)

The closing ceremony also featured a short presentation of WWW 2011, to be held in Hyderabad, India. WWW 2012 will take place in Lyon, France.

Finally, on Saturday, the IW3C2 announced the Brazilian bid as the winner to host WWW 2013, which I was very glad to hear about!

RIAO 2010 in Paris, France.

2010-04-28T13:47:00.016+01:00

The 9th International RIAO Conference has started in Paris, France (28-30 April, 2010). It is unfortunate that it is being held concurrently with WWW 2010 in Raleigh.

The first RIAO conference was held in Grenoble in 1985. RIAO is currently a triennial conference, addressing Information Retrieval research topics of interest to both Academia and Industry. This year, the conference focuses on Adaptivity, Personalization and Fusion of Heterogeneous Information.

The following papers have caught my eyes, while browsing the RIAO 2010 program:

Boiling down information retrieval test collections. T. Sakai et al. (Microsoft Research Asia, CMU)
Improving tag recommendation using social networks. A. Rae et al. (The Open University, Yahoo! Research Barcelona).
Analysis of robustness in trust-based recommender systems. Z. Cheng and N. Hurley (UCD)
Opinion-finding in blogs: A passage-based language modelling approach. M. Saad Missen et al (IRIT)
Predicting query performance using query, result, and user interaction features. Q. Guo et al. (Emory University/Microsoft Research)
Towards a collection-based results diversification. J.A. Akinyemi et al. (University of Waterloo)

In addition, the TerrierTeam has two full papers, which are being presented today at the conference (hopefully, the slides will follow shortly):

Voting for Related Entities by R.L.T. Santos, C. Macdonald and I. Ounis. The paper addresses the problem of entity search, where the goal is to rank not documents, but entities in response to a given query. The paper proposes to tackle this problem as a voting process, by considering the occurrence of an entity among the top ranked documents for a given query as a vote for the existence of a relationship between this and the entity in the query. The approach led to high precision and unparalleled recall compared to TREC 2009 systems.
News Article Ranking: Leveraging the Wisdom of Bloggers by R. McCreadie, C.Macdonald and I. Ounis. The paper investigates how news article ranking can be performed automatically, so as to assist editors in selecting the articles, which should make the front page of their newspaper. In particular, the paper investigates the blogosphere as a prime source of evidence, on the intuition that bloggers, and by extension their blog posts, can indicate interest in one news article or another. The paper proposes to model the automatic news article ranking task as a voting process, where each relevant blog post acts as a vote for one or more news articles. The approach led to the best TREC 2009 retrieval performance in the Blog track.

Craig Macdonald is tweeting the conference, pending an appropriate wireless signal. You can follow some bits of the RIAO conference through the #riao2010 hashtag.

ECIR 2010 in Milton Keynes: A Report

2010-04-07T13:04:00.043+01:00

Last week, five of us attended the ECIR 2010 conference in Milton Keynes. The conference was fairly well-organised, although it markedly lacked the lustre of the previous three editions of the conference. In terms of attendance, only about 170 delegates have registered, much less than Glasgow 2008 (210+), and Toulouse 2009 (180+). Perhaps, the exotic town of Milton Keynes was not deemed to be a very attractive venue for a conference. In fact, apart from attending the conference, there was not much else to do -- e.g. the nearest proper pub was at about 2 miles from the conference venue.

The ECIR 2010 conference has suffered from a new and previously unseen problem: several authors and presenters did not make it to the conference, preferring to give their presentation by proxy or using a pre-recorded talk. No less than 5 no-shows were recorded during the conference. Even the keynote speaker and winner of the first BCS IRSG Karen Sparck Jones award, Mirella Lapata, did not show up and gave her presentation through a pre-recorded video. While Lapata certainly had a valid reason (as probably did the other speakers) not to show up, it is clear that ECIR should concretely deal with such a problem, e.g., by making it compulsory that at least one author of each accepted paper be present during the conference.

In addition, the organisers decided not to have parallel sessions (because of lack of facilities?) during ECIR 2010. Therefore, several full papers were turned into poster presentations, which were held during the short lunch period. This was a very bad move, as because of the setting, these papers received much less attention and credit, even compared to the actual posters, the session of which was rather successful. Some delegates argued that some of the full-papers-turned-posters should have been given a full presentation slot, in lieu of those full papers with a no-show author.

Other than the problems mentioned above, the conference program was generally of a very good quality. In the first day, we enjoyed an excellent tutorial by two MSR researchers on Machine Learning for IR. The tutorial was given by Paul Bennett and Kevyn Collins-Thompson. We also enjoyed an equally excellent tutorial on Crowdsourcing by Omar Alonso from Bing.

In the next days, there were also several good papers that are worth reading:

A language modeling approach for temporal information needs (from Max-Planck)
The role of query sessions in extracting instance attributes from web search queries (from Google)
Interpreting user inactivity on search results (from Univ. of Washington, Univ. of Patras)
Learning to distribute queries onto Web search nodes (from Yahoo!)
Temporal shingling for version identification in Web archives (from Max-Planck)
Evaluation and user preference study on spatial diversity (University of Sheffield)

The best paper award was jointly awarded to:

Promoting ranking diversity for biomedical information retrieval using Wikipedia. Jimmy Huang and Xiaoshi Yin (York University)
Evaluation of an adaptive search suggestion system. Sascha Kriewel and Norbert Fuhr (University of Duisburg-Essen, Germany)

We have also had the chance to present our two full-papers on search result diversification, and learning to select:

Explicit search result diversification through sub-queries by Rodrygo L. T. Santos, Jie Peng, Craig Macdonald, and Iadh Ounis. Rodrygo presented our xQuAD search results diversification framework, and the talk was very well received by the delegates, leading to several questions, and many comments that this was arguably the best presentation of the conference.
Learning to select a ranking function by Jie Peng, Craig Macdonald and Iadh Ounis. This was one of the full-paper-turned-poster presentations. Jie presented the poster, which attracted a lot of attention and led to some very interesting discussions.

Finally, during the posters/demos session, two good contributions particularly caught our attention:

An Empirical Study of Query Specificity (Poster) - Avi Arampatzis and Jaap Kamps
NEAT :News Exploration Along Time (Demo) - Omar Alonso, Klaus Berberich, Srikanta Bedathur and Gerhard Weikum

The conference had also an Industry day, which we missed. You can see a report on the Industry day in the following blog post. During the conference, a few of us actively twittered the conference sessions. You can look at the archived ecir2010 hashtag for more details.

One of the most exciting moments of the conference was our visit to the Bletchley Park as part of the ECIR 2010 social dinner. This was an excellent venue with a lot of history, and the food was also good! During the dinner, we were given an impossible quiz to answer. Despite the wine, and a long day, some delegates did manage to find the answers.

Usually, when ECIR is held in the UK, the last day of the conference is the venue for Annual General Meeting of the BCS IRSG - the umbrella group for ECIR. However, in 2010, there was no AGM. We can only suppose that this was because the 2009 AGM was only held in October, co-located with Search Solutions 2009 at BCS HQ. We say suppose, because at the time of writing, the 2009 AGM minutes are not yet available!

Finally, we would like to thank the organisers for their hard work during the conference, for the idea of the ball-bouncer game during the session breaks, which was really cool/fun and for an overall reasonably organised conference. We look forward to ECIR 2011 in Dublin!

Terrier 3.0 released

2010-03-10T18:56:00.004Z

Firstly, we have a new website for Terrier: http://terrier.org

Also, we have just released Terrier 3.0!

This is a major update to Terrier, including:

support for indexing WARC collections (such as ClueWeb09)
improved MapReduce mode indexing
improved and more scalable index structures
added field-based and proximity term dependence models, such as BM25F, PL2F and Markov Random Fields
new Web-based retrieval interface

Fuller changelog at http://terrier.org/docs/current/whats_new.html

If your looking for our team publications, etc., please see our new team website: http://terrierteam.dcs.gla.ac.uk/

Thanks are due to everyone in the Terrier Team for their hard work to make this release, as well as the contributions and feedback about Terrier from our users and collaborators.

TREC Blog Track 2010

2010-02-23T10:31:00.021Z

The TREC Blog track will be continuing in 2010. In  2009,  the  Blog  track  has  been  markedly  revamped , addressing  more  reﬁned  Blog  search  scenarios  using  the new Blogs08 collection, a  large  sample  of  the  blogosphere covering the period of 14th January 2008 to 10th February 2009.

A summary of the TREC Blog track 2009 edition has been presented by Iadh Ounis at the main TREC conference (Slides). The Blog track 2009 overview paper will be available on the TREC website shortly, once it is updated and reviewed.

The details of the TREC 2010 Blog track are still being finalised by the organisers. However, following the discussions at the TREC 2009 Blog track workshop, here are some salient details (see also the TREC 2009 Wrap-up Slides):

1. Faceted blog search task will run again in 2010: The task addresses  the  quality aspect  of  the  retrieved blogs . It is a feed search task.

We will adopt a two-stage submission procedure: (1) a participating group submits "topically-relevant"blogs for each query; (2) a few standard baselines will be distributed to participants, so that they can re-rank them with respect to various facet inclinations (e.g. opinionated, in-depth, personal).
Groups can participate in stage 2 without stage 1, and vice-versa. Stage 1 is akin to an adhoc blog search task.
More topics for various facet inclinations.

2. Top news story identification task will run again in 2010: The task addresses the  news‐related  dimension  of  the  blogosphere. In particular, it investigates whether the blogosphere can be used to identify the most important news stories of the day. 

Real-time news search task rather than retrospective.
Much larger and a more comprehensive headlines sample, provided by a major news organisation.
A two-stage submission procedure: (1) Groups submit a ranking of top stories for some days per-category (e.g. sport, politics, business, etc.) (2) We will then select some top relevant stories, for which we will ask the participating groups to identify the related blog posts, in a manner that covers the various/diverse aspects of each story.
Groups can participate in stage 2 without stage 1. In the latter case, its is an adhoc diversity blog post search task, where the headline is the query.

We welcome any feedback and comments on the tasks above to trecblog-organisers (at) dcs.gla.ac.uk

Finally, note that if you wish to participate in TREC 2010, you should answer the TREC 2010 call for participation. We will update the Blog track wiki as things become more refined - keep following the Blog track developments as they happen on our dedicated Wiki web site.

AcademTech: Faceted People Search

2009-08-04T14:16:00.017+01:00

AcademTech is a Computing Science-specific expert search engine based on the Terrier IR Platform. Persons working at Computing Science departments in Scottish Universities are considered as candidate experts by the system. Profiles of their expertise evidence are then mined from their homepages, publicly available digital libraries (e.g. DBLP) and related information found on the Web through Yahoo! BOSS. The ranking of experts is provided by a variant of the Voting Model expert search approach.

The system is integrated with a novel faceted search interface to allow users to browse and explore the results using a number of categories such as Location or Conference/Journal publications. Each expert in the system has a profile page containing a number of elements including query specific supporting publications, most informative associated terms displayed as a tag cloud, co-authors and web links. Although the system is currently applied in the context of Scottish Computing Science Academia, it can easily be expanded to go beyond its current Scottish scope, cover other academic fields, and people in general.

I was lucky enough to be able to demo AcademTech at SIGIR 2009 in Boston on July 20th. Thankfully, I spoke to a large number of attendees receiving largely very helpful feedback.

A popular suggestion was to utilize AcademTech's core system in the scope of biology. This would meet the medical field's need for finding related organisms, diseases etc. Possible facets in the area would likely be biological classifications such as species and genus.

Daniel Tunkelang from The Noisy Channel suggested providing profile page-located facets, allowing filtering of search results by features present in a selected expert's page such as co-authors. This would satisfy an example scenario such as "Show me co-authors of this expert who work for the University of Glasgow." Profile facets could also allow the experts publications list to be filtered by a number of fields such as co-author location, conference etc.

Much of the feedback mirrored that of intended future work. Name disambiguation is a high priority update as a current problem with AcademTech is the publication mismatch when multiple experts have the same name. In fact, the system is specifically designed to allow for expansion of facets, and name disambiguation. With a large amount of publication collaborators working in industry a useful move would be to expand to accommodate these experts.

AcademTech is now publicly accessible from http://www.terrier.org/academtech
A description of the system is available in the SIGIR'09 proceedings.

Thank you to all those who spoke to me and gave me some great feedback.

SIGIR 2009: Expert Search from Glasgow

2009-07-21T16:08:00.005+01:00

A short update from SIGIR09 to announce our recently published work on expert search. This should hopefully be the first of a series of a few posts about SIGIR this year.

In On Perfect Document Rankings for Expert Search (Craig Macdonald & Iadh Ounis), we examine the effect of the document ranking to an expert search engine. Intuitively, improving the topical relevance properties of the document ranking usually leads to an improvement in the performance of the generated ranking of documents. In this poster, we examine the extreme case, by making the document ranking component perfect with respect to topical relevance.

In Usefulness of Click-through data in Expert Search (Craig Macdonald & Ryen White), we examine how user clicks on an intranet search engine can be used as features by an expert search engine. The proposed techniques are based on the voting techniques from the Voting Model, but examine documents clicks instead of weighting model scores. To our knowledge, this is the first work examining how clicks can be integrated into expert search.

Finally, the Voting Model was show-cased in the Expertise Search in Academia using Facets (Duncan McDougall & Craig Macdonald), which demoed AcademTech, a faceted search interface for expert search in academia.

CIKM 2011 in Glasgow!

2009-06-04T10:38:00.003+01:00

We are delighted that our bid to host the ACM Conference on Information and Knowledge Management (CIKM 2011) in Glasgow has been successful.

After the highly successful ESSIR 2007 and ECIR 2008 events, we are excited at the prospect of hosting the prestigious ACM CIKM Conference in Glasgow in 2011. We look forward to having our colleagues gather in Glasgow, and to surpassing their expectations.

Further information about the conference (dates, venues, etc.) will be available in due course.

CIKM 2009 will be held on November 2-6, 2009, in Hong Kong. Hope to see you there!

TREC Blog track 2009

2009-04-29T12:08:00.005+01:00

We have just released a draft of the guidelines for the TREC 2009 Blog track.

Compared to previous years, the Blog track 2009 aims to investigate more refined and complex search scenarios. In particular, we propose to run two tasks in TREC 2009:

Faceted blog distillation: a more refined version of the blog distillation task that addresses the quality aspect of the retrieved blogs and mimics an exploratory search task. The task can be summarised as "Find me a good blog with a principal, recurring interest in X". We propose several facets for the TREC 2009 blog distillation task, which may be of varying difficulty to identify for the participant systems.

Top stories identification: a new pilot task that addresses the news dimension in the blogosphere. Systems are asked to identify the top news stories of a given day, and to provide a list of relevant blog posts discussing each news story. The ranked list of blog posts should have a diverse nature, covering different/diverse aspects, perspectives or opinions of the news story.

The new Blogs08 collection, an up-to-date and large sample of the blogosphere from January 2008 to February 2009, will be used for both tasks.

We welcome feedback. Please feel free to post feedback and comments about the proposed tasks for 2009.

Blogs08 Collection Released

2009-04-09T20:05:00.003+01:00

We are pleased to announce that the Blogs08 collection is now ready for distribution. As announced before, Blogs08 is one order of magnitude bigger than Blogs06, and samples the blogosphere from January 2008 to February 2009. The uncompressed permalink size is approx 1.3TB, while including feeds, this amounts to over 2TB of data. As usual, the data is shipped compressed on a SATA hard drive.

The distribution mechanism will be the same as for Blogs06. There is specific information about the size of the collection here, while the instructions for obtaining the collection are here.

If you intend on participating in the TREC 2009 Blog track, please start working on the paperwork right away, so that you can get the collection as soon as possible. Due to the larger size of the collection, we will operate a queuing system for shipping the data. Moreover, if you haven't done so already, respond to the TREC 2009 Call for Participation.

Blog track co-ordinators are finalising the guidelines for this year's tasks and will continue to update the TREC Blog wiki, the TREC blog track mailing list and this blog.

Craig's Thesis Available

2009-03-03T16:52:00.007Z

Following up from my successful defence, I'm pleased to announce that my thesis, titled The Voting Model for People Search is now available online.

My thesis proposes the Voting Model for various people search problems, such as expert search in enterprise settings (find me someone who knows about...) , or blog(ger) search (find me a blog about the general topic...). I also examine the reviewer assignment problem (suggest for me reviewers for this paper...). In general, the Voting Model is concerned with the ranking of aggregates of documents.

Experimental chapters are mainly carried out using TREC Enterprise track and Blog track test collections.

Correlator launched

2009-02-22T11:10:00.013Z

Yahoo! Research has launched a new search engine called Correlator. It uses advanced techniques from Natural Language Processing and Computational Linguistics to locate entities within text and to group sentences about these entities from different documents. In his talk at the ECIR 2008 Industry Day, Hugo Zaragoza who championed the Correlator project at Yahoo! Research Barcelona, described some of the system's underlying approaches and technologies. In a blog post introducing the search engine, he states:

The core of Correlator is a search engine capable of returning not only relevant documents, but also relevant sentences and entities.

Currently, Correlator uses Wikipedia as the underlying document collection. However, the Correlator team contends that this can be extended to other collections and types of documents such as blogs.

I have quickly tried Correlator this morning. My first impression of the system is that it does extremely well on many queries - e.g. results for queries such as "precision and recall" are pretty good and informative. However, there are several areas for improvement when it comes to identifying relationships between entities. For example, for the query "Tony Blair", when searching for names, the system suggests many entities as 'probably related to Tony Blair', however the precise nature of the relationship between the two entities is not stated, e.g. Cherie Blair should be presented as the definite wife of Tony Blair. Indeed, it is left to the user the task of browsing through various possible suggested relationships between the named entities. However, this might be a design choice by the designers of the system, favouring high coverage over high precision.

Relatedly, it is of note that TREC 2009 will include a new Entity track. One of the currently proposed search tasks is the identification of relationships between entities.

Choosing an issue tracking system

2009-02-20T21:31:00.000Z

I recently posted about our deployment of an issue tracking system for use by the Terrier platform.

When we recently made the decision process to an issue tracking system, I reviewed several alternative products, namely Scarab, Mantis, and JIRA. Firstly, I should say that I was already familiar with JIRA through its extensive usage by the Apache Software Foundation, e.g. by Hadoop and Pig.

When I evaluated all three products, I found Scarab to be the nearest to JIRA, however it required much work to setup an issue tracking system that was full of "attributeGroup1" etc. labels. Scarab also lacked querying features: e.g. it had an advanced SQL inspired querying interface, but no easy way to see "All Issues" or "Most Recently Updated Issues". For me, this is a killer problem, as Bugzilla faces the same issue. It should be easy to see what the current problems are, or what issues people are working on. At the end of the day, as a developer, you spend more time querying an issue tracker than you do filing issues.

I'm no expert at HTML & CSS, so for Mantis, I was pleased when it installed with a cleaner default theme. However, with Mantis, the fields for an issue were hard-coded in the submission and display pages. This was ultimately the downfall for Mantis, as I'm keen on minimising the unnecessary fields, and Mantis had many fields that were not appropriate for Terrier. Another downside for Mantis was that issues were numbered 0000001. In JIRA and Scarab, issues have a project prefix, and no leading zeros, making them instantly recognisable outside of the issue tracker - e.g. if I write TR-5 (e.g. in a SVN commit message), then people are likely to know what I'm on about. In contrast 0000005 is not something that using a search engine they would find quickly.

Finally, I installed a trial for Atlassian's JIRA. We found JIRA easy to browse and query existing issues. The dashboard functionality it also useful, and customisable. Finally, we also liked the user/SE friendly URLs used by JIRA for issues: e.g. http://ir.dcs.gla.ac.uk/terrier/issues/browse/TR-1 for issue TR-1. I'm pleased with JIRA, which exhibits an overall very polished UI. Atlassian have also very kindly provided Terrier with an open source license for JIRA. Thanks Atlassian!

WSDM 09 : Your mileage may vary

2009-02-18T10:32:00.005Z

So, for the second year WSDM arrives, attendance has remained constant (despite the economic downturn) and we're all packing t-shirts rather than bags. However, unwrapping the packaging, what do we find?

Initially, the conference seemed promising, beginning with an excellent speech by Jeff Dean. A 101 on efficiency at Google since 1999, well rounded with explanations and statistics in equal measure. Unfortunately, this staring performance seemed to overshadow the rest of the conference - a benchmark never surpassed.

If I was to use one word to describe WSDM'09, then it would be inconsistent. There were some nice speeches, Eytan Adar's talk on detecting how the web changes over time springs to mind (also best student paper), and Songhua Xu gets bonus points for turning up with an interface paper to a predominantly text-based conference. However, many were poorly presented, insubstantial or both.

The dominating topic of the conference was unsurprisingly Wikipedia, with well over 40% of papers giving it a mention. Ignoring the proliferation of Wikipedia papers over the last year, high point here was Eytan Adar's paper on Information Arbitrage Across Multi-lingual Wikipedia, for coming up with something which might actually be useful in practice.

The videos from the conference should be up soon on videolectures.net, and I would recommend Jeff Deans opening speech - for those of you who can survive watching in tiny eye-strain-o vision which comes with flash. As for the rest remember - your mileage may vary.

WSDM 2009 highlights

2009-02-17T23:30:00.006Z

Richard and I went to Barcelona last week to attend the WSCD 2009 workshop and the WSDM 2009 conference. Craig was also there on Monday to present an interesting poster on the usefulness of click-through data for training.

Besides being held in an exciting city (!), WSDM 2009 kept up with its previous edition in bringing together industry and academia to a common, quality forum for Web IR and data mining, with papers covering a wide range of trendy topics -- fairly well summarised by the tag cloud printed on the t-shirts given to the participants! -- from query intent detection, through search results diversification, to tagging-based clustering and classification, and social network-driven marketing analysis, to name a few.

The best paper award went to Fernando Diaz for his work on the selective integration of news content into Web results based on the classification of the newsworthiness of each query. Eytan Adar et al. received the best student paper award for their study of the dynamics of the content and structure of Web documents of varying popularity over a fine-grained timescale. In the new late breaking results session, the award went to Irem Arikan et al.'s paper on applying a language model approach for improving the retrieval effectiveness for queries with temporal expressions. The invited talks by Jeff Dean and Gerhard Weikum were also insightful -- we couldn't attend Ravi Kumar's though. All talks should be available soon from VideoLectures.net.

Overall, WSDM is rapidly moving towards establishing itself among the major IR conferences. In 2010, it will probably be held in Los Angeles, CA, USA.

Twitter and CEOs

2009-02-16T23:50:00.014Z

Thanks to Theo Huibers for pointing out to an article in Forbes about Why Europe's CEOs should Twitter.

The article reports that unlike their counterpart in the USA, the CEOs of European companies are being slow in embracing the Twitter tool. In general, the article argues that European chief executives are not very aware of the benefits of social networking tools to their businesses, missing out on opportunities to engage with their customers.

If this is true, than this is rather worrying. Indeed, I can easily see many scenarios where social networking tools such as Twitter could be helpful for businesses. The Forbes article mentions several of these. For example, a case where the public relations office of General Motors has used Twitter to clamp down on rumours affecting the company. In his blog, Daniel Tunkelang reported a first-hand experience, when one of his technical questions posted on Twitter received care from the president and COO of GoDaddy.com, albeit with a degree of attention that goes beyond what Daniel bargained for.

It is of interest to note that the Forbes article suggests that the Twitter's interface is still too complex for a widespread adoption by end-users and businesses. While I have only been an occasional user of Twitter, I have never had the feeling that the tool was difficult to use. However, I'm happy to stand corrected by HCI experts!

Grid@CLEF track : a framework for IR experimentation

2009-02-11T21:11:00.008Z

Don't be put of by the title, this isn't a post about Grid Computing. Instead, I'm going to talk about the Grid@CLEF task, which defines a framework and TREC-style track for experimentation with various components of IR systems. Disclaimer: I'm pleased to be on the advisory committee of the Grid@CLEF task.

Firstly, I'll give a bit of background. Cross-Language Evaluation Forum (CLEF) is a spin-off from TREC which concentrates on the evaluation of mono-lingual (non-English) and cross-lingual retrieval. CLEF has been running since 2000, and attracts a wide spread of participating research groups from across the globe, reaching 130 for CLEF 2008.

The tracks have now been defined for CLEF 2009, which includes the Grid track. Nicola Ferro (Univ. of Padova) and Donna Harman (NIST) are the big-wigs for this task, with suggestions from the advisory committee. So what does Grid mean in this context? Well, the idea (in my own words) is that the components of an IR system that have effect can be roughly categorised as follows: tokeniser, stopword list, word-decompounder, stemmer, and ranking function. In the Grid track, the concept is that these components can be interchanged, and a fuller understanding of their impact derived. The Grid framework facilitates such interchanges, by defining a way to allow various mixes of components to be attempted, thus creating a "grid" of experimental results.

However, the problem with such an experiment is that often each of these components is tied to an IR system, and that having the IR system itself can have an impact on the results. Instead, the idea behind the Grid track is that the output from each component (tokeniser, stopword list etc) of a given IR system is saved in an XML format, and shared among participants. In this way, every combination of each component can be investigated.

The Grid@CLEF site describes more the intuitions of the task, including an example of how results will be presented.

Here in Glasgow, we like the concept behind the Grid track. Indeed, it has some similarities to the way we ran the opinion finding task in the TREC 2008 Blog track. In the opinion finding task (where the aim is to retrieve relevant and opinionated blog posts about the target topic), the retrieval performance of opinion identification approaches appears to be linked to the ability of the underlying "topical relevance" retrieval approach. To investigate this in TREC 2008, we provided 5 standard topical relevance baselines, which participants were able to use as input to their opinion finding technique(s). You can read more in the Overview of the TREC 2008 Blog track (Iadh Ounis, Craig Macdonald and Ian Soboroff), which should be released in a few weeks time.

I have committed to implementing Terrier support for the Grid@CLEF track. The XML specification is being agreed by the Grid@CLEF organisers and advisers. However, if you are interested in using Terrier on this task, you can follow the progress on the TR-9 issue concerning Terrier's Grid@CLEF support. The exact specification for the Grid@CLEF XML interchange format is still in flux, but once its settled down, Terrier support should be forthcoming.

Building Terrier by Open Collaboration

2009-02-11T12:55:00.010Z

An important benefit of having an open source IR platform, like Terrier, is that users of the platform can contribute code to the platform, and overall, everyone gains. IR platforms which are not open source may be popular, but can stagnate if it does not evolve to meet modern needs. Open source is a good way of building such a critical mass of people to evolve a project.

To facilitate the task of our users who contribute to Terrier, we are in the process of making changes that will also make the development process easier:

Firstly, we have deployed Atlassian's JIRA as an issue tracker for Terrier.
Secondly, we are opening the source code repository for the Terrier platform (in progress).

An issue tracker allows issues (bugs or feature requests) to be named, discussed, and patches proposed. Other contributors may review and discuss these patches before they are committed. All development work on the Terrier open source platform will now be done via the issue tracker. In deciding to deploy JIRA, we did take some time to review several issue trackers. I'll describe these and how we came to our decision in a future post.

The goal of opening our source code repository is that patches submitted by contributors can be made against the latest (trunk) Terrier source, thus ensuring that no stale patches are received. As a committer this will make my job easier.

I recently announced these changes in Rome at the New challenges in Information Retrieval and Text Mining in an open source initiative workshop. You can see my slides from the workshop below:

Craig successfully defends his thesis

2009-01-22T21:09:00.006Z

I am pleased to announce that last Thursday (15th January), I successfully defended my thesis, titled the Voting Model for People Search. I want to give many thanks to Iadh Ounis for supervising my PhD, and also to my committee: convener David Watt, and, in particular, to examiners Ricardo Baeza-Yates and Phil Gray, for the rewarding discussion and constructive feedback.

I have 4 weeks to make very minor corrections to my thesis, after which time it will be available online.

Terrier 2.2. released, with support for Hadoop Map Reduce indexing

2008-12-24T14:13:00.000Z

I am pleased to announce that Terrier 2.2 was released, just before Christmas. While I have chosen only to increase the minor version number for this release, it a is substantial update, consisting of new support for Hadoop, a Hadoop Map Reduce indexing system, and various minor improvements and bug fixes. (I reserve major version numbers bumps for index format changes).

Our Map Reduce distributed indexing strategy builds upon the single-pass indexing strategy first released in Terrier 2.0. In deployment with a Hadoop cluster, Terrier can index large collections of data in a distributed fashion, splitting the indexing process across various Map and Reduce tasks, which can be run on various nodes in the cluster.

In particular, the input data files for the collection are split across many Map tasks. Each Map task indexes its allocated data files using a normal Collection implementation. Postings lists are built, compressed, in memory. Each time memory is exhausted, these miniature posting lists are emitted from the Map task.

The Reduce task is responsible for aggregating the posting lists for the various terms. Firstly, the Reduce input keys are sorted by term, and the values are sorted by source Map task, to ensure that the posting lists for a given term are processed in the correct order. For each term, the temporary posting lists (the reduce input values) are merged into the final compressed inverted index.

The indices created using the Map Reduce indexer are standard Terrier indices. Moreover, by controlling the number of Reduce tasks, the final index can be partitioned into separate indices, in the local inverted file layout (document partitioning). With a different partitioning scheme, global inverted file layout (term partitioning) would also be possible.

You can see the detailed list of changes for Terrier 2.2. in the documentation.

Mining query logs

2008-12-10T16:28:00.000Z

It is often reported in the literature how search engines can use their query logs to improve document ranking. However, the query logs could also be used for various mining activities. For example, an article in The New York Times described how a power cut in the New York area was reflected in the Google's query logs within 2 seconds after its occurrence, while it took about 15 minutes for newswire services to report the same event.

Relatedly, Abdur Chowdhury in his position talk at the SSM 2008 Workshop mentioned that news about a major earthquake in China were reported on Twitter well before the newswire services. A BBC blog post commented on the same issue.

Finally, the BBC recently reported that Google has developed a system to detect flu outbreaks in the USA by analysing the query logs and identifying the location of people issuing flu-related queries.

Unfortunately, query logs are scarcely available to researchers in academia, especially after the AOL data debacle. This limits scientific work in the field, as most current research results using query logs are not reproducible due to lack of publicly shared data. As a consequence, I very much welcome the forthcoming Workshop on Web Search Click Data (WSCD 2009), where the issue of publicly releasing query logs is being addressed as one of the objectives of the workshop.

The TREC 2008 Blog track workshop

2008-11-30T12:21:00.000Z

We just came back from Gaithersburg a few days ago. It was a nice (and cold!) week at the TREC 2008 conference. Besides presenting the main results of our participation in the Blog, Enterprise, and Relevance Feedback tracks, we had fruitful discussions at the Blog track workshop regarding the directions of the track for 2009.

It was a consensus among the attendees that opinion retrieval and polarity detection are still open, relevant problems. Yet a few groups managed to deploy interesting techniques that achieved consistent opinion retrieval performances across several strongly performing baselines in the track this year, polarity detection approaches looked rather naive. It was suggested that polarity detection be investigated at a finer granularity (e.g., at the sentence rather than the document level). This, however, could result in crossing the boundaries with respect to the TAC conference.

Nonetheless, believing that, after three years, the Blog track has contributed a comprehensive experimental setting for those who wish to continue investigating these search scenarios, the organisers decided to discontinue the opinion finding and polarity tasks, at least in their current format. Instead, they propose to investigate the opinionated nature of blogs as one of many interesting facets of a broader search task. This task extends the current blog distillation task by moving beyond topic relevance and introducing different requirements in order to qualify "good" blogs, i.e., blogs that have a recurrent interest in a given topic and that also fulfil a set of predefined "facets". This way, for instance, one could search for humorous blogs about the government, or opinionated blogs about whisky.

Besides this faceted blog distillation task, a second task was considered relevant and worth investigating by the workshop attendees, namely, tracking stories on the blogosphere. The aim is to investigate how stories emerge and evolve along the time frame of the blog corpus. It was also noted that this task could be linked to a news search task so as to draw a connection between stories published on the blogosphere and on the mainstream media.

As pointed out, however, the 11-weeks time frame of the Blogs06 collection does not adequately support the story tracking task. Furthermore, the availability of a more representative sample of the blogosphere is an important step towards addressing blog search as a social media problem. For such, a new corpus will be used in 2009, with a much larger size and time frame.

For those who did not attend the Blog track workshop at TREC, please feel free to post your comments about the proposed tasks for 2009.

Hope you all join us in the TREC 2009 Blog track!

TREC 2008

2008-11-15T16:31:00.001Z

Shortly, we will be travelling to attend the TREC 2008 conference in Gaithersburg, Maryland (18-21 November 2008). We have been very busy analysing the sheer volume of data that was collected in the Blog track this year. Indeed, this year, we ran a very large-scale experiment with the aim to draw a better understanding of the most effective and stable opinion-finding techniques. Moreover, we also tightened up the blog distillation task (feed search task), so as it truly runs as a distillation task. Following the traditional TREC conference cycle, the Blog track 2008 results will be first presented to the TREC 2008 participating groups next year. They will then be made available to all interested parties around February 2009 when the TREC 2008 final Proceedings go online.

Plans for the TREC 2009 Blog track will be discussed and refined during the TREC Blog track workshop in the afternoon of Thursday 20th November.

In addition to our involvement in the organisation of the Blog track, we will be giving a presentation on the work we did this year in the newly introduced Relevance Feedback track. We have also prepared two posters summarising our results in the Enteprise and Blog tracks.

It looks like we are set for a very exciting and busy week. We hope to see many of you in TREC.

SEMAST 2009

2008-11-10T22:31:00.000Z

We are continuing organising events in Glasgow. After the ESSIR2007 summer school and the ECIR2008 conference, we will be organising the second Practical Semantic Astronomy Workshop (SEMAST 2009) from 2nd to 5th March 2009.

The Practical Semantic Astronomy 2009 is the second in a series of workshops first held at Caltech in February 2008. The workshop brings together experts from a broad range of disciplines using semantic technologies, alongside practitioners experimenting with these techniques, to address current problems in astroinformatics.

Our involvement in the organisation of this workshop is under the auspices of the Explicator project, where we have been working with astronomers and physicists on developing techniques to provide intelligent access to multiple sources. The Explicator project supports the efforts of the Virtual Observatory community.

The Virtual Observatory is a loose planet-wide collaboration of astronomy computing projects, aiming to make available the high-volume and rich data of astronomy. Although astronomical data is generally well-described, it is very dispersed, so that there is a substantial data-discovery problem, making it fertile ground for the sorts of semantic approaches applied with such success in other disciplines.

The Explicator project aims to bridge the gap between information retrieval and semantic web technologies in a domain-specific application. The SEMAST 2009 workshop is a continuation of this effort. We hope to see many of you in Glasgow.

Blogging is also about branding

2008-10-21T20:14:00.000+01:00

Recently, Daniel Tunkelang wrote a blog post about why he was blogging. In a nutshell, he considers blogging to be fun and highlights how it can increase the "reputation capital" of the blogger. Daniel Lemire made a follow-up, stressing the networking benefit of blogging. I very much concur with both views. On the other hand, while blogging, both bloggers do not shun away from expressing their thoughts and opinions on various topics ranging from enterprise and blog search to peer reviewing or the benefit of pure theoretical research, through opinions on a search engine such as Duck Duck Go!

Such perspectives and opinions are not only informative and valuable to readers like myself, but they are also extremely important for various organisations. Indeed, according to an article on the BCS news website, blogging is very important for brands. The article quotes Rachel Hawkes, co-founder and editor of the Social Media Portal (SMP):

Blogs provide an opportunity for a two-way interaction to take place between business and consumer. This allows customers to provide 'incredibly valuable' feedback on how the brand is doing in the real world, which can help guide improvements and sales strategies.

The above scenario is one of the motivations for the opinion-finding search task that we have been investigating in the TREC Blog track since three years. The task addresses a search scenario where a user aims to uncover what the bloggers/consumers are saying or thinking about X. If the "user" is a business, and X is one of its products, then “taking the pulse of the blogosphere” is very important for this business's branding. In fact, the opinion-finding task can naturally be associated with settings such tracking consumer-generated content, brand monitoring, and, more generally, media analysis. Findings and insights gained from 3-years of the opinion-finding search task at the TREC Blog track will be discussed in the furthcoming TREC Conference (18-21 November 2008), held in NIST, USA.

CIKM 2008

2008-10-20T18:52:00.000+01:00

We will shortly be travelling to attend the CIKM 2008 conference in Napa Valley. The organisers are announcing that it will be the biggest ever CIKM conference, and hope that it will be the most memorable one.

Following the ECIR 2008 conference example, I'm pleased to note that the organisers are making CIKM 2008 a green conference, through optimal usage of logistics and resources.

The conference has a very exciting scientific program, and an impressive social program, including a Halloween party.

We will be presenting two full papers in the Blog session on Wednesday 29th October 2008, 10:15-11:45am:. Both papers tackle search tasks investigated within the TREC Blog track:

Key Blog Distillation: Ranking Aggregate. Craig Macdonald, Iadh Ounis (University of Glasgow, UK) - The paper addresses the blog distillation task, as task characterised as “Find me a blog with a principle, recurring interest in X.”
An Effective Statistical Approach to Blog Post Opinion Retrieval. Ben He, Craig Macdonald, Jiyin He, Iadh Ounis (University of Glasgow, UK) - The paper tackles the opinion-finding task in the blogosphere, a task characterised by “What do people think about X?”

The third paper of the Blog session is from UMass, a regular participant in the TREC Blog track. It also investigates the blog distillation search task:

Blog Site Search Using Resource Selection. Jangwon Seo, Bruce Croft (University of Massachusetts Amherst, USA)

We hope to see you in CIKM!

TREC Blog track will run in 2009

2008-10-19T15:14:00.000+01:00

Following our previous post, I'm pleased to report that we have just heard that the TREC program committee has accepted our proposal for the blog track to continue in 2009.

The intention is to use a larger Blog collection, and to have at least one search task that goes beyond topical relevance by taking into account a facet representing an attribute of required "quality".

There will be a workshop to discuss the proposed blog search tasks at the TREC 2008 conference on the afternoon of Thursday 20th November 2008.

If you cannot attend TREC, and wish to make any comments or suggestions, please feel free to post your thoughts in this post, or to email them privately, if you wish so.

Conference Deadline Traffic Jam

2008-09-12T22:22:00.000+01:00

I noted today that Matthew Hurst has posted the ICWSM 2009 Call for Papers. Unfortunately the submission deadline is on the 21st January. This is a full 6 weeks later than for ICWSM 2008. Moreover, this falls four days before the SIGIR full paper deadline.

As an IR researcher, we have to target certain conferences. While I'd like to have multiple papers ready for several conferences with similar deadlines in advance, various pressures and reasons don't make that possible (e.g. I'd like a holiday at Christmas!).

The conference deadlines in January and February now look like:

11th January: WWW 2009 Posters due
19th January: SIGIR 2009 Abstracts due
21st January: ICWSM 2009 Papers/Posters/Demos dues
25th January: SIGIR 2009 Papers due
9th February: NAACL-HLT 2009 Short Papers due
22nd February: ACL-IJCNLP Papers due
23rd February: SIGIR 2009 Posters due

Happy writing!

About Blog Search Tasks

2008-09-10T03:00:00.000+01:00

We have been very busy recently with the TREC 2008 Blog track. Now that all runs have been submitted and that the relevance assessments are on-going, it is the time of the year where we start planning for the future of the track at TREC 2009! Indeed, TREC operates a policy where existing tracks are renewed on an annual basis, and following the submission of a proposal.

Back in 2006, when we first proposed the Blog track, our aim was to have a long-term objective for the track, recognising that the richness of the blogosphere and its peculiarities will require several years of investigation before reaching a full understanding of the different blog search tasks, and how they should be effectively addressed. In particular, we proposed to adopt an incremental approach, where we begin with basic blog search tasks and progressively move to more complex search scenarios.

In the first three years of the track (2006-2008), we addressed two main blog search tasks:

Opinion finding: involves locating blog posts that express an opinion about a given target.
Blog distillation: involves locating blogs that are principally devoted to a topic X over the timespan of the feed.

The first task tackles an important aspect of blogs, namely their opinionated/subjective nature, and the tendency of bloggers to express views, thoughts and feelings towards named-entities. This tasks helps users to find out what the bloggers think about X. The second search task addresses a scenario where the user would like to find a blog to follow or read in their RSS reader. Our main findings and conclusions from the first two years of the Blog track at TREC are summarised in the ICWSM 2008 paper, entitled On the Trec Blog Track. The Blog track 2006 and 2007 overview papers provide further detailed analysis and results.

We are now proposing to move to a second phase of the Blog track, where more refined and complex search scenarios should be investigated. In particular, we are thinking to use a new and larger collection of blogs, which has a much longer timespan than the 11-weeks period covered in the Blog06 collection. This allows investigating another important characteristic of the blogosphere, namely the temporal/chronological aspect of blogging, and various related search tasks such as story identification and tracking.

While we were thinking about such possible future tasks, we came across a position paper by Marti Hearst, Matthew Hurst and Susan Dumais, entitled "What Should Blog Search Look Like?", which will be presented in the forthcoming Search in Social Media (SSM 2008) workshop at CIKM 2008.

In particular, Hearst et al. propose that the blog distillation task should be further refined by taking into account a number of dimensions or attributes such as the authority of the blog, the trustworthiness of its authors, the genre of the blog and its style of writing. For example, a user might be interested in blogs to read about a topic X, but where the blogger expresses in-depth viewpoints, backed up by a scientific methodology or evidence. The Cranfield evaluation paradigm adopted by TREC requires deeper thoughts about how relevance assessments should be conducted in such a scenario.

Unsurprisingly for a strong advocate of the importance of user interfaces and visualisation tools for information retrieval, Hearst together with her co-authors propose a faceted blog search interface to help the user explore the attributes of the blogs before choosing those they wish to follow or read, i.e. exploratory search at its best! The conclusion of the paper provides a good summary of Hearst et al.'s views:

For the problem of selecting a blog to read, we propose a faceted interface which highlights different attributes of interest, with a focus on people and on matching the taste preferences of the reader. For the task of “taking the pulse of the blogosphere,” we suggest that blog data be integrated with other social media and that the existing work on tracking trends and aggregating views is heading in the right direction.

As we are trying to wrap up our proposal for TREC 2009, we would like to hear other suggestions and comments about what blog search should look like. Please feel free to post your thoughts and comments in this post, or to email them privately, if you wish so.

From Expert Search to Commoditising Workers

2008-09-01T22:25:00.001+01:00

While I'm putting the finishing touches to my PhD thesis (titled The Voting Model for Expert and Blog Search), I thought I'd pick up on a recent related article.

An excerpt from The Numerati has been published on BusinessWeek.com. In the excerpt, Stephen Baker interviews the scientist Samer Takriti while he was working at IBM . Samer, who is a specialist in Operations Research, is working on commoditising workers. Similar to how supply chains and production lines have been modelled and improved, Samer believes that people can be assigned to projects using combinations of their availability, their scost, and their skills/expertise. The idea is to optimise the use of co-workers, leading to a better productivity within an organisation.

What's really interesting here is that this is a real application of expert search technology, being applied not just to satisfy occasional expertise needs ("I'm stuck, who should I ask for help?"), but in daily use to determine work assignments and to increase productivity. A fusion of search technology with constraint optimisation. Tools like these are likely to become invaluable in assigning jobs in global consultancy companies, where managers are unlikely to know everyone at their disposal. Such tools could even be used to identify the best training path for a co-worker to become skilled and productive in a particular area.

Imagine, says Aleksandra Mojsilovic, one of Takriti's close colleagues, that the company has a superior worker named Joe Smith. Management could really benefit from two or three others just like him, or even a dozen. Once the company has built rich mathematical profiles of Smith and his fellow workers, it might be possible to identify at least a few of the experiences or routines that make Joe Smith so good. "If you had the full employment history, you could even compute the steps to become a Joe Smith," she says.

Van drivers have been having their routes assigned automatically for many years. Why shouldn't consultants at IBM be any different? However, Baker points out that some people may be left out by systems (his example, a senior consultant left out because of his high cost, which Takriti counteracts by allowing senior staff members more "time on the bench" than junior staff, because when senior consultants are utilised they get larger cheques). Even still, the concern is this reliance on an expert search system to assign jobs when "expertise relevance" is an even vaguer concept than "document relevance", and expert search systems are not yet (and might never be) as accurate as a travelling salesman solution or a program to optimise a supply chain.

(Via Slashdot)

SIGIR 2008

2008-08-05T09:40:00.000+01:00

We are just back from Singapore, where we have attended the extremely well organised SIGIR'08 conference. We presented one full paper and three posters.

Craig presented our full paper entitled Retrieval Sensitivity Under Training Using Different Measures. Through a large-scale empirical evaluation, the paper addresses an important practical issue, when deploying a search engine, namely whether it matters which evaluation measure is used during training, especially when the available training data is very incomplete. The paper shows among other results that it is not necessarily appropriate to train by directly optimising the target evaluation measure (e.g. MAP) . In particular, the paper shows that bPref, infAP and nDCG are all superior training measures than MAP when the training dataset is incomplete and when the evaluation measure is MAP. Interestingly, the same research question has been addressed by Stephen Robertson, albeit more theoretically, in his keynote talk at the SIGIR'08 LR4IR workshop, where he justified and illustrated why optimising directly the evaluation measure on the training set is not often a good approach (as we say, "Great minds think alike"!).

The Terrier Team also presented three posters at the conference:

Ranking Opinionated Blog Posts Using Opi nionFinder (Presented by Ben): The paper proposes an approach to use and integrate an NLP opinion-identification toolkit, OpinionFinder, into the retrieval process of an IR system, such that opinionated, relevant documents are retrieved in response to a query. This is one of the very few opinion finding detection approaches that were shown to be effective in the TREC Blog Track.

Limits of Opinion-Finding Baseline Systems (Presented by Craig/Iadh): The paper investigates how the underlying baseline retrieval system performance affects the overall opinion-finding performance. Two effective opinion-finding techniques are applied to all the baseline runs submitted to the TREC 2007 Blog track, leading to interesting insights and conclusions.

Automatic Document Prior Feature Selection for Web Retrieval (Presented by PJ): The paper investigates whether the retrieval performance of a Web search engine can be further enhanced by selecting the best document prior feature (e.g. PageRank, URL-Depth, etc.) on a per-query basis. The paper proposes a novel method for selecting the best document prior feature on a per-query basis.

Ps: Photos are from the SIGIR'08 website.

Welcome to the Terrier Team Blog

2008-08-04T20:54:00.000+01:00

It has been a while since we started thinking about having a blog for the Terrier Team. Actually, since we have been involved in the organisation of a TREC blog track in 2006.

Recently, we have been encouraged by the very informative and interesting information retrieval-related discussions, taking place in blogs such as
The Noisy Channel , the Search Engine Caffè, or the
Window Office .

From mere regular readers of information retrieval blogs, we thought that it is now the right time to become more actively involved in blogging. Hence the creation of this new forum, where we intend to post news about our research work and activities. We hope to share our thoughts on information retrieval research, and to engage in a dialogue with our fellow colleagues and friends.

We do hope that many of you will join us in this forum.