AI-powered search engine makes S’pore Parliament debates more accessible

An artificial intelligence (AI)-powered search engine has been launched that makes combing through decades of parliamentary records more fruitful for the public, and which could raise understanding of how issues evolve in Singapore’s top law-making body.

Called Pair Search, the website search.pair.gov.sg is powered by a large language model (LLM) – the same technology behind ChatGPT – and also lets users sift through case judgments from the High Court and Court of Appeal.

Early results from test users of the prototype search engine have been encouraging, said Open Government Products (OGP) senior software engineer Oh Chin Yang.

OGP is an experimental development arm of the Government that builds technology for the public. The search engine project was conceived at its annual hackathon, as the team felt recent strides in natural language processing and information retrieval could make for a more effective search tool for both public officers and citizens, said Mr Oh.

“The best public policy decision and debate is an informed one, and that starts with search,” he told The Straits Times.

Based on data since it was soft-launched at the start of 2024, 84 per cent of all users who clicked on at least one search result found their searches within the first 10 results shown to them, said Mr Oh, who is one of six members of the Pair product team.

As at April, there have been approximately 2,700 users on Pair Search for parliamentary records, with a combined total of 8,300 search queries, he added.

While there is an existing search platform for Hansard – the official record of debates in Parliament – it is “difficult and unintuitive” to use, said the Pair Search team on the OGP hackathon’s website.

Results were poor as the existing search engine is fully keyword-based, which meant that debates in the House that frequently mentioned a single word in a query were often ranked more highly than more relevant results, it added.

For instance, the team found that searching for “Covid-19 rapid testing” using the Hansard search engine led to eight irrelevant documents among the top results.

All of the top 10 results generated by Pair Search were relevant to the query.

The higher hit rate is due to Pair Search’s use of a combination of advanced keyword matching and contextual search that understands a user’s intent. This means it can even generate relevant results when there are no matching keywords, said the team.

Apart from more relevant search results, the new tool also gives users snippets of each result, like any modern search engine. It can also sort search results by relevance, and not just chronological order.

Long road to fully accessible parliamentary records

While records of speeches and debates in Parliament all the way back to 1955 (when it was known as the Legislative Assembly) can now be freely searched on any internet-connected device, this used to be onerous and costly.

In 1996, Parliament sitting reports began to be put on the internet, but users had to register an account and pay $8 a month to access them through LawNet, a legal research portal.

This changed in 2000 when Hansard reports became freely accessible online, with a search engine reflective of the time, said the spokeswoman for the Office of the Clerk of Parliament. 

However, then Nominated MP Braema Mathi noted in 2004 that the Hansard was limited to reports from the current term of Parliament. Meanwhile, costs to access earlier records had risen to $30 per half hour, excluding goods and services tax.

She called on the Government to make more of the Hansard available, and free for Singaporeans.

Then Leader of the House Wong Kan Seng responded that paying for access to LawNet was not new, and that members of the public who did not want to do so could read the Hansard at the National Library.

In 2009, then Tanjong Pagar MP Baey Yam Keng noted that the Hansard search was still limited to the current term, which was unthinkable for a nation with a leading e-government system.

“I am not sure if the restricted access to our Hansard is a symptom of our governmental system where everything is tied to key performance indicators and cost-benefit analysis,” he added.

In 2011, a Today forum letter urged more thorough access to parliamentary records, given the upcoming general election.

The Government said it would consider the feasibility of doing so in the next term of Parliament, after the election. The full archives eventually became freely searchable online.

The Parliament spokeswoman told ST that the Hansard search engine has gone through refinements prompted by technological advances and user feedback, and moved to its existing platform in 2018.

Improvements over the years include expanding search options to particular terms of Parliament, and for specific parliamentary business items such as Bills and ministerial statements.

MPs have continued to ask for a more feature-rich Hansard in recent years. In 2020, Bukit Batok MP Murali Pillai called for a new information technology (IT) system that would record the outcome of MPs’ proposals that ministers had agreed to study.

“The Hansard is replete with examples of frontbenchers providing holding replies without the ability to check whether there have been any updates,” he said. “Members of the public reading the Hansard will not be able to tell whether the loop has been closed.”

He noted that voters were “maturing, more demanding, and very rightly so, in terms of what they can and should expect from all of us in this House”.

New search engine part of public service AI software suite

While still a prototype, the Pair Search platform is continually being improved, said Mr Oh.

He noted that the search algorithm was last enhanced on April 24 to better understand parliamentary debates at both the paragraph and report level, the latter typically covering a full day’s sitting.

This means that reports that may not be relevant as a whole, but with sections that are highly relevant to the search query, will be ranked more highly, resulting in better search results.

The infrastructure cost incurred since Pair Search’s inception at the start of 2024 has been less than US$2,000 (S$2,700) a month, added Mr Oh.

The Pair Search team comprises one engineer, supported by a designer, and a product manager from the broader Pair team.

Besides search, the Pair team works on related LLM-based projects for public officers, which include Pair Chat (a secure version of ChatGPT), Pair Intern (an e-mail-based LLM assistant), and Pair Noms (an automated meeting minutes generator).

Only Pair Search is available to the wider public.

Testers of Pair Search currently include staff from the Attorney-General’s Chambers, and those who frequently search the Hansard, such as legal policy officers from the Ministry of Law and communications operations officers.

One civil servant, who did not want to be named, said she preferred the test results on Pair Search as they allowed her to preview each search hit, saving her time.

But another said he was unsure about the relevance of its search results, as the prototype had called up some hits that he was not looking for. For now, he would keep to the original Hansard search engine for his work.

The Pair Search team said its next goal is to turn the simple search engine into a more powerful research tool. For instance, the platform could proactively recommend relevant content based on each user’s search patterns.

It is also building more features, such as a summary function that uses AI to pull the best results into a short explanatory paragraph based on the search keywords used.

As to whether there are plans to merge the two search platforms, run them simultaneously, or sunset the older one, the Parliament spokeswoman said this has not been decided, given that Pair Search is still a prototype.

“Future plans will be subject to developments in the IT landscape,” she said.

Source: Straits Times © SPH Media Limited. Permission required for reproduction.

Gender stereotypes mean flexi-work rules can turn into boon or burden for women ADV: Publication: Data Embassies: Purposes, Features and Limitations