Prime Minister Rishi Sunak refused to commit to building the HS2 rail project’s northern leg, fueling speculation that he plans to scrap a major part of the government’s plans to “level-up” disadvantaged regions.

“There are spades in the ground right now at the moment, making sure we complete the first bit of this line,” Sunak said Thursday during interviews on local BBC radio, referring to the London-Birmingham link. He repeatedly avoided answering on whether the second part from Birmingham to Manchester will still go ahead. Another spur to Leeds was scrapped in 2021.

The premier’s refusal to squash speculation that he’s considering shortening the rail project, whose costs are threatening to spiral beyond £100 billion ($122 billion), risks sparking more anger from UK business leaders and politicians across the spectrum. Asked whether he agreed with Conservative West Midlands Mayor Andy Street that the project should go ahead in full, Sunak said only that he agreed on the need for transportation infrastructure.

The controversy makes for an awkward backdrop ahead of the Conservative Party’s annual conference starting Sunday in Manchester. The gathering is a chance for Sunak to outline his vision for Britain and boost his party’s flagging poll numbers against Labour before an election expected to be held next year.

Read More: UK’s HS2 Rail Project Sets Chaotic Scene for Party Conference

The HS2 project has been backed by Sunak’s five immediate predecessors as prime minister and is at the heart of the Tories’ manifesto promise to address imbalances between northern communities and the more affluent south. The Tories drew away traditional Labour voters in those areas in the 2019 election, and Sunak is under pressure to hold onto that support.

During the radio interviews, Sunak gave a clear view of how the government would likely try to sell any scaling back of HS2 as a benefit to the region. “East-west, west-east connectivity across northern towns and cities is important,” Sunak said, implying money saved on HS2 would free up budgets to expand other lines. He also referenced improving roads and repairing potholes.

The politics are sensitive. Sunak also refused to confirm that HS2 would be built to the planned terminus at Euston in central London, amid reports that the government is also seeking cost savings by stopping the line at Old Oak Common in the west of the capital. Asked about Euston, Sunak replied that connections from Old Oak Common are “very strong” because the station will be on the new Elizabeth Line tube route, which used to be known as Crossrail.

But critics have said that stopping the high-speed route outside central London undermines the business case for the flagship line, and there are warnings that even the new Elizabeth Line may struggle to cope with the volume of HS2 passengers transferring to trains coming from Heathrow airport.

It’s not just train users that Sunak risks alienating. Thousands of people have seen heavy disruption and construction in their local areas to make room for HS2, which was promised as a new backbone for the UK’s transport infrastructure but now risks becoming dramatically watered down. Most of those areas don’t have stations on the new line, though the business case for HS2 includes freeing up capacity on existing lines.

During his interview with BBC Radio Three Counties, Sunak was played a clip from a man who has lived in Wendover, northwest of London, for 72 years. He said graveyards and woodland have been dug up near his home. “I know what Wendover was like, and it won’t be the same again. Never,” he said.

Tory party grandees including former Prime Minister Boris Johnson, ex deputy premier Michael Heseltine and former Chancellors George Osborne and Philip Hammond have urged Sunak not to scrap the Manchester link. Labour’s Mayor of Manchester Andy Burnham said it would leave northerners with “Victorian infrastructure, probably for the rest of this century.”

The government has already spent £24.7 billion on HS2 as of June, with a budget for the first phase of the project from London to Birmingham of as much as £45 billion. Beyond that the numbers are more opaque. The official cost of the whole project is £71 billion — though the government’s own review said it could exceed £100 billion.

--With assistance from Stuart Biggs.

(Updates with Sunak comment in sixth paragraph)