Claims the government could delay the UK's departure from the EU have been described as "idle chatter" by a senior Brexit campaigner.
John Redwood said PM Theresa May wanted to "get on with it" and trigger Article 50 of the Lisbon Treaty - which officially begins Brexit.
Mrs May has said she will not invoke Article 50 of the Lisbon Treaty this year.
It was widely thought it would be triggered at the start of next year.
Earlier Downing Street did not comment on press reports Article 50 might not be triggered in 2017.
Following the UK's vote to leave the EU, some Leave campaigners have been calling for Article 50 - which starts a two-year Brexit process - to be triggered immediately.
Mrs May, who is currently on holiday in Switzerland, has said this will not happen this year, and Brexit Secretary David Davis has predicted the trigger will take place "before or by the start of next year".
Asked about reports in the Sunday Times that the planned timetable could be delayed, the prime minister's spokesman repeatedly said "Article 50 won't happen before the end of 2016" but refused to comment further.
If Article 50 did not happen in 2017, it could mean the UK did not actually leave the EU until 2020.
Downing Street said Brexit was "a serious and very complex task".
"Brexit is a top priority," the PM's spokesman said, "it is important we deliver it and make a success of it."
Conservative MP and veteran Eurosceptic Mr Redwood told the BBC he did not "buy" reports of a delay, saying they were "idle chatter by people who don't know the prime minister's mind".
He said his understanding was that Mrs May did not want the issue dominating the next Parliament and wanted to "get on with it".
Brexit, he added, "can happen pretty quickly".
Over the weekend, former UKIP leader Nigel Farage said he would consider returning to the political frontline if Brexit was not delivered, tweeting that he was "starting to detect a lot of frustration" from people who had voted Leave.
It comes after a reported turf war between two of the ministers responsible for making Brexit happen - Foreign Secretary Boris Johnson and International Trade Secretary Liam Fox.
According to a letter obtained by the Sunday Telegraph, Mr Fox launched a bid to take over some of the powers of Mr Johnson's department.
Mr Fox wrote to Mr Johnson to say responsibility for economic diplomacy should be moved from the Foreign Office to his International Trade Department.
Responsibility for delivering Brexit is split between three cabinet posts and departments: Mr Johnson at the Foreign Office, Mr Fox at International Trade, and Mr Davis, Secretary of State for exiting the European Union.
The latter two departments were created in light of Britain's vote to leave the EU in the referendum on 23 June.
In his letter, Mr Fox suggested that it would be "rational" to hand control of the Foreign Office's overseas economic policy to his department so that it could "take clear leadership of the trade and investment agenda".
He said it had become "clear" to him since taking up his new post that "existing cross-Whitehall structures have meant that HM Government has not taken the holistic approach it might have" in this area.
"For British trade to flourish there are clear requirements on us: to build the trade framework in markets through trade policy, to establish the conditions for trade and investment through foreign diplomacy, and to reap the harvest of those efforts," he said.
"The role played by economic diplomacy is crucial to delivery of the objectives I have been set by the prime minister as international trade secretary and I believe it is imperative that this capacity is fully aligned with government resources delivering trade and investment for the UK.
"If we are to have a rational restructuring I think there is a reasonable proposition to transfer the economic diplomacy function from the Foreign and Commonwealth Office into the Department for International Trade."
He said his proposal would enable the Foreign Office to "retain clear leadership on diplomacy and security", including oversight of the intelligence agencies.
Responding to the Sunday Telegraph story, a Foreign office spokesman said: "We don't comment on leaked documents.
"Alongside other Departments, the FCO are seconding a small number of staff with relevant expertise to the new Department for International Trade.
"This is all part of the cross-government effort to ensure we make a success of Brexit."
Labour said "turf wars" were an inevitable consequence of Brexit policy being split between various departments.
Shadow Foreign Secretary Emily Thornberry said Theresa May "created these three separate departments not because it made sense in terms of coordinating Whitehall's management of Brexit, but just to buy the loyalty of Liam Fox, Boris Johnson and David Davis".
She added: "Now we see the inevitable dysfunction and turf wars, as they fight between them to grab work that should have been co-ordinated out of the Foreign Office alone.
"It is high time that the Tories stopped fighting amongst themselves, started thinking about the good of the country, and focused on clearing up the mess that they have created."
Conservative MP and former Foreign Office Minister Alistair Burt said the Foreign Office should retain responsibility for "all elements" of foreign policy, including its trade and economic policy, which he said the department took "very seriously."
Mr Burt told BBC Radio 4's The World At One it was "inevitable" there would be tensions as new departments tried to establish themselves.
He said Mr Johnson, Mr Fox and Mr Davis did not have a track record as "team players" but that they now needed to "put those personalities together".