The Ministry of Justice has rejected an offer of £24.1m from G4S which the security firm says it owes after over-charging for the tagging of offenders.
The company made the offer after admitting the way it billed the UK government was "not appropriate".
But its internal review also concluded it was not the result of "dishonesty or criminal conduct".
It comes after an audit suggested the firm had been charging to tag criminals who were either dead or in jail.
The BBC understands the government is trying to establish with G4S what it is owed.
Meanwhile, a National Audit Office report has revealed that concerns about practices at the company were made in March by a G4S whistle-blower working at a call centre that handled calls over the firm's tagging.
In July, the Ministry of Justice (MoJ) revealed that G4S and a second security company, Serco, had overcharged the government by "tens of millions of pounds" on their electronic tagging contracts.
The audit by accountancy firm PricewaterhouseCoopers - which had been launched in May - alleged the charging discrepancies had started at least as far back as the start of the current contracts - in 2005 - but could have dated back to the previous contracts in 1999.
G4S said on Tuesday that, following an independent review of its tagging contract by law firm Linklaters, it now accepted it had "wrongly considered itself to be contractually entitled to bill for monitoring services when equipment had not been fitted or after it had been removed".
It said it had "issued credit notes totalling £23.3m for amounts incorrectly billed between 2005 and May 2013".
"A further credit note of £0.8m will be issued for billings for the period from June 2013 to date," it said.
G4S group chief executive Ashley Almanza said the company's offer was "an important step in setting this matter straight and restoring the trust which has been earned by our many thousands of committed and hard-working employees over many years".
"The way in which this contract was managed was not consistent with our values or our approach to dealing with customers. Simply put, it was unacceptable and we have apologised to the MoJ."
The company had changed its leadership and was putting in place enhanced risk management and contract controls, he added.
Mr Almanza also said the firm remained "committed to working with the MoJ and the UK government to resolve this matter", and was ready for further negotiations if the audit concluded that it had overcharged by more than the sum offered.
Commenting on Serco's position, the NAO said: "Serco has stated to us that it considers it charged in line with its genuine interpretation of the contract and that it was open about this to the Ministry [of Justice] throughout. Serco has stated publicly that it will refund any agreed overcharges."
'All possible avenues'
The Serious Fraud Office has launched an investigation into G4S and Serco, and both firms have said they are "co-operating fully".
Labour MP Keith Vaz, chairman of the Home Affairs Select Committee, said the company's admission was "staggering".
"The company refused to co-operate with the government, which led to the referral to the SFO, and that should have rung alarm bells. It represents serious corporate failure.
"Contracts of this kind need to be conducted in good faith and represent value for money. Every G4S contract with the government needs to be reviewed immediately, and they should be banned from bidding for any more."
The government said it could not comment as the matter was the subject of a criminal investigation.
On whether it would accept money from the firms, an MoJ spokesman said: "The secretary of state has been clear, we are determined to secure a refund for the taxpayer - we have taken appropriate legal advice and will pursue all possible avenues."
Examples of billing for tagging from the auditors' review in May:
- G4S was told by the Met Police on 14 September 2011 that one of its offenders had been sentenced to two years in prison. The company removed its monitoring equipment, but by 20 May 2013 the court had not provided the relevant cessation paperwork. At that date the total time charged without monitoring equipment installed was 612 days - at a cost of approximately £3,000
- On 28 October 2010, G4S suspended visits to an address where a number of breaches of curfew had been reported and the equipment was removed. A few days later it told the court the equipment was no longer required, chasing it for written confirmation in December 2012. There was no response so billing continued. The total time charged without electronic monitoring equipment installed as of 20 May 2013 was 935 days at a cost of approximately £4,700
- Serco was unable to install equipment at a property in July 2008 and police later said the subject was due to be arrested. In 2009 Serco attempted twice to obtain details regarding the case but was not successful. As part of a wider review Serco attempted to visit the property in October 2010 but was informed that nobody had been living there for 18 months. Serco lodged a further request for information on bail status with the court in April 2012, but no response has been received. Serco has billed for almost five years at an approximate cost of £15,500