Theresa May is under mounting pressure to sack Damian Green after a Cabinet minister and a Downing Street aide urged her to axe her deputy.
It comes after David Davis has made clear he is ready to quit if Mr Green is sacked over police 'smears' about pornography being found on his Commons computer during a 2008 raid.
The Brexit Secretary is understood to be furious at apparent efforts by former police officers to settle long-standing scores by forcing Mr Green out.
Yet a Cabinet source told the Times: 'Damian might have survived if this was dealt with immediately but the longer this is drawn out, the more there is a sense this is unsustainable. There are now so many allegations out there, it is hard to work out how they will be dealt with.'
Another Tory source said at least one Downing Street senior aide believed that Mr Green would have to be axed or resign.
The source said: 'There are senior figures in No 10 who think he should resign.'
It comes after the detective who seized Mr Green's computer nine years ago said thousands of images were found and there was 'no doubt whatsoever' the MP himself had been accessing them.
The raid was highly controversial at the time, as Mr Green - then shadow immigration minister - had been embarrassing the police with a series of leaks.
Accosted by reporters as he left his constituency HQ in Ashford today (pictured), Mr Green again flatly denied viewing any pornography on his office computer
Neil Lewis (pictured left), former Scotland Yard detective who examined Damian Green's work PC in 2008, said today it contained thousands of pornographic images and says there is 'no doubt whatsoever' it was his. Former Met assistant commissioner Bob Quick (right) was in charge of the leak inquiry that triggered the raid
Mr Lewis' notebook shows how pornography had been found on the computer - he also says there was more on the work laptop
Mr Davis was shadow home secretary until a few months before the raid on Mr Green's Commons office in November 2008, and shared the view of many politicians that the police action flouted parliamentary privilege rules
Mr Davis was shadow home secretary until a few months before the raid in November 2008, and shared the view of many politicians that the police action flouted parliamentary privilege rules.
A source close to Mr Davis said today: 'It is right that allegations of misconduct towards individuals are properly investigated but police officers have a duty of confidentiality which should be upheld.'
A mutual friend of Mr Davis and Mr Green told the Evening Standard the Brexit Secretary had 'put his cloak around' his colleague and made his position clear to the PM 'in words of one syllable'.
Meanwhile, the deputy PM is facing a second sleaze inquiry after it emerged MPs are required to sign an 'acceptable use policy' for computers at parliament.
The Commons standards commissioner could stage a separate investigation to the ongoing probe by the Cabinet Office, which is due to report any day. Theresa May, who has known Mr Green since their days together at Oxford University, will have the final say over whether the First Secretary stays in his job.
In the latest wave of allegations against Mr Green, former detective Neil Lewis disclosed his notes from the raid and claimed that on some days porn was browsed and open on the computer for hours.
He told the BBC: 'The computer was in Mr Green's office, on his desk, logged in, his account, his name.
'In between browsing pornography, he was sending emails from his account, his personal account, reading documents... it was ridiculous to suggest anybody else could have done it'.
He added: 'I was surprised to see that on a parliament computer. I had to take a step back because I wasn't expecting that'.
The Tories put out footage of the 2008 raid on Damian Green's office
Damian Green and Bob Quick crossed swords in 2008 when the Met assistant commissioner took dramatic action in an inquiry into leaks from the force.
Mr Quick decided to arrest then then shadow immigration minister.
The Tory MP was held for nine hours while his Commons office, two homes and constituency office, were searched and computers removed by counter-terrorism officers.
The episode sparked a huge inquest at the Commons into whether parliamentary privilege should have protected the material held by an MP.
In the ensuing political storm, it emerged Mr Quick's wife was running a car hire firm from their home and details of their address were published on a website.
The officer then accused the Tories of being 'wholly corrupt' in leaking the story to intimidate him and his investigation.
But he was forced to apologise after then party leader David Cameron said the claim 'Tory machinery' was mobilised against his investigation was 'completely baseless'.
Mr Green was later cleared of any wrongdoing. Mr Quick complained bitterly that the investigation cost him his career.
He quit in April 2009 after being photographed arriving at Downing Street with documents detailing a counter-terror operation clearly visible. He later claimed he might have survived the gaffe if it had not been for the Green affair.
The police officer, who worked as a lead in digital crime after being a leading counter-terrorism officer, was working on Operation Miser in 2008.
This was an investigation into Home Office leaks that led to Mr Green's Commons office being searched by police and his computers being analysed.
The raid sparked an outcry that targeting Mr Green - shadow immigration minister at the time - was a flagrant breach of parliamentary privilege rules.
The leak inquiry was being led by Met assistant commissioner Bob Quick, who has also raised allegations about the pornography find. The claims have dramatically widened the Cabinet Office inquiry, which was originally looking at accusations Mr Green made a clumsy pass at a Tory activist.
But a host of Conservative MPs have accused the police of pursuing an anti-Tory witch-hunt against Mr Green that was 'straight out of the mafia playbook'.
Tory MP Crispin Blunt suggested that as a shadow minister, Mr Green would have been too busy to look at porn online while at work.
He said: 'The account that Neil Lewis has given of the use of the computer, based simply on his notebooks from some time ago... on his own account doesn’t bear the slightest relation to the kind of life a member of parliament leads.
'We don’t have time to sit in our offices breezing through leisure websites of whatever type — we largely move from one meeting to the next.'
Former Cabinet minister Andrew Mitchell, an ally of Mr Green, said the 'hounding of Mr Green over information which everyone is clear was entirely legal and which he has emphatically denied either downloading or viewing is completely wrong'.
'Now, nine years later, after a pretty contentious raid of a senior