Mr Fallon admitted that he touched Ms Hartley-Brewer’s knee during a Conservative Party conference dinner.
But, Mr Fallon insists he apologised for the incident at the time and both consider the matter closed.
A friend of the Defence Secretary said: “Julia’s a good friend of Michael’s. He overstepped the mark when he put his hand on her knee.
Mr Fallon admitted that he touched Ms Hartley-Brewer’s knee during a dinner
SNP investigating sexual misconduct claims after Sturgeon call
MP sex scandal on same scale as ‘EXPENSES’ shock
"She made clear it was unwelcome and he rightly apologised 15 years ago.”
Ms Hartley-Brewer said she threatened to punch the married father-of-two if he carried on his inappropriate behaviour.
She said: “I calmly and politely explained to him that, if he did it again, I would punch him in the face. He withdrew his hand and that was the end of the matter.”
Anti-Brexit protest ahead of Theresa May's Brexit speech
Fri, September 22, 2017
Live photos as anti-Brexit British expats protesting in Florence ahead of Theresa May’s speech
1 of 10
Protesters with placards in Piazza Santa Maria Novella (Santa Maria Novella's Square) for the event of the no Brexit Demonstration, in Florence
The talkRADIO presenter said she did not feel like she was a victim of sexual assault and found the incident nothing more than “mildly amusing”.
It comes after Theresa May vowed to crack down on any MP proven to have acted in a sexually inappropriate manner.
The Prime Minister wrote to Commons Speaker John Bercow yesterday demanding new rules on MP’s behaviour saying the problem “cannot be tolerated any longer”.
Theresa May's most powerful quotes
Thu, August 31, 2017
1 of 8
'I will be ruthless in cutting out waste, streamlining structures and improving efficiency'
Mrs May has asked the Cabinet Office to investigate the conduct of international trade minister Mark Garnier who admitted to calling his secretary “sugar t**s” and sending her to buy sex toys for him.
Former cabinet minister Stephen Crabb admitted to sending “explicit” messages to a 19-year-old woman after he interviewed her for a job.
Other allegations include a current Cabinet Minister who is alleged to have put his hand on a female journalist’s thigh and said: “God I love those t**s”.
Theresa May has vowed to crack down on any MP proven to have acted inappropriately
A second Cabinet Minister is alleged to have had an affair with a junior female MP who is also an MP.
A former minister is accused of propositioning his secretary by asking her to “feel the length of my ****”.
The Guido Fawkes website released a spreadsheet of Tory MPs which includes two Cabinet ministers, 18 other serving ministers, 12 other MPs accused of inappropriate behaviour towards women and four accused of inappropriate behaviour towards men.
Mark Garnier admitted to sending his secretary to buy sex toys
Some of the allegations on the spreadsheet include “handsy at parties,” “impregnated former researcher and made her have an abortion and had other female researchers” and “video exists of three males urinating on him”.
Three Labour MPs and an MP representing another party have also been accused of harassment.
Labour suspended Jared O’Mara MP last week over misogynistic behaviour.