The Labour Party in England has condemned the decision not to remove the whip from the former minister.
However, Mr Spencer said the police must carry out their investigation before “we can assess where we’re at”.
“They are very serious allegations and we do take those allegations very seriously,” he said.
“I think it is down to the police to do that thorough investigation, not for the Whips Office to investigate this alleged crime, it is for the police and the authorities to do that.
“Once they’ve come to that conclusion, then we can assess where we’re at and the position that the MP find themselves in.”
The Metropolitan Police received allegations on Friday of sexual offences and assault relating to four incidents at addresses in London, including in Westminster.
The force said a man was arrested on Saturday on suspicion of rape and taken into custody at an east London police station. He has been released on bail to a date in mid-August.
Shadow safeguarding minister Jess Phillips criticised the Tories for failing to suspend the whip from the MP.
The Labour MP told Times Radio it sent a “terrible message” that senior figures are still able to secure “protection” with their Westminster status.
“While pending a police investigation for a sexual crime, I think it is only right that the whip is withdrawn,” she said.
The investigation comes days after former Tory MP Charlie Elphicke was convicted at Southwark Crown Court in a separate case of sexually assaulting two women.
Elphicke had the whip removed in November 2017 when the allegations first surfaced, but it was controversially restored before a confidence vote in Theresa May’s leadership the following year.
However, when Elphicke was formally charged in July 2019, the party suspended him again on the same day.