Anthony Albanese has been accused of failing to act after one of his party members was charged with two counts of assault.
Opposition MPs have called on Labor's Luke Gosling to step down from his special envoy role after he allegedly assaulted NT Administrator David Connolly earlier in July.
The member for the Darwin-based seat of Solomon, 54, has denied the allegations.
But the prime minister should have taken Gosling out of his role as special envoy for defence, veterans' affairs and northern Australia, Opposition leader Angus Taylor said, accusing his counterpart of having "failed".
“This is a test of leadership for him (Anthony Albanese), and when it really matters, whether it’s on the economy, whether it’s on protecting our way of life, when it really counts, the prime minister is nowhere to be seen,” Mr Taylor told reporters.
"This is a very serious ... situation, and arrest charges (are) being laid against the most senior official in the Northern Territory. The prime minister needs to step up.”
Mr Albanese is on leave, with deputy prime minister Richard Marles acting in his place.
Nationals senator Bridget McKenzie also said Gosling should stand aside from the special envoy role until legal proceedings had concluded.
"It's an important part of our democracy of keeping integrity within our Westminster system that the prime minister does do the right thing and stand Mr Gosling aside until his name is cleared or until the matter is dealt with in court," Senator McKenzie told News 24.
Liberal senator James Paterson on Thursday also said he hoped Gosling would move aside.
Gosling's bail mention is due to be heard in Darwin Local Court on Friday afternoon ahead of the case's next mention on September 1. He has denied the allegations.
The attorney-general Michelle Rowland would not be drawn on whether her colleague should move aside from his role.
"As you can appreciate, because the matter is before the court, I'm limited in what I can say," she told Channel Nine.
"The commentary around this matter needs to be handled very seriously."
Northern Territory senator Malarndirri McCarthy told ABC Radio she was upset by the news, as she had found Gosling to be a "very good and decent person".
She said conversations about whether he should step down would need to be between Gosling and the prime minister.
The alleged incident reportedly occurred at the NT Thailand Grand Festival in Darwin on July 18, with Gosling posting photos from the event on social media that day.
Gosling, a retired army officer who has represented Solomon since 2016, released a statement after his arrest.
''These are ridiculous allegations that I categorically deny," he said in the statement.
"I’ve had no dealings with the administrator since congratulating him following the announcement of his appointment late last year."
