The 2-0 victory over Sunderland came at a cost, however, as Belgian striker Origi was only on the pitch because of a potentially serious ankle injury to playmaker Philippe Coutinho, who left the field on a stretcher with his right leg in a protective splint.
Jurgen Klopp's side lost top spot after last week's goalless draw at Southampton, so it was imperative for their title ambitions they found another way of winning after blowing away many teams already this season.
Sunderland by no means made it easy for them and frustrations were beginning to grow until Origi, getting a rare chance because a calf injury ruled out Daniel Sturridge, struck in the 75th minute.
Milner's 90th-minute penalty made the result safe and put an end to the visitors' mini-revival after back-to-back wins had lifted them off the foot of the table.
Former Everton and Manchester United manager David Moyes' poor record at Anfield, where he has drawn seven, lost six and never won, meant his side were never going to be anything other than compact and aiming to be hard to beat.
They conceded 78 per cent of possession but in the opening 45 minutes managed to limit their hosts to a handful of half-chances as Georginio Wijnaldum's early shot on the turn was easily saved, Dejan Lovren directed two late efforts, one a header, off target and Sadio Mane fired straight at Jordan Pickford.
Part of Sunderland's success in negating the threat of opponents who had scored 17 times in five previous home league matches was Jason Denayer's man-marking job on playmaker Coutinho.
Wherever the Brazil international went when Liverpool had the ball Denayer was always in his shadow.
So it was ironic that the one occasion the on-loan Manchester City defender lost his man resulted in Coutinho's injury, as Didier Ndong's follow-through caught the right foot of the forward who was trying to jink past him.
Sunderland's one chance before the break fell to former Everton midfielder Steven Pienaar but goalkeeper Loris Karius just got out quick enough to block.
In the second half Wijnaldum bundled a volley wide, Roberto Firmino was denied by Pickford at his near post and Emre Can missed the target as Sunderland became even more defensive, with at least a couple of Liverpool attacks seeing every player but Karius in the final third.
When Firmino flashed a cross through the six-yard box neither Origi nor Mane could apply the finishing touch, but their frustration finally ended 15 minutes from time.
Jordan Henderson picked out Origi on the left of the penalty area and he managed to thread a shot across a crowded box and inside the far post.
When a charging Mane was brought down by Ndong in added time, Milner stepped up to score his fifth of the season, affording Klopp the luxury of handing 17-year-old academy graduate Ben Woodburn a debut.