Alexander pounced in the six-yard box on 62 minutes after Chris Dickson had notched his fourth goal in a week for Gillingham.
The Gills, buoyed by their terrific win over Luton in midweek, began the game in dominant fashion against a Millwall side short on confidence.
The new front pairing of Delroy Facey and Dickson gelled from the start and it was they who combined in the 16th minute to give the home side the lead.
Facey's well-directed flick reached the path of Dickson, who neatly touched the ball over Paul Robinson before volleying past Rab Douglas from ten yards.
But Millwall came fighting back and looking to expose Gillingham's fragile defence, with both Alan Dunne and Will Hoskins missing half-chances.
The same pair were both guilty of glaring misses before the break, failing to connect with headers from pinpoint crosses when it appeared easier to score.
The opening stages of the second half brought a flurry of chances as Andy Frampton's free-kick was smartly turned away by Simon Royce, while at the other end Dickson saw his side-footed effort rebound agonisingly off the far-post.
That miss was punished in the 62nd minute when Alexander flicked the ball over Royce at the near post and over the line from close range, with Dunne making sure from under the crossbar.
The Lions went in search of a winner and should have been celebrating when, with five minutes remaining, Hoskins' deflected shot reached Dunne, but the midfielder fired wide with only the keeper to beat.
Gillingham's Ian Cox had a late header comfortably saved by Douglas as both sides settled for a point that does little to improve their league positions.