The hosts dominated proceedings from start to finish but failed to break the deadlock until midway through the second half despite former Lions striker Barry Hayles being shown a straight red card for his late and dangerous clash with Millwall skipper Paul Robinson.
Man of the match Lewis Grabban was a constant thorn in the Robins side and could have put the Lions in front on 18 minutes having lashed a curling left-foot effort just inches over having turned neatly away from Alan Wright on the edge of the Cheltenham box.
The games turning point came on 26 minutes when a laboured challenge from Hayles on Robinson, who had clearly won the header, saw the on-loan striker controversially shown the red card on his return to the Den.
Ten-man Cheltenham dug deep and a curling left-foot effort from Alex Russell from the edge of the box rattled David Forde's right-hand post.
The visitors carved out the best chance of the game ten minutes after the interval when Scott Murray hooked a cross in to the danger area that Paul Connor powered goalwards, only to be denied by a match-winning wonder save from Forde at point-blank range.
Having had one penalty appeal waved away 60 seconds earlier, Alexander finally got his reward when referee Jon Moss pointed to the spot after Gavin Caines had hauled the goal machine down.
The Kennington-born striker stepped up himself to send keeper Shane Higgs the wrong way and fire the south Londoners in front on 68 minutes.
With the game opening up, substitute Adrian Forbes David Martin into space down the Lions left and with debutant Drissa Diallo backing off, the former Dartford winger let fly a screamer.
His left-foot effort nestled in the very top corner giving Higgs no chance and he raced towards the wowed Lions faithful.
Martin Allen's side refused to give up and ran themselves into the ground almost earning themselves a late consolation goal as Dialla's header was hacked off the line by Ali Fuseini inside the last ten minutes.