Peter Vincenti's winner at Rotherham put Rochdale into the FA Cup third round for the first time since 2005.
Lee Frecklington seized on a loose pass and slotted past Josh Lillis to put the League One side ahead early on.
Kieran Agard was twice denied by goalkeeper Lillis before Rochdale, fifth in League Two, recovered.
Matty Lund turned in Michael Rose's cross to level and the Millers had Michael Tidser sent off before sub Vincenti converted Joe Bunney's centre.
Tidser received his second yellow card 16 minutes from time and Rochdale, who had improved in the second period, were able to make the advantage of an extra man count.
Home keeper Adam Collin kept out efforts from Scott Hogan, Jack O'Connell and Rose either side of the midfielder's dismissal.
Vincenti had been brought on as a first-half substitute for centre-back Oliver Lancashire as visiting manager Keith Hill changed formation following a poor start to the match.
Ben Pringle curled a late free-kick wide as 10-man Rotherham tried in vain to earn a replay.
VIEW FROM THE DRESSING ROOM
Rotherham boss Steve Evans told BBC Radio Sheffield:
"Obviously I'm upset. I don't think we got going in any stage of the match but despite that we got in front.
"In the second half we were trying to find a bit of play and struggling.
"The game-changer moment to go out of the cup was down to Michael Tidser getting sent off."
Rochdale manager Keith Hill told BBC Radio Manchester:
"It was a very good response to a very bad first 25 minutes of the game. We could have lost the game in that first 25 minutes and that would probably have been down to ourselves.
"Oliver [Lancashire] has been unbelievable for me, and he'll be a brilliant player for us, but I had to make the change early.
"Our game plan didn't work, so I had to change it. And I changed it so that we could win the game.
"This wasn't about the league - this was about me trying to earn money for the football club. I thought we were business-like and I made a business-like decision in the early stages."
Rotherham United 21 Collin 14 Bradley 03 Skarz 08 Frecklington Booked 20 Morgan Booked 04 Arnason 02 Brindley Booked (Davis - 76' ) 23 Tidser Dismissed after an earlier booking 22 Agard 09 Revell 18 Pringle Substitutes 01 Shearer 05 Davis 07 Worrall 31 Rowe Rochdale 01 Lillis 03 Bennett Booked 25 Rose 22 O'Connell 06 Lancashire Booked (Vincenti - 22' ) 05 Eastham 12 Lund Booked 38 Dicker 11 Hogan (Allen - 86' ) 40 Henderson 16 Done (Bunney - 78' ) Substitutes 04 Cavanagh 07 Vincenti 08 Tutte 21 Thomson 22 Rafferty 24 Allen 39 Bunney
Ref: Graham Scott