Jamie Elliott scored two tries on his Heineken Cup debut for Saints
Last year's finalists Northampton crashed out of the Heineken Cup after an ill-disciplined defeat at Castres.
The sides scored four tries each in an action-packed encounter but Saints lost Stephen Myler and Tom Wood to the sin-bin either side of half-time.
Giant wing Romain Martial scored twice for the hosts and fly-half Romain Teulet landed eight out of nine kicks.
Teenager Jamie Elliott scored a brace of tries for Northampton, who have now lost all three of their pool games.
Northampton were without
suspended duo Chris Ashton and Phil Dowson
and their back line had an unfamiliar look with Wood, Ben Nutley and Samu Manoa all handed starts, while Steven Myler was preferred to Ryan Lamb at fly-half.
NORTHAMPTON - HEINEKEN CUP RECORD
Failed to qualify
Failed to qualify
Failed to qualify
Saints coach Jim Mallinder asked his side to start quickly but it was Castres, bottom of the pool at kick-off, who had the lead inside five minutes.
Ibrahim Diarra broke the Saints line in midfield with a powerful run and Castres moved it out to the right where Martial freed his arms brilliantly to offload in the tackle and feed Steve Malonga to go in at the corner.
Saints provided the perfect response inside 10 minutes after Castres were penalised for collapsing the scrum, and Saints turned down the three-point opportunity to kick for the corner.
After a neat catch and drive down the left, first Soana Tonga'uiha and then Mark Sorenson made a push for the line, and Sorenson eventually forced it over the line with help from Wood.
Myler added the conversion to level the scores but it was Castres who immediately went back on the front foot, again attacking down Saints' left.
Saints captain Dylan Hartley was penalised for coming up in the scrum and the reliable Teulet made no mistake from the resulting penalty to put his side back in front.
Saints came back once more and they should have been ahead when Foden was released down the left, but he came inside instead of feeding it wide and the attack lost momentum.
And they were made to pay when Teulet slotted over another penalty following Steven Myler's high tackle on Diarra.
NORTHAMPTON - NEXT THREE FIXTURES
Northampton won possession from the restart and George Pisi made a neat break down the hosts' right. The new-look Saints back line worked it neatly across to teenager Elliott, who went in at the corner to score on his Heineken Cup debut.
Myler kicked just wide from the conversion but Teulet made no mistake from a penalty shortly afterwards, awarded for offside against Tom May.
And Northampton were dealt a blow five minutes before the break when Myler was yellow-carded for a second high tackle.
Castres made their one-man advantage pay in the short time before the interval, Chris Masoe chipping through for Martial to touch down, before Teulet converted to give his side a 23-12 advantage at the break.
The visitors were handed a gift three minutes into the second period when Diarra's speculative pass out wide to Malonga was intercepted by Jamie Elliott, who broke away down the left to reduce the deficit.
And although Teulet kicked another penalty after Tonga'uiha was penalised in the scrum, it was Saints - buoyed by the return of Myler from the sin bin - who began to dominate.
Their pressure told when Pisi and Foden broke down the right, and scrum-half Lee Dickson fed Samu Manoa to go in at the corner, and it looked as though Saints would complete the comeback.
But they lost momentum when Wood was shown yellow by Lacey for persistent offences at the breakdown just after the hour mark.
The impeccable Teulet kicked the penalty and the extra man again told for Castres as they forced persistent pressure before eventually going over again through Martial's second try, after good work from substitute Pierre Bernard and Teulet.
Diarra then sealed a bonus-point win for the home side after another impressive break from man-of-the-match Martial.
Northampton director of rugby Jim Mallinder:
"When you have two yellow cards you have to ask why?
"I don't want to question the referees but we have had the same problem against Munster and we have to look at why.
"The yellow card for Steve Myler and a high tackle was a close one but we just aren't getting those decisions at the moment.
"We are out of the competition and that is hard and we are playing for pride now.
"Next week is all about that and we will work hard and go out and play hard like we always do in all competitions. That won't change.
"We had some young players out there and they would have learned a lot from the experience they have had out here."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.