With Chance Ruffin (0-2) on the mound to start the 10th, Kozma singled with two outs to keep the inning alive and Oliver Perez took the mound.
Walks to Matt Carpenter and Jon Jay loaded the bases before Perez let go of a pitch that got past catcher Mike Zunino, allowing Kozma to score for the win.
"We've got a lot of young players up here," Mariners manager Eric Wedge said. "When you're playing tight ballgames like a lot of our games have been and you're tight late, that's when it really shows itself."
The victory gives St. Louis a one-game lead over Pittsburgh for first place in the NL Central, as the Pirates lost to the Cubs on Friday.
Zunino's leadoff home run to start the fifth was the only run of the game until the Cardinals worked one across in the eighth.
With Charlie Furbush on the mound, Brock Peterson lofted a pop up to shallow right that Nick Franklin drifted back to play. He lost track of the ball, though, and it fell with Peterson reaching second base. Kozma came out to run and stole third as Carpenter walked, and Jay followed with a grounder to second, beating out the double play to tie the game.
"That was an inning where we made something out of nothing," said St. Louis manager Mike Matheny. "Pete has real good instincts when he's on the base and he won this game for us."
Kevin Siegrist (3-1) allowed runners to reach second and third with two outs in the 10th, but Abraham Almonte flied out.
Neither starter factored into the decision as Hisashi Iwakuma gave up just three hits and two walks over seven innings for Seattle, while Adam Wainwright tossed eight innings and gave up a run on six hits.
St. Louis has won six of its last seven games, while Seattle has dropped its past five games.
The last time these teams met was in 2010 when the Cardinals took two of three from the Mariners in St. Louis ... Both teams left seven men on base ... James Paxton makes his second career start for Seattle on Saturday, and Michael Wacha will take the mound for St. Louis.
09/14 02:09:45 ET