It looked like staying that way until full time too, but in a game full of high-quality goals Toivonen saved the best until last as he marked his 50th cap by scoring the victor in incredible fashion with the last kick of the match following a terrible error from Hugo Lloris.
The video of Giroud's stunning goal is provided below.
Sweden have won nine matches out of last twenty games.
The game was starting to warm up and there ought to have been an opener on 35 minutes when Bernard Mendy did superbly down the left before sending in an inviting ball for Olivier Giroud who could not get quite enough on the header.
France were looking the more likely side to find a victor and Olsen had to repel an ambitious Griezmann free-kick, but Sweden still possessed some threat, with Johansson sending a 30-yard drive just off-target and a dipping Emil Forsberg set-piece forcing Lloris into action.
He was around five yards inside his own half.
'16 champ Djokovic stunned by Thiem in French Open quarters
Thiem erased the first with a forehand volley and the other with a service victor that prompted Djokovic to roll his eyes. Djokovic's backhand really let him down in the tiebreaker: All seven points won by Thiem ended with that stroke.
I misjudged you: Piers Morgan apologizes to Ariana Grande
Grande wore a white jumper emblazoned with the slogan One Love Manchester as she performed. "To the families, we love you so much".
What's new in WatchOS 4?
There are Photos, content from Apple news, coaching reminders from the Activity App, the weather, a reminder from the Breathe app. As well as being customizable, faces can now dynamically change based on various criteria, such as location and time of day.
Switzerland kept themselves ahead of Portugal at the top of Group B with a 2-0 away win over the Faroe Islands. Instead, they'll remember Lloris's howler. In the last 5 Sweden vs France Head-to-Head matches, France won 3 times, Sweden once and 1 ends in a draw.
A poor clearance by Tottenham 'keeper Hugo Lloris fell to Ola Toivonen and the former Sunderland loanee fired home from the half-way line.
With twenty seconds of added time remaining and Les Bleus bound for a creditable point in Stockholm the goalkeeper dribbled too far with the ball before passing to Sweden forward Ola Toivonen on half way.
Portugal's forward Cristiano Ronaldo (C) celebrate scoring with his teammates during the FIFA World Cup 2018 qualification football match between Latvia and Portugal in Riga, on June 9, 2017.
For Deschamps's side it was the first defeat in qualifying and they next face a tough test against the Netherlands in Paris.
A win at the Friends Arena in Solna would have moved European Championship runners-up France six points clear of Sweden.