Mario Pasalic's first-half goal for Atalanta was cancelled out by a late equaliser from Marquinhos before Eric Maxim Choupo-Moting popped up for a stoppage-time winner to end a dream run in the Champions League for the Serie A club.
"When I came on, I thought to myself, 'We can't lose, we can't go home like that.' I was confident in myself, confident in the team and then the rest is the history of Paris," said Cameroon international Choupo-Moting.
"It was a crazy game, a tough game and opponents."
PSG missed a golden chance to open the scoring inside five minutes, but Neymar shot well wide of goal after doing well to break in behind the Atalanta defence.
High-flying Atalanta, who led Serie A with 98 goals in 38 league games this campaign, took the lead in the 26th minute when Pasalic curled a first-time shot in from the right side of the area that left Keylor Navas no chance.
Neymar continued to find space in the attacking third and came close to levelling shortly after the Pasalic goal, but he blazed a shot just outside of the post and then blasted another over the goal right before half-time as Atalanta took the lead into the beak.
"I never thought we'd be back home tomorrow," Neymar said. "At every moment, we just went for one thing: progress to the last four.
"Nobody will ever get it out of my head that I want to go to the final. We're a formidable squad, a family, with the attitude that it's impossible to knock us out."
PSG found chances harder to come by in the second half and Thomas Tuchel brought on Kylian Mbappe on the hour mark. The France international started on the bench as he'd just returned from an injury sustained in the French Cup final win on July 24.
Mbappe's first chance came in the 73rd minute after he charged into the Atalanta area only to see his near-post effort kicked away by goalkeeper Marco Sportiello.
Sportiello kept two more dangerous-looking Mbappe efforts out right around the 80-minute mark as the Ligue 1 champions looked to find a way back on level terms.
Disaster struck as Atalanta tried to see out the game when Marquinhos poked home a mis-hit shot from Neymar and Choupo-Moting redirected Mbappe's square ball past Sportiello for the win.
Atalanta had a chance to equalise but Luis Muriel, after breaking clear, ran the ball out of play and his team mates slumped to the ground in despair as the final whistle went in a famous victory for PSG.
Choupo-Moting's goal -- the latest regulation game-winning goal in UCL quarterfinals history -- sees PSG into the semifinals for the first time since 1994-95 after enduring last-8 exits in their previous four attempts.
The intensity of Tuchel's reaction was intensified by the fact that, with an ankle injury he picked up last week, he was forced to sit and watch on an ice-box for most of the game with his leg strapped in a "moon boot."
"Imagine what I would have done if I had two legs!" he told the postmatch news conference. "Maybe today would be the day you would see me sprint for 40 metres. But it's the club's birthday today, exactly 50 years, so we will not forget this birthday for sure."
Atalanta coach Gian Piero Gasperini was left with a mixture of regret and pride in his team after the heartbreaking result.
"There is a great sense of regret. It seemed we were going to do it and that would have been a great achievement," said Gasperini, whose side had taken a first-half lead.
"We still have [the memory of] a Champions League where we have grown as a team against some of the greats of Europe and I can't do anything other than thank my lads."
The Ligue 1 side will face either Atletico Madrid or RB Leipzig, who meet on Thursday, in their semifinal.
Information from Reuters was used in this report.