But the star veteran says he is excited at the chance to repay the "massive favour" his teammates have done him in winning through to a semi-final clash with Geelong this Friday night, when he will return from a three-week suspension.
"Being out for the three weeks I guess the first two weeks just being home and away season (games) probably knowing that we were going to play finals weren't as important as Saturday night," Harvey said at Arden Street on Monday.
"But Saturday night I was very nervous going into the game and driving there knowing that I was helpless to help the boys.
"They've done me a massive favour and now I'm pretty excited.
"The fact I wasn't there really, really [cut] deep.
"Now I want to repay the boys, I want to repay the football club, our supporters and everybody to do with the North Melbourne Football Club."
Harvey was in the North coach's box on Saturday night as the Roos came back from 33 points down early in the third quarter to win by 12 points.
The 36-year-old said he was pretty quiet during the game but never lost faith that his teammates could come back and win.
Nonetheless, Harvey said he was happy to be trading a seat in the coach's box for a more familiar on-field role against Geelong.
"You feel so helpless (in the coach's box). All the work's done during the week and it's up to the boys on game day," Harvey said.
"There's guilt, regardless of the situation or the scoreboard, knowing that you're not injured. You haven't got a broken arm sitting there knowing that you can't play.
"You're suspended knowing that your body is 100 per cent right ready to go and you can't play to help the team.
"That was the biggest thing, I just felt helpless."
Harvey said his body felt fresh after three weeks off, but the only expectation on him this week against the Cats would be to play his role.
North lost both of its home and away matches against Geelong this season relatively comfortably, by 20 points in round 10 and by 32 points in round 19.
Asked where the Cats had got hold of North in those two encounters, Harvey said Geelong's speed through the midfield with the likes of Mathew Stokes and Steven Motlop had been the biggest factor.
Daniel Wells missed both of those clashes with a foot injury and in his third game back, against Essendon on Saturday night, the midfielder was the Roos' best four-quarter performer.
Harvey said having Wells back to take on the Cats would help the Roos match the Cats' outside run.
The five-time North best and fairest winner was not sure whether the Roos' win over Essendon – its first finals win for seven years – would help take a monkey off the club's back.
But he said the win and the experience of playing in front of nearly 79,000 people would be invaluable for young Roos such as Ben Cunnington, Jack Ziebell and Ryan Bastinac.
"The boys said it was the loudest crowd they've ever played in front of, so just that experience for those boys they'll take that into this week for sure," Harvey said.
