Freo drew level with the injury-depleted Tigers 19 minutes into the final term and looked set to finish the stronger side.
But Hayden Ballantyne's incorrect entry to the playing arena with 4:38 left on the clock saw the ball taken off Adam McPhee on the half-back flank and given to Jack Riewoldt in the goal square.
The Coleman Medal frontrunner duly converted the set shot which sparked the final flurry that consigned Fremantle to a 19-point defeat.
"It's a pretty heavy penalty to pay," Harvey said after the match.
"I'm not talking about the result or whether it cost us a victory - but [in terms of] momentum and where the state of play was. I'll have a look at it and we'll work through it.
"[The rule] has been around for a while hasn't it? It's common sense probably. Do you want a finals game [decided] like that? I'm not sure we do, but that's the rule so we live with it."
Harvey noted that Richmond was down to two fit players on the bench in the second half and made his players aware of it, but did not change his rotation policy in a bid to take advantage.
As displeased as he was over the interchange bungle, Harvey felt his side's inability to move the ball effectively inside forward 50 had ultimately proved more costly.
"Our kicking wasn't good enough," he said, conceding the absence of mature-age rookie sensation Michael Barlow was a factor.
"We were tending not to find a player who was unpressured and tried to find a pass into a pressured situation - that's decision making. That's an issue that we'll have to take up with the players."
"We were just looking for a bit of polish and poise and we didn't have enough of that today. We just couldn't control the game enough to be able to assert ourselves."
The loss will allow either Carlton or the Western Bulldogs to move one game closer to Fremantle's fourth spot when the sides clash on Sunday evening.
"We're not looking at that," Harvey said.
"We actually don't look too much into where we're at on the ladder. We just persevere with our game plan and try to progress it really."
Luke McPharlin, Chris Tarrant, Des Headland, Nat Fyfe and Greg Broughton are among the names that will push for returns from injury next week according to the coach.
