"Poltergeist" star Dominique Dunne's death at the hands of her boyfriend enraged a very famous actor so much ... he told her family he could have the guy murdered.

Dominique, the daughter of famed writer/producer Dominick Dunne and sister of actor/producer Griffin Dunne, was strangled to death by BF John Sweeney, a famed L.A. chef, back in 1980. Sweeney was charged with murder but, in what almost everyone involved says was a gross miscarriage of justice, he was convicted of manslaughter and only served several years in prison.

Griffin, Dominick and family friend/Oscar-winning producer Charles Wessler went on a mission to haunt Sweeney, getting him fired from at least 7 jobs after his release from prison.

In a new TMZ documentary on FOX Monday night at 9/8 Central, Wessler tells the story at Dominique's funeral, where a very famous actor -- an actor who everyone knows -- came up to him and said he knew "people" who could take care of Sweeney for good -- i.e., kill him.

Wessler made it clear ... it wasn't something the family wanted, but it shows the level of outrage.

As the wheels of justice began to grind, it became apparent the legal process had gone horribly wrong. The judge would not allow the jury to hear from Sweeney's prior girlfriend who had an horrific story of how he repeatedly brutalized her to the point she ended up in the hospital several times. The woman broke her silence for the first time in our documentary.

As for the offer to perform a hit on Sweeney, judging from the way Wessler tells the story, the actor may have had mob ties.

You can hear the pain Dominique's family and friends experienced, for never being able to properly avenge Dominique's death -- something we examine in detail in our new doc -- "Killing a Movie Star: A Grave Injustice."