PORTLAND - A federal appeals court has rejected a challenge to the Internal Revenue Service in a cattle-ranching scam that cost investors in 41 states more than $100 million.

The 9th U.S. Circuit Court of Appeals on Thursday ruled in favor of the IRS in a lawsuit filed by dozens of victims of the so-called "phantom cattle" scam by former Oregon rancher Walter J. Hoyt III, nicknamed the "Paper Cowboy" for herds of cattle that existed only on paper.

Between 1971 and 1996, Hoyt organized partnerships that were partly designed to offer investors a tax deduction in addition to any profits on livestock sales. But the partnerships eventually became what some investigators called the biggest agricultural scam in U.S. history.

The investors had sued the IRS after the agency disallowed tax deductions based on the phony livestock operation and ordered payment of penalties and interest. Some of the investors claimed bankruptcy and other financial hardships as a result.

The investors had blamed the IRS for failing to stop the Hoyt operation despite continuous audits of Hoyt for 24 years.

A three-judge panel noted in their unanimous ruling that Hoyt was convicted of bankruptcy fraud, mail fraud and money laundering, but he never was indicted for tax crimes investigated by the IRS.

The tax agency began a criminal investigation in 1984 that ended without charges in 1987. A second investigation that began in 1989 also ended without prosecution.

The investors said that until 1995, the IRS allowed Hoyt to claim the tax refunds, which he used to encourage more investors to become partners.

Still, the judges said the investors "have no right to ... relief for any allegedly unconstitutional actions of IRS officials engaged in tax assessment and collection."

An IRS spokesman in Washington, D.C., said the agency does not comment on lawsuits still in court.

The attorney for the scam victims, Wendy Pearson of Seattle, said she was disappointed by the ruling.

"If ever there was a case that warranted a bit of relief, this is it," Pearson said.

She said the investors relied on Hoyt to represent them in all dealings with the IRS under provisions of the Tax Equity and Fiscal Responsibility Act of 1982.

The judges noted that Hoyt's partners executed "power of attorney" documents that authorized Hoyt to act on their behalf in tax matters. But they also said that the tax act "ensures that individual partners retain meaningful opportunities to participate in partnership audits and litigation."

Pearson said the case was an opportunity for the 9th Circuit to clarify the tax act to avoid such cases in the future, but the ruling indicates "the court is basically leaving it to Congress."

She said the 99 investors she represents will discuss the possibility of an appeal to the Supreme Court.

Hoyt is serving a 19-year sentence at a minimum-security federal prison in Arizona. At its peak in 1987, his business sprawled across more than 500,000 acres of ranches and leased federal land in the high desert of eastern Oregon.