Threatening pamphlets. Canceled shows. It was 1965, and embattled country music star Johnny Cash was facing a boycott in some parts of the Jim Crow South. But the reason was not his recent arrest for potential drug smuggling—it was his appearance on the steps of a courthouse with a woman some thought was African American.

Back in 1951, Cash was just an Air Force radio operator about to be sent overseas to intercept Soviet transmissions. That was about the time he met Vivian Liberto, a shy 17-year-old from San Antonio, at a skating rink.

After a courtship that included thousands of letters, they married in 1954. Soon after, Cash skyrocketed to fame as a rockabilly and country artist. His deft songwriting and deep voice soon gained him a fanbase, as did his outlaw-like image. Not only did he wear black to nearly all of his performances, but Cash pushed the stodgy boundaries of country music with his anti-authoritarian songs and on-stage attitude.

As he climbed to country stardom, Cash developed an addiction to prescription drugs—and a passion for another married woman, June Carter. His marriage with Vivian was on the rocks when, on October 4, 1965 he was arrested at the U.S.-Mexico border after purchasing a large quantity of amphetamines and sedatives from a Mexican dealer. Customs agents found 475 Equanil tablets and 688 Dexedrine capsules stashed in his guitar case and threw him in jail. Cash spent a night in jail and, two months later, plead guilty to the possession of illegal drugs.

He got off with a deferred sentence and a $1,000 fine—and had no idea that, as he walked down the courthouse steps in El Paso, Texas, with his wife Vivian, he was about to spark a firestorm.

An Associated Press photo of Cash and Vivian ran in newspapers the next day—and to some readers, it appeared that Vivian, an Italian-American woman who was rarely photographed, was Black.

The National States Rights Party, an Alabama white supremacist group, republished the photo in its newspaper, The Thunderbolt, with an article that dripped with racist rhetoric. The money generated by Cash’s hit records, it claimed, went “to scum like Johnny Cash to keep them supplied with dope and negro women.”

Cash was harassed and boycotted by some Southern fans. “Johnny and I received death threats, and an already shameful situation was made infinitely worse,” recalled Vivian in her 2008 memoir.

In an October 1966 article, Variety described Cash as “the innocent victim of a targeted hate campaign in the south.” The “racial error,” wrote the anonymous author, had sparked boycotts and threats. “In the code of the south,” the article continued, “there is no greater crime than miscegenation.” At the time, interracial marriages were banned throughout the South.

Though the National States Rights Party was not the Ku Klux Klan, it had close ties to the organization and in publicity about the campaign against Cash, many outlets—and Cash himself—identified it as the KKK.

“Cash’s manager had to respond,” says Cash biographer Michael Streissguth, author of Johnny Cash: The Biography. “He was out there saying that Cash was not married to a Black woman.” Cash made a statement that his wife was, in fact, white, and threatened a lawsuit.

“I remember talking to his daughter Roseanne about it,” says Streissguth. “She got a letter from him saying ‘I’m sorry I haven’t been home, but I’ve been out fighting the KKK.’ She said she took the letter and ripped it in half—it was just another excuse for his long absences from home.”

Streissguth finds it troubling that Cash felt he had to deny being married to a Black woman so vehemently. But, he says, Cash’s career shows he was racially tolerant. He points to Cash’s partnerships with Black artists on his ABC television show and songs like “All of God’s Children Ain’t Free,” which touches on issues of racial equality, as better indicators of Cash’s own feelings about race. Cash also commented on the United States’ treatment of indigenous people on his 1964 album Bitter Tears, a concept album that explores the destruction of Native American land and atrocities against Native Americans.

The incident “had the potential to affect his core, Southern audience,” says Streissguth, but ultimately it remained a footnote in his bigger story.

So did the National States Rights Party. Though The Thunderbolt had a subscriber base of 15,000 at its height, the party itself was small and only played a brief role in the history of American hate. “Its propaganda and public activities are all geared to arousing the passions of avowed racists and hatemongers, and in some instances, at least, it has been successful,” the FBI wrote in a 1966 report.

But its campaign against Cash only partly succeeded. “There were more cancellations of his concerts over the drug arrest than these charges the separatist group made,” says Streissguth.

Cash and Vivian’s marriage ended in 1967, a year after the stressful campaign lost steam. That same year, the U.S. Supreme Court ruled anti-miscegenation laws unconstitutional in Loving v. Virginia. Today, attitudes about interracial marriage have changed dramatically. According to a 2013 Gallup poll, 87 percent of Americans favor marriage between Black and white people—up from a mere four percent in 1958.