In his memoir Coming Unbuttoned (1993), Broughton recounts his childhood, reflects on his work, and remarks on his love affairs with both men and women.

Among his male lovers were gay activist Harry Hay and publisher Kermit Sheets. In 1962, Broughton married Suzanna Hart. The couple was divorced in 1978. On Christmas Eve 1976, Broughton celebrated his relationship with artist Joel Singer in a marriage ceremony. Eschewing the labels homosexual, heterosexual, and bisexual, the poet and filmmaker describes himself as a "pansexual androgyne."

This witty and impudent confession is the work of a cultural pioneer whose adventures among the famous and the infamous extend from New York circles of the '30s to the avant-garde antics of San Francisco in the '60s and '70s. Born a gleeful poet in a solemn family, James Broughton survived military school, Stanford University, the merchant marine and journalism before his passion for cinema and his dedication to poetry crystallized in 1948 with his first book and the first of his many films. In the '50s he worked in London and Paris; and for many years he occupied a special place in the San Francisco Bay Area as a performer, playwright and professor.

In "Coming Unbuttoned" Broughton shares intimate memories of Anais Nin, Alan Watts, Robert Duncan, Maya Deren, Jean Cocteau, W.H. Auden, Pauline Kael, Kenneth Rexroth, Robinson Jeffers, and the poets of the Beat Generation.