Born and raised in New Orleans, Louisiana, Fillmore began learning about the blues at an early age. "I done lived the blues," he once said in an interview. "The blues is about picking cotton, working in the fields, living in the streets, and you know I did all these things."
In 1955, he moved to Los Angeles to pursue a musical career, playing by himself in the street and later starting a band called Eddy N & the Blues Slayers. During this time he had a relationship with Etta James before she achieved her fame as a blues singer and recorded a few 45 rpm records himself. His most successful record from this time was titled "You Got the Nerve of a Brass Monkey".