Williams was born into a musical family and fathered it. His brother Rudolph was a clarinetist and saxophonist, while his sons Rudy, Phil, and Estella were both saxophonists. Estella was also a pianist and singer. He was Charles Mingus’s uncle. Williams began on the violin. In his teens, Williams attended Tuskegee and studied with Major N. Clark Smith. He learned many instruments before settling on the clarinet. In 1914, he moved to Cincinnati and began playing saxophone with a local band. Before moving to Chicago in 1923, he also managed his own band for many years. A year later, he moved to New York City and led bands at the Rosemont Ballroom in New York City. From 1926-8, he was resident at the Savoy Ballroom in NYC and led his own band, The Royal Flush Orchestra. He recorded the Okeh label with the band as a clarinetist, saxophonist, and vocalist. Williams decided to leave his group’s leadership in the hands of others and moved to Chicago where he briefly assumed control of Dave Peyton’s bands. In 1929, he returned to New York and led bands there until the end of the 1930s. Williams quit music and became a real estate agent, performing occasionally with his own band in the mid-Atlantic states. After he stopped performing, he was appointed to the New York Local 802 musician’s union. He resigned in 1964. His nephew Mingus, who had Williams perform as a guest at the Town Hall concert in October 1962, interrupted Williams’s retirement. Williams was not recorded on the album that resulted from this notoriously chaotic event. Mingus might have had problems with the union. At least one biographer has suggested that Williams’s appearance could have been part of a mutual back scratching arrangement. Allmusic