Born in 1940, Barbara Smith grew up in a rural home near a dirt crossroads known as Center Point, about ten miles east of Pittsburg, Texas. She was the youngest of seven children, one of whom died in childhood. Both her parents were teachers, though her father had to give up teaching to fight in World War II.

From an early age, music was a part of her daily life. One of her older brothers was something of a prodigy on the piano, and her mother and other older siblings were quite fond of singing. By the time she was six, it was not uncommon for her to spend an evening singing Mozart with her parents and four other siblings while her brother played the piano. Barbara Smith and her family also sang at the Baptist church that they attended every Sunday. At school, she found supportive teachers and opportunities to compete musically with the University Interscholastic League.

Given this background, it is natural that when The University of Texas began accepting African-American undergraduate students in the fall of 1956, Smith was among the first African-Americans to apply, and be accepted, to their music program.