With his first wife, Norma Sincock, James was the father of four children: Carol, John, Beverly and Barbara Pryor.

After Norma passed away in 1988, he married Suzanne Klusmann, who was the mother of five children with her first husband James Olson, from whom she was divorced in 1985. They are Michael, David, Kristin, Melissa and Jennifer Olson.