Harald is described by Snorri Sturluson to have been physically "larger than other men and stronger". He is said to have had light hair and beard, a long "upper beard" (moustache), and that one of his eyebrows was somewhat higher situated than the other. He also reportedly had big hands and feet, and could measure five ells in height. It is not known whether Snorri's description of Harald's physical appearance actually represents historical facts.

Harald himself composed skaldic poetry. According to Lee M. Hollander, composing poetry was normal for Norwegian kings, but Harald was the only one who "showed a decided talent." According to one poem, Harald had mastered a number of activities that were considered sports in the Viking Age, in addition to poetry, brewing, horse riding, swimming, skiing, shooting, rowing and playing the harp. The sagas state that Harald and his Varangians at least once took a break during the siege of a town to enjoy sports.

With regards to religion, Harald had, according to DeVries, a "religious inclination towards Christianity" and was "publicly close to the Christian Church" although he was influenced by the Eastern Christian culture of Russia and the Byzantine Empire, having spent most of his life there. He was clearly interested in advancing Christianity in Norway, which can be seen by the continued building and improvement of churches throughout his reign.

Despite this, DeVries notes that Harald's "personal morality appears not to have matched the Christian ideal,” citing his marriage arrangements.

Harald married Elisiv of Kiev (born c. 1025 – died after 1066) around 1044/45 and they had an unknown number of children, possibly several.

According to Snorri Sturluson, they had two daughters:
• Ingegerd (c. 1050 – 1120). Married first to the future Olaf I of Denmark, and after his death, to the future Philip of Sweden.
• Maria (died 25 September 1066). Promised away for marriage to Eystein Orre (brother of Tora Torbergsdatter), but reportedly died on Orkney the same day that Harald (and Eystein) died at Stamford Bridge.

According to the sagas, Harald married Tora Torbergsdatter (c. 1025 – after 1066) around 1048. Some modern historians have disputed this, since Harald in that case would be in a bigamous marriage, as he was still married to Elisiv. It is nonetheless possible that such a marriage could take place in Norway in the 11th century, and although Harald had two wives, only Elisiv is noted to have held the title of Queen. Harald and Tora had at least two children:
• Magnus II (c. 1049 – 1069). Reigned as king of Norway from 1066 to 1069.
• Olaf III (c. 1050 – 1093). Reigned as King Olaf III of Norway from 1067 to 1093.