Born in Wichita, Kansas but raised in Omaha, Nebraska, Sayers graduated from Omaha Central High School. There he set a state long jump record of 24’113⁄4″. He went on to play college football at the University of Kansas and was a two-time All-American.

During his Jayhawk career, he rushed for 2,675 yards and gained 3,917 all-purpose yards. In 1963, he set an NCAA Division I record with a 99-yard run against Nebraska. In his senior year, he led the Jayhawks to a 15-14 upset victory over Oklahoma with a 96-yard kickoff return. Sayers is considered by many to have been the greatest open field runner in college football history.