Born near Salt Sulphur Springs, Virginia (now West Virginia), Harvey attended common schools in Indiana, Illinois, and Iowa. He became a civil engineer and headed west as a prospector to Pike's Peak in 1859 as a Fifty-Niner. After meeting several discouraged miners along the way, Harvey decided to settle instead in Kansas Territory, so he acquired a plot of land in Riley County near Fort Riley and engaged in agricultural pursuits. From 1861 to 1864, he served with the Union Army during the Civil War, advancing to the rank of captain in the 4th Kansas Infantry, which failed to complete organization and was consolidated with other recruits to form the 10th Kansas Infantry.
Early in 1860 the fourth U. S. Cavalry was station at Fort Riley but it was not there for long and soon after the outbreak of the influx of Kansas regiments began. A company was organized at Ogden Kansas called the “Ogden Mudsills”. This was a volunteer company which afterwards became company G., of the 10th., Kansas. James M. Harvey was elected captain of the company. While captain of the fourth his company was known as the “Ogden Mudsills”. The meaning of the word “Mudsills”, applies to the people of the lower classes in the Southern part of the United States. The origin of the term comes from the use of the sill of the lower part of the structure of any building erected in swampy places, or in streams for mills. Being the underlying piece of timber, and lying in the mud, you will see how apt the term became when applied to the classes. It was a term used prior to the Civil War in the discussion of slaves and slave owners, and white trash, and the lower classes of the Northern citizen towns.
Harvey was elected to the Kansas House of Representatives, 1865-1866, and then elected to the Kansas Senate in 1867-1868. He was Governor of Kansas for two terms, serving from 1868 to 1872, and then elected as a Republican to the U.S. Senate to fill the vacancy caused by the resignation of Alexander Caldwell, where he served from February 2, 1874, to March 3, 1877.
After his Senate term, Harvey worked as a government surveyor in New Mexico, Utah, Nevada, and Oklahoma, before returning to Kansas in 1890 to resume agricultural pursuits. He died near Junction City, Kansas in 1894. Interment was in Highland Cemetery, Junction City.