Sebelumnya, gelar juara kelas berat milik Patterson direbut oleh [[Sonny Liston]] yang mendapat kemenangan KO pada ronde pertama. Dalam pertandingan kedua, Patterson kembali kalah pada ronde pertama. Liston kemudian kehilangan gelarnya karena dikalahkan Ali. Sementara itu, Patterson memenangkan pertandingan melawan [[Eddie Machen]] dan [[George Chuvalo]]. Pada saat itu Patterson menjadi penantang utama gelar juara yang dipegang Ali.
Pada tanggal 22 November 1965 Patterson melawan Ali untuk pertama kalinya. Patterson kalah secara ''technical knockout'' pada penghujung ronde ke-12 meski sempat bersikap optimistis sebelum pertandingan. Bukannya mencari kemenangan KO dengan cepat, Ali justu mengolok-olok, mempermalukan, dan menyiksa Patterson sepanjang pertandingan hingga wasit menghentikan pertandingan pada ronde ke-12. ASeorang jurnalis ''[[The New York Times]]'' reportermengeluh complainedbahwa thatmelihat thepertandingan fightitu wasseperti likemelihat watchingseseorang someoneyang "pullingsedang themenarik wingssayap off a butterflykupu-kupu."<ref>Sports Illustrated, 13 June 2016, pp. 39.</ref>
OnPada Septembertanggal 20, September 1972, Patterson foughtmelawan Ali foruntuk thekedua second timekalinya. Saat Atitu agedia sudah berumur 37,Pattersontahun wasdan stoppedupayanya indihentikan thepada 7thronde roundke-7. ThePertandingan defeatitu provedmenjadi topertandingan beterakhir Patterson's last fight, although therewasmeskipun neverdia antidak announcementpernah ofmengumumkan retirementpensiun.<ref name="ESPN">{{cite news|url=https://espn.go.com/classic/s/Ali_record.html |title=Muhammad Ali's ring record |work=ESPN |date=19 November 2003|access-date=4 October 2016}}</ref><ref name="SI">{{cite news|url=https://www.si.com/boxing/2015/09/24/muhammad-ali-floyd-patterson-las-vegas |title=Rabbit hunt in Vegas |work=Sports Illustrated |date=22 November 1965|access-date=6 October 2016}}</ref><ref name="Boxing News">{{cite news|url=http://www.boxingnewsonline.net/on-this-day-muhammad-ali-toys-with-poor-floyd-patterson/ |title=On This Day: Muhammad Ali toys with poor Floyd Patterson |work=Boxing News |date=22 November 2014|access-date=6 October 2016}}</ref><ref name = "Glory Years">{{cite book|title=Muhammad Ali: The Glory Years|authors= Felix Dennis and Don Atyeo|publisher=miramax books|pages = 136–8;196|year=2003}}</ref><ref name = "Hauser">{{cite book|title=Muhammad Ali:His Life and Times|author= Thomas Hauser|publisher=Simon & Schuster|pages = 138–41;248|year=1991}}</ref><ref name = "Remnick">{{cite book|title=King of the World|url=https://archive.org/details/kingofworldmuham00remn|url-access=registration|author= David Remnick|publisher=Random House|pages = [https://archive.org/details/kingofworldmuham00remn/page/271 271–83]|year=1998}}</ref><ref name="SI 2">{{cite news|url=https://www.si.com/vault/1965/11/15/607630/the-greatest-meets-the-grimmest |title=THE GREATEST MEETS THE GRIMMEST |work=Sports Illustrated |date=15 November 1965|access-date=6 October 2016}}</ref><ref name="SI 3">{{cite news|url=https://www.si.com/vault/1965/11/29/612642/champion-as-long-as-he-wants |title=CHAMPION AS LONG AS HE WANTS |work=Sports Illustrated |date=29 November 1965|access-date=6 October 2016}}</ref>
