In early jazz, what principal instrument played the role that the saxophone later assumed?

Alright, this is an interesting question, let's talk about it.

In early jazz, the primary role later filled by the saxophone was actually played mainly by the clarinet.

Think of it this way:

In the earliest jazz bands, like those popular in New Orleans at the beginning of the 20th century, the core ensemble functioned like a lively conversation group. This group had three main voices, often called the "Front Line":

Cornet/Trumpet: Like the leader or main speaker. The loudest voice, responsible for carrying the primary melody, clearly stating the tune's theme.
Clarinet: Operated like the quick-thinking, highly talkative "spark" of the group. Its range is high, with a bright, clear tone. While the leader (trumpet) played the main melody, the clarinet improvised within the gaps, embellishing with dazzling, fast runs of notes to "add extra layers," making the music sound richer and more exciting. This role was the precursor to the later saxophone solo.
Trombone: Played the role of a steadying presence, like a "steadiness-minder." Its voice fills the lower register, often using glissandos to flesh out the harmony and provide structural underpinning for the whole sound.

Why Did the Saxophone Take Over Later?

As time progressed into the Swing era (roughly the 1930s), bands became significantly larger. The clarinet, while agile, could sound relatively thin and lacked projection in this louder, denser ensemble setting.

This is where the saxophone's advantages came to the forefront:

Fuller Sound: The saxophone (especially the tenor and alto saxophones) produces a warmer, louder, fuller, and more substantial sound than the clarinet. In a big band, its sound cuts through easily and stands out.
Greater Expressiveness: The saxophone possesses a wide range and a highly versatile tone. It can sing soulfully and vocally expressively, or produce rough, harsh growls. This made it exceptionally well-suited for the intensely personal, emotionally charged improvisation required in jazz solos.

Therefore, the role the saxophone later assumed essentially amplified and expanded upon the clarinet's role of "virtuosic display and improvisation." Furthermore, due to its powerful expressiveness, it also gradually edged out the trumpet as the dominant "melodic leader."

Put simply: The saxophone inherited the clarinet's dazzle and dexterity while gaining the sheer volume and emotional depth needed to hold its own against an entire big band, ultimately becoming jazz's defining solo instrument.