A small but growing number of Democratic members of Congress — about 20 as of Friday afternoon — have publicly called on President Biden to withdraw from the presidential race.

Nearly all are white, and many are members of the moderate New Democrat Coalition. But Biden still has strong support from most of the Congressional Black Caucus, which boasts about 60 members of the House, and the Congressional Hispanic Caucus, with about 40.

His fate may depend on whether he can hold them.

The fear of many Democrats — whether they are saying so publicly or not — is not only that Biden will lose the White House but that he will also doom their chances of winning the Senate or the House of Representatives. Congressional leaders have carried their concerns to Biden, but they have no power to force him out, and they worry that an open conflict could be disastrous for the party.

Biden has scoffed at polls suggesting most Democratic voters believe he should step aside after his disastrous debate with former President Trump last month.

“I’m determined on running,” the president said at a news conference Thursday. “I’ve got to finish the job because there’s so much at stake.”

He said he would not reconsider unless aides told him “there’s no way you can win.”

In the middle are two leaders whose decisions may be key to his fate: former House Speaker Nancy Pelosi (D-San Francisco) and the old-school patriarch of the Black caucus, Rep. James E. Clyburn (D-S.C.).

Pelosi, who retired as Democratic leader in 2022, remains the party’s most respected political tactician. On Wednesday, she pointedly told Biden that he needs to make a considered decision whether to run or not — even though the president had already said his mind was made up.

“It’s up to the president to decide if he is going to run,” she said. “We’re all encouraging him to make that decision, because time is running short.”

Meanwhile, Clyburn has expressed full support for the president — but he has pointedly added that Biden still has time to reconsider.

“The conversation should focus on the record of this administration … and let Joe Biden continue to make his own decisions about the future,” Clyburn said Friday. “If he decides to change his mind later on, then we will respond to that.”

Clyburn has also noted that he thinks Vice President Kamala Harris would make a fine candidate if the president pulls out.

A former congressional aide who’s in touch with Democratic leaders said Clyburn and others in the Black Caucus appear intent on ensuring that Harris will be next in line if Biden withdraws. “They’re making sure she’s the only alternative,” the former aide said. “The longer this goes on, the easier it will be to put her in.”

Clyburn has a special tie with Biden: In the 2020 primary race for the Democratic nomination, when Biden’s campaign was struggling, it was Clyburn who helped deliver a key victory in South Carolina thanks to overwhelming support from the state’s Black voters.

Even before that episode, Biden considered Black Democrats a cornerstone of his coalition, and he has remained attentive to their concerns. He named Harris, a Black senator, as his running mate, and appointed the Supreme Court’s first Black woman, Ketanji Brown Jackson. He waged a spirited fight for voting rights legislation that stood little chance of passage. He increased federal aid to historically Black colleges and universities.

And when his debate performance sparked panic among some Democrats, the first group he called for support was the Congressional Black Caucus.

Most members of the caucus responded with fervent endorsements. Most represent heavily Democratic districts, so their reelection is at little risk if the president loses.

“It ain’t going to be no other Democratic candidate,” Rep. Maxine Waters (D-Los Angeles) said fervently at the Essence Festival in New Orleans last week. “It’s going to be Biden.”

But even in the Black Caucus, a few cracks have appeared. This week, Reps. Emanuel Cleaver II (D-Mo.) and Ritchie Torres (D-N.Y.) expressed fears that if Biden falters, the party’s chances of winning a majority in the House will suffer too.

The chair and vice chair of the Congressional Hispanic Caucus, Reps. Nanette Diaz Barragán (D-San Pedro) and Adriano Espaillat (D-N.Y.), issued a statement supporting Biden, but two members of their caucus, Reps. Raul M. Grijalva (D-Ariz.) and Mike Levin (D-San Juan Capistrano) have called on Biden to step down.

Those calling on Biden to withdraw come from a variety of districts and backgrounds. Most are moderates, but at least six are members of the Progressive Caucus. On the other hand, several progressives who have criticized Biden for his policies on Israel and Gaza, including Reps. Alexandria Ocasio-Cortez (D-N.Y.) and Ilhan Omar (D-Minn.), have publicly supported the president.

The question now is whether the trickle of voices publicly calling on Biden to withdraw — less than 10% of the House Democratic caucus so far — turns into a deluge, prompting Pelosi, Clyburn and others to escalate their pressure on the president.

Several people close to Democratic leaders said the battle probably has weeks to run as new poll numbers appear, constituents weigh in, and more members make up their minds.

The real deadline, Clyburn and others said, isn’t until the Democratic National Convention begins on Aug. 19. At that point, the delegates who formally designate the party’s nominee can make up their minds.

Even Biden acknowledged that the delegates are free to vote for any candidate, even though almost all of them were chosen in primaries that he won.

“If all of a sudden I show up at the convention and everybody says we want somebody else, that’s the democratic process,” he said at his news conference. Then he grinned, and added in a confident whisper: “It’s not going to happen.”

Read more from columnist Doyle McManus on Trump and California: