Conservative Rep. Ken Buck is just one of several House Republicans standing in the way of the right's push to impeach President Joe Biden.

But his high-profile seat on the key House Judiciary Committee, recent outspoken interviews railing against the House GOP's investigative efforts, and long track record of bucking his own party have put a target on his back in conservative circles.

Now, there is a serious effort underway to find a candidate to mount a primary challenge against Buck in his solidly red district in eastern Colorado, three GOP sources told CNN -- the latest sign of tension as the House GOP grapples with internal divisions over everything from its agenda to former President Donald Trump.

Rep. Marjorie Taylor Greene of Georgia, an ally of House Speaker Kevin McCarthy and one of the most vocal proponents of impeaching Biden, said it's up to Buck's constituents to decide whether he is the best person to represent his district. But Greene -- who has faced repercussions herself for crossing some of her former House Freedom Caucus colleagues earlier this year -- told CNN there is an "unbelievable" level of frustration with Buck inside the House GOP, and said she doesn't think he should still be able to serve on the Judiciary panel or the GOP whip's team, which is responsible for vote counting.

"This is the same guy that wrote a book called 'Drain the Swamp', who is now arguing against an impeachment inquiry," Greene told CNN. "I really don't see how we can have a member on Judiciary that is flat out refusing to impeach. ... It seems like, can he even be trusted to do his job at this point?"

She added: "I don't know how we can have a whip (team member) that continues to vote against our conference bills. That's completely wrong."

Buck declined to comment for this story, though he has taken his escalating feud with Greene public in recent media appearances. On MSNBC over the weekend, he called Greene's impeachment comments "absurd" and forcefully pushed back on Greene's "untrue" suggestion that January 6, 2021, defendants are being mistreated.

A spokesman for the Judiciary Committee declined to comment, and the office for House GOP Whip Tom Emmer -- who is friendly with Buck -- did not return a request for comment. But GOP Rep. Ben Cline of Virginia, also a member of the panel, told CNN, "Ken Buck is an important member of Judiciary, and I respect him greatly."

Allies close to Buck say he isn't afraid of a primary challenge, noting he easily beat back a GOP opponent in 2022, after facing blowback from the MAGA wing for his vocal support for former GOP Rep. Liz Cheney of Wyoming and his decision to certify the 2020 election results.

But privately, a half a dozen sources said there is growing frustration -- including among the leadership ranks -- with Buck, who has voted against the House GOP's signature parental rights bill, their annual defense policy bill and the bipartisan debt ceiling deal. Buck also was among a group of Republicans who tanked a procedural vote on a gas stove bill in protest over spending levels and has openly suggested on CNN that McCarthy is just using impeachment as a "distraction" from the other issues he is facing.

Yet one of Buck's biggest offenses on the right has been his vocal opposition to launching an impeachment inquiry. Buck, who is a member of the ultra conservative House Freedom Caucus, has said he doesn't believe the House has produced any evidence that Biden profited off his son's foreign business deals and cast doubt on whether the evidence even exists -- particularly significant since he a senior member on the committee that would help oversee an official impeachment inquiry. The Judiciary Committee has also been one of the key panels involved in investigations into the Biden family.

Buck's opposition has caused a headache for House GOP leaders who are working to convince not only moderate holdouts, but committee members like Buck who are positioned on the front lines of the effort. One of the GOP sources complained that Buck has been far more vocal on television -- including in appearances on CNN -- about his impeachment stance than he has been with his colleagues, further causing some heartburn. And Democrats have been quick to amplify Buck's public comments, which has only added fuel to the fire.

One GOP lawmaker, who serves on a different committee investigating Biden, said there's been a lot of "grumbling" inside the GOP about Buck over his comments about their Biden probes.

"I often side with Buck, but not on this one," the member said.

Another GOP lawmaker, granted the condition of anonymity to speak freely, said Buck's position against a Biden impeachment "makes no sense" and that Buck should at least be open to supporting an inquiry given what House Republicans have uncovered so far. But, this lawmaker said even given the growing agitation, it was too soon to talk about a primary challenger for Buck.

'I don't think he's on an island'

Buck may be the most public about his skepticism to his party's impeachment efforts against the president -- as well as Homeland Security Secretary Alejandro Mayorkas, another top target on the right -- but he certainly is not the only one.

Even some of his Republican colleagues on the House Judiciary Committee have previously raised concerns about moving forward with an impeachment effort at various points in the process as their panel's investigation has played out. But they have been less publicly vocal in their resistance compared to Buck.

"I'm sure he'll face some backlash from the far right in his district. But ... I don't think he's on an island on his own," Democratic Rep. Diana DeGette, a fellow member of the Colorado delegation and an impeachment manager during Trump's second trial, told CNN. "I think a lot of the Republicans feel this way. He's coming out and saying publicly what a lot of Republicans are saying privately."

While it is traditional for impeachment articles to proceed through the Judiciary panel, House Democrats brought their second impeachment against Trump straight to the House floor. House Republicans can only lose two Republican Judiciary Committee members if the articles move through the committee and all members of the panel vote. On the House floor, the House GOP conference can only lose four votes given the current margins in the House, though that can fluctuate with absences.

Since members have been out of town for the last six weeks, sources said it is difficult to grasp just how strong the opposition is to a Biden impeachment effort within the conference. While GOP leadership has informally been taking the temperature of the conference over the summer, one source told CNN, they are waiting until members return to Congress this week to get a more official sense of where things stand. House Republicans will hold a special conference meeting on Thursday to specifically discuss the topic, according to GOP sources.

Leadership largely views Buck as a lone operator and thinks he may vote against impeachment no matter what, and aren't working him hard to change his tune. Other members, however, are seen as more moveable. They are hoping to convince skeptics to at least get on board with an impeachment inquiry, which leadership has argued is different from an actual impeachment vote and would just strengthen their investigative powers.

"We're gonna have to find the answers," McCarthy told reporters on Monday. "And this is all information that just has been coming forward that we've been able to find out. But the other information is we find that the Biden family delays everything. It benefits them to delay the information. The American public deserves to know."

Talk of a Buck primary challenge grows

Buck -- a former prosecutor with the Department of Justice and former Colorado GOP chairman who came to Congress in 2015 -- once again finds himself in the right's crosshairs, despite being a staunch conservative himself. Buck has drawn the ire of the MAGA wing of the party for voting to certify the 2020 election results on January 6, 2021, standing up for Cheney amid efforts to remove her from Republican leadership, and criticizing Trump's legal issues and third presidential campaign.

He also put himself at odds with both McCarthy and GOP Rep. Jim Jordan of Ohio in the previous Congress, when he teamed up with Democrats on anti-trust legislation. While Buck was the top Republican on the anti-trust subcommittee at the time, when House Republicans took over the majority that subcommittee gavel went to GOP Rep. Thomas Massie of Kentucky instead.

Among the names of people being floated to potentially challenge Buck in a primary, according to several sources familiar: state Sen. Jerry Sonnenberg, who sources said thought about challenging Buck last cycle but ultimately did not, and state Rep. Richard Holtorf, a pro-Trump Republican who is currently minority whip in the Colorado general assembly.

The budding retaliation against Buck mirrors a similar playbook that members of the GOP have used against other vocal detractors who have broken out of their prescribed mold. Greene, for example, was voted out of the far-right Freedom Caucus in July for her growing allegiance to McCarthy and his leadership team. Unlike the majority of the House Freedom Caucus, Greene stood by McCarthy in his fight to become speaker and opposed members of the group who blockaded the House floor for a week over the debt ceiling deal House GOP leaders negotiated with the White House.

There's also been talk among hardliners of supporting primary challenges against other Republican members who have opposed the GOP's agenda and criticized Trump or conservative colleagues. GOP Rep. Tony Gonzales of Texas has already drawn a primary challenger from Brandon Herrera, who posted on Instagram over the weekend that he received a tour of the Capitol with help from the communications director for hardline GOP Rep. Matt Gaetz of Florida, who has spoken highly of Herrera's campaign in the past.

Some Republicans, however, have speculated whether Buck would even run again, given his recent behavior, though Buck has previously said on the record he fully plans to seek reelection.

"Everyone on Capitol Hill knows that Ken Buck has given up on his work with the Freedom Caucus, the Foreign Affairs Committee and the Judiciary Committee so that he can try-out for jobs with TV networks or the Biden administration" a Republican source familiar with the internal discussions told CNN. "He's totally abandoned all principles to try and make a name for himself. It's sad that such a formerly great member would do so."

A third GOP lawmaker was even blunter: "We call him buckle."

But Buck -- who has relished in his maverick status inside the conference -- so far still has allies both in the Freedom Caucus and on the Judiciary Committee.

GOP Rep. Chip Roy of Texas, who is a member of both, told reporters on Monday: "You can agree to disagree. We have slightly different views about what high crimes and misdemeanors look like. I've studied this and looked at the history of it, and I'm more than comfortable about an impeachment inquiry. But I look forward to talking to Ken about it in person."