Catherine, Princess of Wales, Prince William, Prince of Wales and King Charles III

King Charlesis kicking off the Christmas celebrations at his annual pre-Christmas Palace lunch, but not everyone will be in attendance.

PEOPLE understands thatKate Middleton,Prince Williamand their three children,Prince George, 11,Princess Charlotte, 9, andPrince Louis, 6, will not be present at this year’s festivities.

The Prince and Princess of Wales were never expected to attend the lunch at Buckingham Palace on Thursday, Dec. 19. The couple and their children are already in Norfolk, England, more than 100 miles away from London, ahead of the royal family’s private Christmas celebrations at the Sandringham Estate.

PEOPLE understands that William and Kate are looking forward to catching up with family at Sandringham over Christmas as usual.

Prince William, Prince George, Princess Charlotte, Prince Louis and Kate Middleton at the 2023 Christmas carol service.Aaron Chown - WPA Pool/Getty

Kate Middleton High Fives Boy Scout in Sweet Moment from Christmas Carol Concert

Aaron Chown - WPA Pool/Getty

While Kate and William may not have joined in the festivities, Charles, 76, still hosted several members of the royal family for Christmas lunch, including his younger brotherPrince Edwardand his wifeSophie, the Duchess of Edinburgh,who were seen arriving at Buckingham Palace on Thursday.

Charles' brotherPrince AndrewandSarah Fergusoncanceled their appearance at the Christmas luncheon in the wake of backlash around the Duke of York’ssuspected connectionto an alleged Chinese “spy.”

Prince Andrew and Sarah Ferguson at the Easter Mattins Service at Windsor Castle on March 31, 2024.Samir Hussein/WireImage

Prince Andrew, Duke of York and Sarah, Duchess of York attend the Easter Service at Windsor Castle on March 31, 2024 in Windsor, England.

Samir Hussein/WireImage

PEOPLE understands that the Duke of York, 64, reached the decision after discussing the matter with his ex-wife, the Duchess of York, who convinced him that this was the right move. The same is true of Prince Andrew’sdecision to skipthe royal family’s Christmas gathering at Sandringham next week on Dec. 25.

Royal biographer Ingrid Seward tells PEOPLE that Andrew will have also spoken things over with his daughters,Princess BeatriceandPrincess Eugenie, who are protective and loyal towards him.

“Despite [the Duchess'] missteps in the past, she does know how things will play out socially. She would tell him it’s best to stay away. And Beatrice would have backed her up in that,” says Seward.

Prince Andrew and Ferguson, who is popularly known as Fergie, divorced in 1996 after a decade of marriage, but remain close today and continue to live at Royal Lodge in Windsor.

The royals' Christmas lunch is a private family gathering, and a tradition King Charles has continued from his late motherQueen Elizabeth’s reign. Dozens of family members conveneabout a week before Christmasfor the upbeat holiday meal at Buckingham Palace in London. From there, the late Queen would head to Sandringham House, her private estate in the Norfolk countryside, and step out for church at St. Mary Magdalene on Christmas Day with her family in tow.

King Charles and Queen Camilla at Sandringham for Christmas with Prince George, Princess Charlotte, and Prince Louis in 2022.Samir Hussein/WireImage

SANDRINGHAM, NORFOLK - DECEMBER 25: Princess Charlotte, Catherine, Princess of Wales, Camilla, Queen Consort, Prince Louis, Prince George, King Charles III and Prince William, Prince of Wales attend the Christmas Day service at Sandringham Church on December 25, 2022 in Sandringham, Norfolk. King Charles III ascended to the throne on September 8, 2022, with his coronation set for May 6, 2023.

The royal family’s Christmas plans have been under unusual scrutiny this year amid fallout around Andrew’s alleged links to the “spy” scandal. Charles' younger brother has been underharsh spotlightsince Dec. 13, when it emerged that he reportedly had close professional ties to a Chinese national, alleged to be a spy. The story made headlines when a British court upheld an existing decision to ban the man from the U.K. on national security grounds following a special immigration investigation, drawing attention also to his reported relationship with Andrew.

The businessman — initially codenamed only as H6, and later named on Dec. 16 as Yang Tengbo (also known as Chris Yang) — became a close confidant of King Charles' brother and allegedly earned “an unusual degree of trust” from the royal, theBBCreported.

The Duke of York took aforced step backfrom public duties following hisbombshellNewsnightinterviewin November 2019 about his speculated connection to convicted sex offenderJeffrey Epstein, who died in prison while awaiting trial.

Prince Andrew, Sarah Ferguson, Sir Timothy Laurence and more royals at the Thanksgiving Service for King Constantine of Greece at St. George’s Chapel at Windsor Castle on Feb. 27, 2024.Andrew Matthews - WPA Pool/Getty

Prince Andrew, Duke of York, Sarah, Duchess of York, Zara Tindall, Sir Timothy Laurence, Mike Tindall and Anne, Princess Royal attend the Thanksgiving Service for King Constantine of the Hellenes at St George’s Chapel on February 27, 2024

Andrew Matthews - WPA Pool/Getty

Andrew has repeatedly denied any wrongdoing, and his transition away from royal responsibilities was made official when Queen Elizabethstripped him of his military titles and patronagesin January 2022 amid Virginia Roberts Giuffre’scivil sexual assault lawsuit. An out-of-court settlement for an undisclosed sumwas reachedthat February.

Looking ahead to the royal family’s Christmas plans on Dec. 25, PEOPLE understands that Prince Andrew, Fergie, Princess Beatrice, Princess Eugenie, and their respective families will not join King Charles at Sandringham because of the spy scandal. Though Andrew hasn’t held a working royal role for years, he continues to join his family for major ceremonial events and holiday celebrations.

Can’t get enough of PEOPLE’s Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

The York family had been widely expected to stay at Wood Farm on the Sandringham estate for Christmas like they did last year, but the plans have apparently pivoted.

PEOPLE understands that Beatrice, 36, and Eugenie, 34, had invitations to spend Christmas with their respective in-laws some time ago and have decided to do so. Their attendance at Sandringham had been in question, an insider says, while their parents will likely keep a low profile at Royal Lodge, their home on grounds of Windsor Castle.

Step inside the royals' holiday traditions with PEOPLE’s Special Edition:Christmas at the Palace: A Royal Family Album.

source: people.com