But his tough talk was largely overshadowed by the powerful image of athletes from both Koreas marching together under a unified flag and North Korean cheerleaders rooting for a combined women's hockey team, even though it was routed 8-0 by Switzerland in its first game on Saturday.
Kim Yo-jong is the North Korean leader's younger sister.
They met and lunched with South Korean President Moon Jae-in on the second day at the presidential complex.
Aides did not immediately say whether the invitation was discussed as the two sat next to each other viewing several race heats Saturday.
Moon can only visit North Korea on one condition - that Kim agrees once and for all to give up his nuclear weapons.
Winter Olympics: Korean athletes under unified flag at opening march
That display of unity comes after sustained overtures made by South Korea's president Moon Jae-in. The Winter Olympics typically see a bit more ratings vacillation than its summer counterpart.
The 28-year-old is a crucial part of the North's Games delegation led by Kim Yong Nam, North Korea's nominal head of state.
"We will not allow North Korean propaganda to hijack the message and imagery of the Olympic Games", he said in Japan earlier this week.
Pence is viewing the short-track competition, in which eight Americans are participating. In public comments, he repeatedly emphasized the strength of the U.S. alliance with South Korea and the need for North Korea to give up its nuclear ambitions.
North Korea's participance in the games in union with South Korea is significant in that it opens a door for a possible peace agreement and reunification of the Korean Peninsula in the future.
International Olympic Committee President Thomas Bach will visit North Korea after the Pyeongchang Winter Olympics, Reuters news agency reported Monday. One of the meetings resulted in a personal invitation for Moon to visit Pyongyang, delivered by Kim Yo Jong.
Kim's sister ends Olympic visit, leaving South to mull offer
The previous meetings came after rounds of global talks aimed at eliminating the North's nuclear program, which eventually failed. Another focal point is whether Kim Yo Jong has any message for Moon from her brother.
Prime Minister Shinzo Abe demanded of Kim Yong Nam that North Korea resolve the issue of Japanese nationals abducted to North Korea and abandon its nuclear and missile development programs.
Mattis' remarks come after South Korean President Moon Jae-in was presented a formal invitation to visit North Korea, potentially setting up the first meeting of Korean leaders since 2007. "It is important to move forward with the drills as planned", Abe said, according to South's presidential Blue House.
Moon, who endeavors to play the role as a "mediator" between the DPRK and the United States, is making great effort to sustain the momentum for inter-Korea talks and lay the foundation for direct dialogue between Washington and Pyongyang.
Kim Yo Jong was seated next to Moon and chatted from time to time and applauded the performers.
Precautions need to be taken against a situation in which a rift would emerge in global efforts to contain North Korea, while there is no progress being made on the North Korean nuclear issue.
Olympic results: Women's snowboard slopestyle final
According to Paul Carr of ESPN Stats and Info, Anderson became the first woman to win two Olympic gold medals in snowboarding. Neither Jenson nor Norendal was able to complete a clean second run despite the fact each had a strong showing in the first.
The officials toasted with the Korean liquor "soju" and were served buckwheat and potato pancakes, dried pollock soup, grilled halibut and other dishes. In January 1968, Kim Yo Jong's grandfather, founding North Korean president Kim Il Sung, sent a squad of North Korean commandos to Seoul who tried unsuccessfully to kill then-president Park Chung-hee.