"In light of what we all know, I presume that the vote of no confidence will go ahead, and Pedro Sánchez will become the prime minister and I want to be the first to congratulate him," Rojoy told Parliament.
"I will accept the result of the vote as the democrat I am, but I cannot agree with what has been done."
The vote began shortly after 11 a.m. local time (5 a.m. ET). If the motion passes, Rajoy's government will be the first in Spanish history to fall as a result of a no-confidence motion.
Sanchez, the leader of the main opposition Spanish Socialist Workers' Party (PSOE), would then automatically become prime minister.
As soon as the vote of no confidence passes, the President of the Parliament, Ana Pastor, will inform the king. From that moment, according to the Spanish constitution, Sanchez will be considered to be in office.
Sanchez has said that he would call for snap elections should he become Prime Minister.
Rajoy was greeted with applause by his fellow Popular Party members as he arrived in the chamber where lawmakers were debating the motion of no confidence Friday morning.
Visibly moved as he gave what may be his last address as Spain's leader, he also thanked those in his party and the Spanish people. "It has been an honor to be the Prime Minister of Spain, it has been an honor to leave a better Spain than I found," he said.
"I wish my substitute could say the same when his day comes, for the good of Spain."
Spanish Prime Minister Mariano Rajoy attends a session at the Lower House of Parliament in Madrid on Wednesday.
Before Rajoy's arrival, Rafael Hernando, spokesman for the Popular Party in Parliament, asked Sanchez to withdraw the vote of no confidence, saying that he was proud of Rajoy as Prime Minister. "He is an honest man who has worked tirelessly to rebuild a country that the PSOE left devastated," he said.
However, Margarita Robles, spokeswoman for the Socialist Party in Parliament said: "Spain does not deserve a prime minister who is not here and who lies in the courts of justice to cover corruption."
On Thursday, Rajoy chose not to attend a parliamentary debate on his future, when several opposition parties openly said their lawmakers would vote against him. That means 180 of parliament's 350 members, a clear majority, will voice their lack of confidence in the leader.
Rajoy's Popular Party has been plagued with corruption allegations for years. Rajoy has faced the humiliation of testifying in cases against members of his party, though he has not been among the accused.
The confidence motion was filed by Sanchez after a court convicted Rajoy's former aides of running slush funds to help finance Popular Party election campaigns. The court also questioned the credibility of Rajoy's testimony during the proceedings.
It was the first time in Spain that a party had been convicted and fined on corruption-related charges.
CNN's Laura Perez Maestro reported from Madrid