(CNN) - When Terrence Byrd was stopped by police in 2014 for a minor traffic infraction, he couldn't have expected that his case would eventually come before the Supreme Court.
In the heat of the moment, he had to deal with the fact that he was the unauthorized driver of a rental car.
And that there were 49 bricks of heroin in his trunk.
After his arrest, however, Byrd's lawyer would seek to suppress the evidence against him, citing the Fourth Amendment's guarantee against unreasonable search and seizure.
The government claimed it did not need his consent to search the car because Byrd had no reasonable expectation of privacy due to the fact that his name was not listed on the rental agreement.
Byrd was subsequently convicted of drug charges and sentenced to 120 months in prison.
Four years later, Byrd's appeal has made it to the highest court in the land and the justices will determine whether Byrd had diminished privacy rights under the Fourth Amendment.
Privacy experts are watching to see whether the justices will broaden the government's power to search under the Fourth Amendment. Some fear that a ruling against Byrd might disproportionally affect lower-income Americans, who are more likely to depend upon rental cars for everyday travel because they can't afford their own vehicles.
In court Tuesday, Justice Sonia Sotomayor was concerned about the impact a ruling against Byrd would have on police power. She noted that even though Byrd's name was not on the rental agreement, his friend, who had rented the car, allowed him to drive it.
"If we rule that someone without permission has no expectation of privacy even when the renter has given it to them, then what we're authorizing is the police to stop every rental car and search every rental car, without probable cause," she said.
Justice Stephen Breyer worried about the impact of the court's opinion on other contract-related cases concerning, for instance, "houses, apartments, sublets, summer cabins," all of which are governed by "understandings" and "leases."
Thinking out loud, Breyer tested several different rules hoping to find something that would be easy for police officers to understand. "Fourth Amendment law is too complicated in a sense already," he concluded.
Justice Elena Kagan noted that the contract violation -- someone driving a rental car even though his name wasn't on the list -- is not that rare.
"Let's be frank," she said, it's "not uncommon."
"What eliminates the right of privacy that you would normally get by opening up the car of a door and sitting in the front seat and turning the ignition key?" she asked.
On the other hand, Chief Justice John Roberts looked at the case from the perspective of the police officers making the stops and being forced to determine whether the driver had permission from the renter to drive the car.
"Are (police) supposed to conduct an inquisition?" he asked.
The case dates back to 2014, when a police officer in Pennsylvania became suspicious because Byrd was sitting so far back in the driver's seat he was barely visible. The officer noted that the Ford Fusion was a rental car and began to follow it on a four-lane highway. When Byrd failed to properly move into the right lane after passing a slower truck, the officer pulled him over for the minor violation, according to police.
After some questioning, Byrd admitted that he had a marijuana cigarette in the car, according to police. The officers informed him that because his name was not on the rental agreement they could search the car without his consent. Later, police said they found the heroin. Among the charges was one count of possession of heroin with the intent to distribute.
A three-judge panel of the 3rd US Circuit Court of Appeals ruled against Byrd, citing its own precedent "that society generally does not share or recognize an expectation of privacy for those who have gained possession and control over a rental vehicle they have borrowed without the permission of the rental company."
In briefs to the Supreme Court, a lawyer for Byrd emphasized that his client's girlfriend had allowed him to drive the car that she had rented.
"She has given him both possession and control of the car, and he reasonably believes that he can exclude strangers and the government from intruding upon his private personal family possessions stored in the car," Robert M. Loeb wrote. Loeb stressed that Byrd's constitutional protections should not materially change because it turned out that he was not the authorized driver under a rental agreement.
Lawyers for the government, however, argued in court papers that he "did not own the car, had not rented it, and was not allowed to drive it."
"He cannot assert Fourth Amendment rights to object to its search," Solicitor General Noel Francisco said.
The American Civil Liberties Union filed a brief in support of Byrd arguing that the lower court opinion would "severely curtail the scope of the Fourth Amendment" and affect a "broad swath of the population, especially individuals who have come to depend on rental cars for everyday travel because they cannot afford to purchase their own vehicle."