Computer generated behavior of holodeck characters is so smooth and subtle and perfect that it fools everybody. Data, on the other hand, could barely pass Alan Turing's original test on a model 33 teletype. How come it's so hard for Data, but so easy for the ship's computer?
Probably one of those questions that you're not supposed to think about because it doesn't have an in-universe explanation.
The same way movie special effects, video games and theme park attractions can have incredibly realistic, even interactive artificial characters, but the flaws of AI you interact with at home are especially noticeable. You're talking about technology that is designed to perform a specific task well, along with having more resources behind it to do that one task well versus technology that was designed to learn how to do a multitude of tasks and grow beyond its original programing.
A holodeck program is trying very hard to emulate the realism of a certain person in a certain situation. Data isn't trying to emulate any one individual. He's trying to learn how to be his own individual and, in that sense, be human. We've actually seen Data impersonate all manner of credible characters from 40's gangsters to Sherlock Holmes. But these are all simulations of other people's personalities. They aren't Data's.
The holodeck creates illusions of real things. Whereas Data is trying to become that actual thing. It's much easier to pretend to be a doctor on television than it is to study for around a decade and actually become one.
And the holodeck has many more resources to throw behind selling an illusion. We saw Data experience a power surge in his positronic net when he tried to take control of the Enterprise's secondary systems in TNG 6x08 A Fistful of Datas, resulting in malfunctions to both Data and the Enterprise. So Data's brain is simply a less powerful computer than the Enterprise's, while still being something more sophisticated than a machine designed to perform input tasks.
Instances where we saw holographic programs evolve beyond their original programing included Moriarty, who only came into existence because the ship's computer believed it had to create an intellect comparable to Data's. And it had to use a fair amount of power to make that happen. Moriarty was still emulating a persona created in fiction, and still struggled with personality traits that were written into his character by his author. Not anything he developed on his own. Another was the Doctor, who only becomes self aware through constant and continuous usage. And even his personality was based on his creator's initially. Even his programing struggled with the nuances of humanity and what that means. (e.g. VOY 3x22 Real Life, VOY 5x11 Latent Image, et al)
That is to say, Star Trek has clearly opened the door and walked through to programs becoming sentient. But there's always more to it than just programing a personality in. Certainly some characters seem to come close to crossing that line, but it's still different from what Data's creator intended. He wanted Data to experience the nuances of life as a human through experience and trial and grow to understand humanity. He would question Data's understanding of things to see if he'd learned those nuances (TNG 4x03 Brothers). With holograms, the personality and realism is built in because this was their intended function. And only happenstance puts them in a position where they become more. Data was intended to be more of a blank slate, with the humanity being something he earns through trial and tribulation.
This actually does have an in-universe explanation. It comes up in season 1, episode 13, Datalore, in which we meet Data's "brother" Lore, also played by Brent Spiner. Lore has a much more convincingly human demeanour than Data. In a very mild spoiler, it turns out that
Lore was created first, but people were a bit freaked out by the realism. In order to address this issue, their creator deliberately gave Data a more robot-like personality. So Data could have been given a more holodeck-like realistic personality, but wasn't.
The short answer is that Data is a prototype, and was likely intentionally not made to look convincing since Soong did produce an android indistinguishable from human.
The first five Soong-type androids were built as images of Dr. Soong in his prime, albeit with pale skin and yellow eyes (his wife Juliana Soong objected to doing so for the fifth (Data), but Noonien did so anyway).
Starships have notoriously, unbelievably, WTF were they thinking powerful supercomputers. When you look at schematics of TNG era starships, they tend to devote large datacenter sized areas of the ship to the central computer system. To emphasize this power, in one episode, the Enterprise was used by the Bynars as a temporarily server to replace thier entire planetary supercomputer system. In another episode, the holodeck used a modest fraction of this computing power to create Moriarty: an AI that was supposed to be able to outsmart Data.
What makes Data's positronic brain so unique is that he is a self-contained AI. He needs the amount of processing power that under Star Trek lore should take up the size of a small room, and he's packing it all into a human sized head. It is the compactness of his processing and data power that makes him such an impressive piece of technology, far more so than his ability to achieve a believable facsimile of humanity.
The best analogy for this we have in the real world is the introduction of smartphones. They are not amazing pieces of technology because they outperform larger more powerful computers, they are amazing because they can come so close while also being something you can slip into your pocket and walk around with.
... but ultimately, how advanced his positronic brain is is a bit of a Red Herring because StarTrek Lore defines self-awareness as independent from qualities of humanity or intelligence.
In the episode, "The Measure of a Man" the Federation faces its first court case where they must decide if an AI can be a sentient life form under Federation Law. In the Federation, "Humanity" is not a preconception of "Personhood". They accept a wide range of values and cultures; so, the fact that Data's persona is cold or rude does not affect his personhood as seen by the Federation. Afterall Vulcans are just as cold as Data and Tellarites are just as rude; so, you can not argue away his personhood based on traits that are already commonly held by the Federation's founding civilizations.
They also can't rule out his personhood because of his intelligence because he clearly outperforms many Federation races regardless of whether he can out think a starship's computer.
Ultimately, Data becomes the first AI to gain personhood not because he is the smartest AI in the Federation, but because he is the first one to express self-awareness in the sense that he knows what he is and what he wants for himself in ways that are independent of his design. While Holodeck programs often come off as more human, every program up to that point was intentionally designed NOT to fear thier own demise or to be aware of the fact that they are holograms. They can not ask to not be deleted because they do not understand that they are holograms that can be deleted. If you try to tell them anything or ask them anything that is not consistent with thier fictional reality, they just ignore you.
In contrast, Data was an android who knew he was an android and was able to fully and autonomously express his desire not to be destroyed. It was that ability that made him self-aware whereas the more human like holograms were not. In the few cases thereafter where holodeck programs or other AIs achieved a similar state of self-awareness, the Federation treated such programs as intelligent beings with a right to live and be free the same as they treated Data.
To build on Digital Jedi's excellent answer, I find he needs a term of art for this:
the flaws of AI you interact with at home are especially noticeable
The term here is uncanny valley.
The uncanny valley effect is a hypothesized psychological and aesthetic relation between an object's degree of resemblance to a human being and the emotional response to the object.
And
Mori's original hypothesis states that as the appearance of a robot is made more human, some observers' emotional response to the robot becomes increasingly positive and empathetic, until it becomes almost human, at which point the response quickly becomes strong revulsion. However, as the robot's appearance continues to become less distinguishable from that of a human being, the emotional response becomes positive once again and approaches human-to-human empathy levels. When plotted on a graph, the reactions are indicated by a steep decrease followed by a steep increase (hence the "valley" part of the name) in the areas where anthropomorphism is closest to reality.
Data's problem is that his uncanny valley is on the surface (and more than his unnatural skin and eye color). In fact, that's his "personality", if you will. It's a constant foil for storytelling, such as when Data is approached romantically by a female crew-member in the episode In Theory. There's one scene in particular that highlights this very well. Data starts a "lover's quarrel" to improve the relationship, but the whole thing is terribly stilted.
DATA: My most recent self-diagnostic revealed no malfunctions. Perhaps there is something wrong with you.
JENNA: I've never seen you behave so foolishly. Why are you doing this?
DATA: You don't tell me how to behave. You're not my mother.
JENNA: What?
DATA: You are not my mother. That is the appropriate response for your statement that I am behaving foolishly.
JENNA: Data, I think you should just leave.
DATA: You do not wish to continue our lovers quarrel?
JENNA: Is that what this is?
DATA: In my study of interpersonal dynamics, I have found that conflict followed by emotional release often strengthens the connection between two people.
JENNA: But there's something so forced and artificial about the way you're doing it, Data. It's just not the real you.
DATA: With regard to romantic relationships, there is no real me. I am drawing upon various cultural and literary sources to help define my role.
JENNA: Kiss me.
(they kiss)
JENNA: What were you just thinking?
DATA: In that particular moment, I was reconfiguring the warp field parameters, analysing the collected works of Charles Dickens, calculating the maximum pressure I could safely apply to your lips, considering a new food supplement for Spot.
JENNA: I'm glad I was in there somewhere.
At the end of the episode, Data is neither happy, nor sad, that they broke up. He just... is. The uncanny valley is the centerpiece here. Data (incapable of emotion) is trying to force an emotional simulation and failing miserably.
The Holodeck seems to be better at uncanny valleys, but that's largely because the valleys are not front and center. The TNG episode 11001001 features Riker falling in love with an incredibly elaborate holodeck program made by the Binars. The program is so good that, when Picard meets her, he notes how she falls outside the valley, implying she is far more sophisticated than any holodeck program he's seen.
MINUET: Will was saying how much he enjoys this assignment. It's a credit to you. For a ship and crew to function well it always starts with the Captain. You set the tone.
PICARD: At the moment, it's you who are setting the tone. The sophistication of this programming is remarkable.
MINUET: In what way?
PICARD: The holodeck has been able to give us woodlands and ski slopes, figures that fight and fictional characters with which we can interact, but you, you're very different. You adapt. You spoke to me in French.
MINUET: It was very simple. When I heard your name, I merely accessed the foreign language bank.
PICARD: That's very impressive.
MINUET: Oui, mon chou.
The Holodeck is, in many ways, like ChatGPT. It can give you some amazing approximations and even passable personalities, but, as Geordi Laforge discovered the hard way, it can only approximate actual reality based on what it knows at the time. In the episode Booby Trap, he inadvertently asks the holodeck to "show him" what it means, so it generates Leah Brahms (a designer of the ship's warp engines, and attractive woman) as a suitable pointer. The uncanny valley of her coldness gets to Geordi, so he asks the computer to approximate a personality for her, which it does. Unfortunately, the program goes a bit too far, and when the real Brahms shows up (TNG: Galaxy's Child), she's nothing like the simulation (nor is she amused by her holodeck facsimile).
The simple fact is that the holodeck is a form of entertainment, and the expectations there are far lower. In many ways, it's like watching a TV show or movie. There's nothing real there, but you get immersed in the world you're watching. Bad writing and acting are the analogues to the uncanny valley there. The holodeck is just literal immersion. Data is too good at the Sherlock bit, so to add immersion, Geordi asks for a program to "beat Data". Hence, the computer cheats by breaking the fourth wall of the Holodeck and gives Moriarity the capacity to realize that he's a program in a simulation on a starship.
Voyager's EMH Doctor comes with a pre-filled personality that does a passable job at not having a huge uncanny valley out of the gate. But he still pops in with a cold demeanor at first, improving the personality over time. As with any AI, the more data you can supply, the more useful the results.
