AUDIE CORNISH, HOST:
Fossils have provided a detailed record of early human skulls, but not the brains inside them. So researchers are using genetic material taken from those fossils to study how the human brain evolved. NPR's Jon Hamilton reports on an effort to grow human brain cells containing an ancient gene.
JON HAMILTON, BYLINE: The gene is called NOVA1. It's among the genes that have changed significantly since Neanderthals roamed the Earth. Alysson Muotri of the University of California, San Diego says NOVA1 plays an important role in brain development.
ALYSSON MUOTRI: So that gene would be perfect to start measuring if they can change the way the brain develops.
HAMILTON: To find out, Muotri and his team turned to brain organoids, or mini brains. They're tiny spheres of human brain tissue that grow in the lab and mimic early brain development. But Muotri says existing organoids only had the modern version of the NOVA1 gene.
MUOTRI: So we swap the archaic version for the modern version.
HAMILTON: And the team started growing organoids carrying this archaic gene. Muotri says they were clearly different.
MUOTRI: The archaic version of the gene changes the way the shape of these organoids are formed.
HAMILTON: The surface of each sphere was uneven instead of smooth. Also, Muotri says, individual cells behave differently.
MUOTRI: The archaic version of the brain organoid, we see an acceleration of maturation. So the neurons start to get more active at very early stages compared to modern humans.
HAMILTON: A brain that matures quickly is more capable at birth, but it's less likely to continue developing during childhood, the period when modern humans acquire important social skills. Muotri says there's probably a reason why humans with the new variant of NOVA1 survived while those with the old version died out.
MUOTRI: It might suggest that sometime during evolution, we acquired that mutation, and it bring us tremendous advantage to be able to have this complex brain later in life.
HAMILTON: That idea is consistent with what some other scientists have learned from studying ancient genes in a different way. Dr. Karen Berman of the National Institute of Mental Health is part of a team that's been studying the Neanderthal genes that are still present in many humans, especially those of European and East Asian ancestry.
KAREN BERMAN: It's very exciting to be able to use our knowledge of archaic DNA to really explore what makes us who and what we are.
HAMILTON: Berman says people with more Neanderthal genetic material tend to have skulls more like those of Neanderthals. And she says there are hints that Neanderthal genes emphasized brain networks devoted to visual and spatial abilities rather than social interactions.
BERMAN: It may have been the lack of these social networks that led to the Neanderthal dying out.
HAMILTON: Berman's research partner, Dr. Michael Gregory, says it's easy to imagine how Neanderthal brains could have become outdated.
MICHAEL GREGORY: If they went foraging for foods or hunting, they could find their way back and forth to different places better. But they couldn't form groups quite as well, in theory, to fight common predators or something like that.
HAMILTON: And group cooperation has been critical to the success of modern humans. But Gregory, a psychiatrist, says all this research on ancient brains is about more than human history.
GREGORY: The thing that drives us is not just how did things happen to develop over time and over evolution, but also how did they go awry and cause neuropsychiatric diseases, and why?
HAMILTON: Gregory hopes that by answering those questions, scientists will be able to develop better treatments for problems like schizophrenia and autism. The new research appears in the journal Science. Jon Hamilton, NPR News.
(SOUNDBITE OF GRAMATIK'S "MUY TRANQUILO") Transcript provided by NPR, Copyright NPR.