In linguistics this has been a major conundrum, especially in earlier centuries when there was no worldwide communication; in science fiction it has been an occasional but recurrent theme in the context of “first contact” stories. These two situations may sound similar, but they invite very different analyses.
In the branch of linguistics known as “fieldwork,” a linguist would move temporarily into an unfamiliar culture — usually one whose language had not been much (if at all) studied before — and would figure out the language with the help of cooperative locals. Think of a European scholar investigating some remote tribe in New Guinea or central Africa. The only reason that this endeavor is even possible is that there are many universals among human languages. Without the possibility of infinite variety, the linguist knows what sorts of things to look for, ranging from the existence of nouns and verbs to certain kinds of linguistic rules. Every human language has numbers, every human language has certain roles that nouns can have in a sentence (subject, object, etc.), and every human language has a preferred basic word order (subject-verb-object, abbreviated SVO, in English for instance). Of course there are different types of languages, and specific rules vary, and vocabulary can completely vary, but possibilities are still constrained.
Not so in the science fiction of first-contact stories. By definition the aliens are not human, so their languages don’t obey the constraints of (human) universal rules. If a species is at all humanoid — the Klingons, for instance — people may speak a humanoid language obeying some of the human constraints. (More on Klingon at another time.) Then, speaking of Star Trek, all you have to do is use the Universal Translator device after a remarkably brief period of training! If a species is not the least bit humanoid, then all bets are off. The 2016 movie Arrival is perhaps the best example here. I was reminded of it yesterday while engaging in my see-all-the-important-episodes-of-Star-Trek pandemic project; the TNG episode “Darmok” presents an encounter (second contact, not first, but never mind) with the Tamarians, an alien race that speaks a language that the universal translator has trouble with. It has trouble not because of vocabulary or syntax but because they speak entirely in cryptic metaphors, such as “Darmok and Jalad at Tanagra.” Without the shared experience of the Tamarians, humans cannot figure out the import of these metaphors. Picard and the Tamarian captain are deliberately stranded on an isolated planet, where they have to learn to communicate without a shared basis, and where all communication with the outside world has been cut off, Outward Bound style. For more information, read the article on the Tamarian language in memory-alpha.fandom.
These two situations — earth-based fieldwork and science-fiction first contact — collectively remind me of Willard van Orman Quine’s famous essay on Gavagai, summarized here by an anonymous Wikipedia author:
The gavagai thought experiment tells about a linguist, who tries to find out, what the expression gayvagai means, when uttered by a speaker of a yet unknown, native language upon seeing a rabbit. At first glance, it seems that gavagai simply translates with rabbit. Now, Quine points out that the background language and its referring devices might fool the linguist here, because he is misled in a sense that he always makes direct comparisons between the foreign language and his own. However, when shouting gavagai, and pointing at a rabbit, the natives could as well refer to something like undetached rabbit-parts, or rabbit-tropes and it would not make any observable difference. The behavioural data the linguist could collect from the native speaker would be the same in every case, or to reword it, several translation hypotheses could be built on the same sensoric stimuli. Hence, the reference between the term gavagai and its referring object is language- or framework-dependent, and therefore inscrutable.
Finally, as I was drafting this post, I discovered an old post on this very subject in All Things Linguistic! It’s very informative. Do read it.