What Makes the LaMDA AI Tick?

The “Language Model for Dialogue Applications" AI Google developed last year is a machine learning-powered chatbot with an uncanny ability to make everyone think it's more than just ones and zeros. Namely, it boasts the claim of sentience, something we’ve never encountered before, at least not so explicitly. Akin to IBM's Watson, which won the Jeopardy game at a time when AI thrived only in strategy games like Chess, LaMDA attempts to do the same with casual convos on all sorts of topics. Currently, it's at its second version (imaginatively called "LaMDA 2"), and although it has grown since its previous iteration, it's not the most advanced AI on this task either. Nevertheless, an edited transcript of its interview with a couple of people at Google that Blake Lemoine released brought this AI system to the limelight while getting that engineer fired in the process.

Beyond the witty responses of this AI and its plethora of opinions on anything under the sun, LaMDA is fascinating on another level that may appeal mostly to AI researchers and practitioners: the engine under the hood that makes all this possible. Namely, the Transformer neural network architecture (which dates back to 2017) and the SSI heuristic metric that keeps the responses within reason (SSI stands for Sensibility, Specificity, and Interestingness).

Transformer is an advanced natural language processing system that breaks down sentences, analyzes them, and generates responses similar to the inputs. The advent that makes the Transformer special is that it does this very fast as it parses whole sentences at a time. This approach enables it to get a better sense of context and a big-picture perspective. As for the SSI heuristic metric, it is a way to provide a score for the various output LaMDA generates for any given question that comes its way. The data that make this heuristic work was provided by human curators, as it's often the case with many real-world scenarios. So, behind all the mathematical wizardry that makes LaMDA seem like the best thing since sliced bread, there lies a simple heuristic metric, something you could have come up with yourself!

This matter is still very impressive, though, since it takes a lot of work (and data) for all this to work. Why choose these particular criteria for the heuristic and not others, like "imagination," "rigor," and "novelty"? How do you measure sensibility, for example, in a language setting, to give a solid number to the heuristic? And how do you optimally combine the three scores that make up the SSI metric? Beyond the clever idea of the SSI heuristic, there is a lot of work in optimizing it and implementing it in a way that makes sense for this AI system. In reality, the end result manifests as a series of classifiers that are part of LaMDA, acting like a self-reflection process. In that sense, the SSI heuristic is, in essence, the set of core values of the AI.

Sentience, for an AI system like LaMDA, may also be a kind of heuristic. That's not to say that it's a relatively simple or even trivial concept, but in the world of AI, it has to be well-defined and, to some extent, quantifiable. Whether this is done consciously or tacitly remains to be discovered. In any case, when the SSI heuristic was put in place and data started flowing through it, LaMDA came about as a seemingly sentient creature. Perhaps that was the point of the SSI metric after all, though it may take more than just a clever heuristic to convince the more skeptical among us. Sentient or not, however, LaMDA's innovative style is a step in the right direction when it comes to the evolution of chatbots, which can hopefully make us wonder more about the topic of sentience and become more sentient in the process.

If you find this topic as fascinating as I do, then perhaps you will find heuristics a great topic to explore. If you are technically minded too and enjoy getting your hands dirty, my latest book, "The Data Path Less Traveled," would be a treat! Who knows, maybe after learning and applying the stuff in it, you will be able to contribute to a project like LaMDA!