Generative Multimodal Models | YU Global

Post By

Joseph Rosenbaum, MSW

Generative Multimodal Models | YU Global

Golems

In a world rapidly transformed by technological marvels, the lines between myth and reality blur, revealing parallels that are as enlightening as they are cautionary. This series of blogs embarks on a journey through time and thought, drawing a compelling connection between ancient legends and the cutting-edge of modern technology. Today, we delve into the story of the Golem, a mythical figure from Jewish folklore, and unravel its unexpected relevance to today's advancements in Artificial Intelligence, particularly Generative Large Multimodal Models (GLMMs).

From the cobblestone streets of 16th century Prague to the digital highways of the 21st century, this blog invites you on a thought-provoking odyssey. An odyssey that questions not just the capabilities of our creations but also the responsibilities and consequences that accompany them. Join us as we explore how the legend of the Golem serves as a mirror, reflecting our hopes, fears, and ethical quandaries in the age of artificial intelligence.

The Golem of Prague

After the destruction of the second temple in ancient Judea, the Jewish people were scattered across the eastern hemisphere, forced to migrate to the Middle East, and Europe. There fragments of Judaism took with them the Torah - a fairly portable series of stories many now call the Old Testament.

Jews, like most refugees in history, were taken in by other nations devoid of any desire, and most were relegated to Ghettos. These enclosed areas were often overcrowded slums as Jews were not given rights to expand beyond the confines of these restrictive zones.

The upside was that nations hosting the Jewish people often left them to their own devices. They may not have received the sorts of rights or support that citizens would, but they were able to autonomously form their own micro governments and build their own institutions.

They were not only limited physically, but occupationally as well. The common stereotype of the Jewish money lender, for example, arose from the Christian-fueled restrictions of loaning money and collecting interest. The picture of the crooked nosed and greedy usurer arose due to religious nations exploiting a loophole whereby they could have the Jews perform this necessary economic task, and take their cut without diverging from their religious practices. This meant extractive interest rates were the norm driven by elitist greed, but those same elites could point the finger at the Jews. This scapegoating followed the Jews throughout the diaspora, eventually culminating in the holocaust.

The Jewish position in these communities was always an erratic one, driven by the whims of whatever rules were imposed upon them, or through fear mongering drummed up to distract the populace from weak leadership, so these strange people were seen as a drain on the cultural zeitgeist.

Pogroms came and went with the seasons, shuffling the Jews from one nation to another, forced to restart and relive this recycled trauma. The practice of identifying Jews through a badge was a practice in history long before the rise of the Nazis - a symbol to mark them as the perennial outsiders.

And yet, the culture flourished, roses growing from concrete. It was from this rhythm we find ourselves in 16th century Prague. The Jews were able to find a brief respite from oppression and set roots for a while. In this period of semi-stability rose Rabbi Loew, lovingly referred to as the "Maharal" - a type of nickname meant as a mix of "Moreinu Ha-Rav Loew" or "Our Teacher, Rabbi Loew".

Rabbi Loew, beyond being a spiritual guide for the community, was also a philosopher who preferred to blend rational thought with mystical insights in order to understand both Jewish religious texts, and their rich traditions.

The good Rabbi, though, is best known as a folk hero amongst the Jews not for his insights and good nature, but for the mystical tale of how he protected the community from the inevitable rise of anti-Semitic vitriol.

The rumors began small, but spread like wildfire across Prague - those nasty Jews were kidnapping children, and performing the ritual of blood libel. The meme of Jews as pedophilic vampires swept across the city, and resulted in a violent swell of rage.

The story goes that the Rabbi rushed to the banks of the nearby Vltava River and fashioned a clay Golem to protect the Jewish community from these attacks and false accusations of blood libel. The Golem, named Joseph, was brought to life through rituals and Hebrew incantations, and the insertion of a shem (a paper with sacred inscriptions) into its mouth or on its forehead.

The Golem, animated by this mystical magic, protected the community, scaring away the mob, and served as a guardian. Witnessing the immense strength of this creature frightened many of the Jewish people, fearful of both retaliation from the people of Prague, but also uncertain of the power of the Golem. What if it turned against them?

Only Rabbi Loew was able to deactivate the Golem through removing the shem, which he did every Sabbath to ensure it rested.

Soon the community's fears began taking shape in reality as the Golem started to act in some erratic, and even violent ways. It would take its instructions a little too literally, or misinterpret the will of the Rabbi. People would get hurt, or buildings damaged, and no matter how much specificity the Rabbi made with his commands to the Golem, the situation quickly unraveled.

Eventually, the Maharal made the difficult decision to remove the shem of the Golem for good, unwilling to risk the lives of his people. So he brought the clay protector to the attic where he stored the most precious and sacred of Hebrew texts, and took the shem from Joseph for the last time, hoping to both never have a need for the Golem's protections, but also saddened by his own inability to control the creature.

GolemAI

The "vibe" of a Golem is often maligned as a piece of lore. In no small part due to its connections with the Jews, and the sort of implicit anti-Semitism that surrounds us, but it's an important and relevant fable for us in a contemporary setting, and important to understand that the Golem was not evil or inherently bad despite the end of the story. Those who brought it to life were not evil either, focused only on protecting themselves.

There are many similar archetypes of this story. Someone, in hopes of protecting the people they love, unleashes something they don't understand. At first everything seems to be fine, but then it goes off the rails, and the story concludes with the moral of not toying with powers we do not fully understand, even if our intentions are pure.

I would like for us to look at Artificial Intelligence through this lens - one where AI is neither good nor bad, but instead a power with unknown downstream consequences that are worth taking some deep consideration as individuals, as society, and as humans.

Tristan and Aza Raskins, the co-founders of the Center for Human Technology, coined the term Generative Large Language Multimodal Models, acronymized as the letters GLLMMs - or Golems, which is a purposeful elicitation of the folklore, and Rabbi Loew's story. I decided to leave out the "Language" to just make it GLMMs, as these models have become multimodal from the ground up.

The story of the Golem, a clay figure brought to life by sacred incantations and commands, mirrors our contemporary journey with Generative Large Multimodal Models (GLMMs). Just as the Golem was animated by words, GLMMs spring into action upon receiving prompts. This semblance is not just in their activation but also in their inherent limitations. The Golem, despite its strength and initial purpose of protection, lacked a true understanding of the world and the nuances of human interaction. It would interpret commands too literally or fail to grasp the context, leading to unintended and often hazardous outcomes.

Similarly, GLMMs, while remarkable in their ability to generate content based on prompts, do not possess a genuine world model. Their responses, though sometimes astonishingly apt, can also be misaligned with the intent or complexity of the command. This disconnect can result in outputs that are at best irrelevant, and at worst, detrimental.

This parallel between the Golem and GLMMs serves as a modern parable. It cautions us about the allure of seemingly powerful tools and the critical need for clarity in our interactions with them, especially when we face danger or existential threat. It reminds us that without a deep understanding of their nature and limitations, we risk unleashing forces that, though created for benefit, may spiral beyond our control. In both the ancient tale of the Golem and the ongoing evolution of AI, the lesson is clear: power, without understanding, careful guidance, and above all wisdom, can lead to outcomes far from our intended goals.

This leads us to a thorny ethical bramble as GLMMs proliferate. We find ourselves at a crossroads reminiscent of Rabbi Loew's dilemma with his creation, the Golem. The Golem, once a protector, began to exhibit unpredictable and potentially dangerous behavior, posing a moral dilemma for the Rabbi. His ultimate decision to deactivate the Golem, acknowledging its power and the risks it posed, echoes the ethical considerations we must grapple with in the realm of AI.

In the world of Artificial Intelligence, we are constantly pushing the boundaries of what these technologies can do. But with this progress comes a responsibility akin to Rabbi Loew's – the responsibility to discern when our creations might be veering off course. What are the signals that indicate our AI systems are not aligning with ethical standards or societal values? How do we recognize when they are causing more harm than good? And most importantly, do we have the collective will to 'turn off' or significantly alter these systems, should the need arise?

These questions do not have easy answers. But as we continue to advance in AI, they are questions we cannot afford to ignore. Just like Rabbi Loew, we are tasked with the weighty responsibility of overseeing our creations, ensuring they serve the greater good rather than becoming a source of unintended harm.

Our journey with AI is not just a technological one; it is deeply ethical. It challenges us to consider not just what we can create, but what we should create, and how we govern these creations. The story of the Golem is not just a tale of a time long passed; it is a story for our times, urging us to tread thoughtfully and responsibly as we forge ahead into the unknown territories of artificial intelligence.

Access to Power

I will leave you with something to chew on.

In the lore of the Golem, creating such a powerful being was not a trivial task. It required deep knowledge, spiritual insight, and careful ritual. This complexity served as a natural barrier, ensuring that only those with the requisite wisdom and respect for the power they wielded could bring a Golem to life. But what happens when the power to create and control something as influential as a Golem – or in our case, advanced AI – becomes readily available to the masses?

Imagine a world where powerful AI tools are as ubiquitous as smartphones. Where creating and deploying an AI with significant capabilities is as easy as downloading an app. Maybe at the time of reading this, you don't have to imagine because it's already a reality.

This democratization of power brings with it profound questions. How do we ensure responsible use when the barriers to entry are so low? What happens when tools, capable of great influence and change, are in the hands of many, regardless of their understanding of the technology or its potential consequences?

This scenario isn't just hypothetical. With the rapid advancement and increasing accessibility of AI, we are fast approaching a reality where powerful technologies are within reach of a vast majority of the population. This shift necessitates a rethinking of our approach to AI governance, education, and ethical frameworks.

We must consider not only the safeguards necessary to prevent misuse but also the educational efforts required to ensure that users understand the potential impacts of their AI creations. The story of the Golem serves as a cautionary tale, reminding us that with great power comes great responsibility – a responsibility that must be shouldered not just by the creators of AI technologies, but by all who wield them.

As we step into this new era of accessible power, we must ask ourselves: are we ready to handle the responsibility that comes with it? How do we prepare for a world where the creation and control of powerful technologies are no longer the domain of a few, but the privilege and challenge of many?

Stay updated on the latest in continuing education

Subscribe to receive updates to our blog.

Recent Posts

In today's rapidly evolving world, we're inundated with information from countless sources. From global events...

August 14, 2024

Have you heard the term “step change” more frequently? Here’s why.

July 18, 2024

In 2024, the role of an instructional designer has become increasingly crucial. Instructional designers are the...

July 1, 2024