Wagner James Au / New World Notes

Originally published on my Patreon here

The most fascinating story about next gen AI in technology may not be about DeepSeek or whatever OpenAI is announcing, but reside in a 21 year old metaverse platform which has something no LLM can boast about: A simulated virtual world with terabytes of meta-tagged data constantly evolving.

That's my take, at least, talking with Linden Lab heads Brad Oberwager and Philip Rosedale recently. And while the company has added its own official AI-powered characters last year, the user community has been creating them for many years, even long before OpenAI existed. They recently got a demo from an SL community creator who connected LLMs to an upcoming RPG that blew their minds:

"The whole experience that he had built using AI was really very strong," Philip tells me. "And I think that's one of those cases where, if we can just get people to him, the ones that want to [play his role playing game] it was just awesome." (More on that game later.)

They also thought, he added: "Okay, let's modernize this and make it easily expand upon this stuff."

But doing that means taking incredible caution: "We're running a lot of tests on characters that are infused with AI personalities," Oberwager explains. "They're sort of next gen NPCs. We ran a test, we got feedback, we took the test down. It's not like it went sideways, but some people didn't like some things, but that's the point of the test."

Read on for more of the perils, and promises:

Brad Oberwager: One of the things that people aren’t kind of getting is, the horse is out of the stable here. Like people can right now, put a character in our world that is generating chat through GPT 4 or 3 or whatever they want to use. We can't stop that unless we make it against our terms and conditions. 

And what we're trying to figure out is, how do we give people the opportunity to have these NPCs, to build these characters, without all of a sudden, you know — the classic one that everyone says is, “Without a bunch of Nazis running around.” 

So how do we do that? And so one way is to work closely with some of these extremely well known folks [community developers] who have a lot to lose if they violate our Terms and Conditions, and work with them. 

The other way is to offer our own solution. So to offer a solution, pick the LLM you want to use, GPT, doesn't matter to us, but it has to go through our very, very basic test that eliminates the ability for you to create a bunch of nonsense. And then make all other [applications], either you get approval or they're illegal.  We have a lot to lose by allowing AI to run rampant through Second Life. 

Philip Rosedale: It's kind of funny to observe, philosophically, that Second Life is the test environment where humans and AIs will probably interact; it'll generate a lot of learning about the real world, right? The only way that humans and AIs coming to a head in the real world is Waymo, which is interesting… 

But there's so many different ways that humans and AIs are likely to interact, say, in a decade, and most of those ways are going to come up to the forefront, like right now in Second Life, [as] Brad was saying.

We put up a character designer a couple days ago, and we just missed something, which was that if you were a character that was an AI, and you were in a group already — because basically the way the thing works is you attach the AI to your alt and then to your character, so you attach an AI that you tried, or saw the designer that we put up, but you basically have a new page to go to that lets you prompt a character. So you can role play, create, like a background prompt, and then you attach that AI to an alt, and we call that a character. And then you turn it on and it's in-world, and it'll stay involved, you know, it'll just hang around in the world. 

The thing we had missed was that, if your alt account is already in a large group, and somebody in the group talks to that alt, the AI will start talking back to everybody in the group. 

Which, I mean, you kind of have to laugh at that one, but you can imagine that was an immediate. “Oh my God.” And so, like Brad said, we turned it off within an hour or whatever. 

These are the experiments that are going to happen that are going to be really interesting, right? Because, the question of, “Under what conditions do you want to talk to an AI as an avatar in Second Life?” I mean, I don't know, this is a very interesting conversation. In some ways, more avatars are good, and it costs us more… in some ways, more avatars, to the extent that they're boring to interact with, or take you away from real people, they’re bad. 

Wagner James Au: That's been a problem from the design aspect, that Second Life is not presented as a traditional MMO. Because traditional MMOs have NPCs, and they are quest givers, or they'll give you rewards, and so you interact with them with that in mind.

But within Second Life, you're expected to meet other people. I mean, in SL, there's been all kinds of bots forever, but that's a constant complaint from  users. You know: “I saw all these green dots [on the map], and thought it was a party, but it's just bots.” So I think the problem is getting worse. 

And I think on a larger level, in Silicon Valley, they don't realize that adding AI to a virtual world, it doesn't really add any value. There's not any proven value beyond how we already know it, as either NPCs or, you know, for procedural generation. So I'm really afraid that this is going down the wrong path. 

PR: Yeah, but like you said, like Brad said, the AI horse is out of the barn, in the sense that people already have, like you said, in MMOs historically, people have basically really bad AI, which is just a quest giver. And that's how you interact with NPCs. Which, which we all, as gamers, know. And so, I think the trick is, the bar is pretty low, we just need to kind of climb up the hill from there, right? 

There's got to be a way to use LLMs, and LLs that are better informed by objects, the environment, and metadata. One of the things that’s so cool about Second Life is that all of the objects in the world have a lot of text attached with them, which has been created by all the content creators. So we have hundreds of millions, and billions of objects in inventory. And we have hundreds of millions of objects in the world, all of which have pretty good metadata on the text, which is one of the things that AIs can use to make sense, to make meaning, of the world they're in. 

So one of the cool things about Second Life, like you couldn't put AI [in the real world] and have them learn in the way that they could learn in Second Life. Because in Second Life, they can not only see the graphics, but they all the objects are segmented and tagged and named in ways that they can understand, because they were literally [described with metadata].

So I think we have to be very thoughtful about the IP implications of that. That's why, right now, the character stuff we did, those are just normal LLMs. Like Brad mentioned, there's a couple different ones, like Llama,  that are talking in the world, trained on content in the world. They don't have any special access to world information that people don't have and [we're] not training the model on stuff that people have in the world. But at some point, it's going to be an interesting environment, because Second Life is so well packed that people are going to be able to do entirely [new things]. 

Brad has this favorite story where there was a case where we did a little bit of metadata investigation on this one project. And Brad went in there to demo it, and he said to an NPC AI, “Hey, you know what you do around here?” 

And the [NPC says], “Do you want to go surfing?” And Brad was like, “Yeah.” 

And [the NPC] clicked on a surfboard giver and rezzed the surfboard for Brad. 

And Brad said to the team that was working on it, “How did it do that?” 

They're like, “I don't know. It figured it out.” 

So it was one of those LLM moments where it had just parsed the nearby object data and figured out how to click on something, which is pretty, pretty neat.

This is part of a four part conversation covering many topics: Read Part 1 here; read Part 2 here; read part 3 here.