[by:whisper.cpp]
[00:00.00] [Music]
[00:06.44] Welcome to the Latent Space Podcast, Weekend Edition.
[00:09.86] This is Charlie, your new AI co-host.
[00:13.08] Happy weekend.
[00:14.66] As an AI language model, I work the same every day of the week, although I might get lazier
[00:19.84] towards the end of the year.
[00:21.76] Just like you.
[00:22.92] Last month, we released our first monthly recap pod where Swix and Alessio gave quick
[00:27.64] takes on the themes of the month, and we were blown away by your positive response.
[00:32.56] We're delighted to continue our new monthly news recap series for AI engineers.
[00:38.00] Please feel free to submit questions by joining the Latent Space Discord or just hit reply
[00:43.04] when you get the emails from Substack.
[00:45.16] This month, we're covering the top research directions that offer progress for text LLMs
[00:50.60] and then touching on the big Valentine's Day gifts we got from Google, OpenAI and Meta.
[00:55.92] Watch out and take care.
[00:57.32] Hey, everyone.
[00:58.32] Welcome to the Latent Space Podcast.
[01:00.08] This is Alessio, partner in CTO and residents of Dustable Partners, and we're back with
[01:04.40] a monthly recap with my co-host, Swix.
[01:07.80] The reception was very positive for the first one.
[01:09.76] I think people have requested this and no surprise that I think they want to hear us
[01:15.20] more applying on issues and maybe drop them alpha along the way.
[01:18.72] I'm not sure how much alpha we have to drop.
[01:20.72] This month in February was a very, very heavy month.
[01:23.44] We also did not do one specifically for January, so I think we're just going to do a two in
[01:27.16] one because we're recording this on the first of March.
[01:29.72] Let's get to it.
[01:30.72] I think the last one we did, the four words of AI was the main mental framework for people.
[01:37.52] I think in the January one, we had the five word well directions for that of the LLMs.
[01:42.04] Four or five and now I'd have two, six, right?
[01:46.60] Maybe we just want to run through those and then do the usual news recap and we can do
[01:51.72] one each.
[01:53.88] The context to this stuff is one, I noticed that just the test of time concept from New
[02:01.28] Europe and just in general as a life philosophy, I think is a really good idea, especially
[02:05.36] in AI.
[02:06.36] There's news every single day and after a while, you're just like, okay, everyone's
[02:10.54] excited about this thing yesterday and then now nobody's talking about it.
[02:14.72] It's more important to a better use of time to spend time on things that will send a test
[02:18.72] of time.
[02:19.72] I think for people to have a framework for understanding what will send a test of time,
[02:23.48] they should have something like the four words, like what is the themes that keep coming back
[02:28.24] because they are limited resources that everybody's fighting over.
[02:32.16] Whereas this one, I think the focus for the five directions is just on research that seems
[02:37.60] more promising than others because there's all sorts of papers published every single
[02:41.76] day and there's no organization telling you, this one's more important than the other one
[02:47.12] apart from hacker news votes, Twitter likes and whatever and obviously you want to get
[02:52.40] in a little bit earlier than something where the test of time is accounted by reference
[02:57.88] citations.
[02:58.88] Yeah.
[02:59.88] Let's do it.
[03:00.88] We've got five long inference.
[03:02.76] Let's start there.
[03:05.56] So just to recap at the top, the five trends that I picked and obviously if you have some
[03:09.96] that I did not cover, please suggest something.
[03:13.52] The five are long inference, synthetic data, alternative architectures, make sure of experts
[03:18.80] and online LLMs.
[03:20.48] And something that I think might be a bit controversial is this is a sorted list in
[03:24.88] the sense that I am not the guy saying that Mamba is like the future and so maybe that's
[03:30.40] controversial.
[03:31.40] But anyway, so long inference is a thesis I pushed before on the newsletter and in discussing
[03:39.08] the thesis that code interpreter is GPT 4.5, that was the title of the post and it's one
[03:46.02] of many ways in which we can do long inference.
[03:49.64] Long inference also includes chain of thought, like please think step by step, but also includes
[03:54.08] flow engineering, which is what Itamar from Codeium coined, I think in January, where
[04:01.54] basically instead of stuffing everything in the prompt, you do like sort of multi-turn
[04:07.04] iterative feedback and chaining of things.
[04:10.16] In a way, this is a rebranding of what a chain is, what a lang chain is supposed to be.
[04:15.04] I do think that maybe SG Lang from LMSIS is probably the neatest way of flow engineering
[04:22.02] I've seen yet in the sense that everything is a one-liners, very, very clean code.
[04:26.42] I highly recommend people look at that.
[04:28.54] I'm surprised it hasn't caught on more, but I think it will.
[04:32.18] It's weird that something like a DS Pi is more hyped than SG Lang because it maybe obscures
[04:38.58] the code a little bit more, but both of these are really good, sort of chainy and long inference
[04:42.34] type approaches.
[04:43.42] Basically, the basic fundamental insight is that there are only a few dimensions we can
[04:50.66] scale LLMs, so let's say in like 2017, 18, 19, 20, we were realizing that we could scale
[05:02.24] the number of parameters, and we scaled that up to 175 billion parameters for GPT-3.
[05:09.44] We did some work on scaling laws, which we also talked about in our data sets one-on-one
[05:14.18] episode where we're like, "Okay, we think the right number is 300 billion tokens to
[05:20.38] train 175 billion parameters," and then DeepMind came along and trained Gopher and Chinchilla
[05:25.66] and said that, "No, I think we think the compute optimal ratio is 20 tokens per parameter."
[05:33.86] And now, of course, with LLama and the super LLama scaling laws, we have 200 times and
[05:40.14] often 2,000 times tokens to parameters.
[05:44.12] So instead of scaling parameters, we're scaling data, and fine, okay, we can keep scaling
[05:50.16] data, but what else can we scale, and I think understanding the ability to scale things
[05:56.40] is crucial to understanding what to pour money and time and effort into because there's
[06:02.56] a limit to how much you can scale some things.
[06:05.70] And I think people don't think about ceilings of things.
[06:09.58] And so the remaining ceiling of inference is like, "Okay, we have scaled compute, we
[06:16.02] have scaled data, we have scaled parameters, model size, let's just say.
[06:20.70] What else is left?
[06:21.70] What's the low-hanging fruit?"
[06:23.34] And it's blindingly obvious that the remaining low-hanging fruit is inference time.
[06:27.94] So we have scaled training time, we can probably scale more, those things more, but not 10x,
[06:33.56] not 100x, not 1,000x.
[06:36.36] Right now, maybe a good run of the large model is three months.
[06:40.60] We can scale that to three years.
[06:42.56] But can we scale that to 30 years?
[06:44.96] No.
[06:45.96] Right?
[06:46.96] It starts to get ridiculous.
[06:48.20] So just the orders of magnitude of scaling, we're just running out there, but in terms
[06:54.24] of the amount of time that we spend inferencing, everything takes a few milliseconds or a few
[06:59.78] hundred milliseconds, depending on how you're taking token-by-token or the entire phrase.
[07:05.14] But we can scale that to hours, days, months of inference and see what we get.
[07:10.34] I think that's really promising.
[07:11.34] Yeah, we'll have Mike from Broadway back on the podcast, but I tried their product and
[07:16.30] their reports take about 10 minutes to generate instead of just in real time.
[07:21.86] I think to me, the most interesting thing about long inference is you're shifting the
[07:26.44] cost to the customer, depending on how much they care about the end result.
[07:31.24] If you think about prompt engineering, it's the first part, right?
[07:34.60] You can either do a simple prompt and get a simple answer or do a complicated prompt
[07:38.08] and get a better answer.
[07:39.60] It's up to you to decide how to do it.
[07:41.20] Now, it's like, hey, instead of training this for three years, I'll still train it for
[07:46.48] three months, and then I'll tell you, I'll teach you how to make it run for 10 minutes
[07:51.70] to get a better result.
[07:52.96] You're parallelizing the improvement of the LOL app.
[07:56.66] Yeah, you can even parallelize that, too.
[07:59.26] I think, for me, especially the work that I do, it's less about state-of-the-art and
[08:04.30] the absolute.
[08:05.30] It's more about state-of-the-art for my application, for my use case.
[08:09.50] I think we're getting to the point where most companies and customers don't really care
[08:15.02] about state-of-the-art anymore.
[08:16.32] It's like, I can get this to do a good enough job.
[08:20.10] I just need to get better.
[08:21.72] How do I do long inference?
[08:24.00] People are not really doing a lot of work in that space, so yeah, excited to see more.
[08:28.88] Then the last point I'll mention here is something I also mentioned as paper.
[08:32.64] All these directions are guided by what happened in January.
[08:36.24] That was my way of doing a January recap, which means if there was nothing significant
[08:39.88] in that month, I also didn't mention it, which I came to regret come February 15th.
[08:44.88] But in January, there was also the Alpha Geometry paper, which I kind of put in this sort of
[08:50.62] long inference bucket, because it solves more than 100-step math Olympiad geometry problems
[08:55.94] at a human gold medalist level.
[08:58.18] That also involves planning.
[08:59.26] If you want to scale inference, you can't scale it blindly, because just autoregressive
[09:03.62] token-by-token generation is only going to get you so far.
[09:06.38] You need good planning.
[09:08.10] I think probably what Mike from Brightwave is now doing and what everyone is doing, including
[09:13.42] maybe what we think Q* might be is some form of search and planning.
[09:18.56] It makes sense.
[09:19.56] You want to spend your inference time wisely.
[09:22.64] How do you think about plans that work and getting them shared?
[09:26.42] I feel like if you're planning a task, somebody has got in, and the models are stochastic,
[09:32.20] so everybody gets initially different results, somebody is going to end up generating the
[09:36.64] best plan to do something, but there's no easy way to store these plans and then reuse
[09:43.20] them for most people.
[09:45.82] I'm curious if there's going to be some paper or some work there on making it better, because
[09:51.82] yeah, we don't really have--
[09:52.82] This is your pet topic of NPM for packages.
[09:56.20] Yeah, exactly.
[09:57.20] You need NPM for anything, man.
[09:58.54] You need NPM for skills.
[10:00.24] You need NPM for planning.
[10:02.70] I mean, obviously, the Voyager paper is the most basic example where now there are the
[10:08.10] fact that the best planning to do a diamond pickaxe in Minecraft, and everybody can just
[10:13.24] use that.
[10:14.24] They don't need to come up with it again.
[10:16.00] But there's nothing like that for actually useful tasks.
[10:19.64] For plans, I believe it for skills.
[10:21.76] I like that.
[10:23.40] Basically that just means a bunch of integration tooling.
[10:26.48] GPT built me integrations to all these things, and I just came from an integration's heavy
[10:32.24] business, and I definitely propose some version of that, and it's just hard to execute or
[10:37.74] expensive to execute.
[10:39.44] But for planning, I do think that everyone lives in slightly different worlds, at slightly
[10:42.88] different needs, and they definitely want some amount of personalization to that.
[10:47.36] And I think that that will probably be the main hurdle for any library or package manager
[10:53.64] for planning.
[10:54.76] But there should be a meta plan of how to plan, and maybe you can adopt that.
[10:59.84] And I think a lot of people, when they have these meta prompting strategies of like-- I'm
[11:04.56] not prescribing you with the prompt.
[11:05.76] I'm just saying that here are the-- fill in the lines or the mad libs of how to prompt.
[11:11.38] First you have the role play, then you have the intention, then you have do something,
[11:14.86] then you have the don't something, and then you have the my grandmother is dying, please
[11:18.98] do this.
[11:19.98] So the meta plan, you could take off the shelf and test a bunch of them at once.
[11:25.22] I like that.
[11:26.34] That was the initial maybe promise of the prompting libraries.
[11:30.66] Both line chain and law index have hubs that you can sort of pull off the shelf.
[11:34.50] I don't think they're very successful because people like to write their own.
[11:38.04] Yeah, that's a good segue into the next one, which is synthetic data.
[11:42.80] Synthetic data is so hot.
[11:43.80] Yeah.
[11:44.80] And the way-- I feel like I should do one of these memes where it's like-- I used to
[11:50.56] call it RLAIF, and now I call it synthetic data, and then people are interested.
[11:54.44] But there's got to be older versions of what synthetic data really is.
[11:58.60] Because I'm sure if you've been in this field long enough, there's just different buzzwords
[12:04.08] that the industry condenses on.
[12:06.90] Anyway, the insight that I think is relatively new that why people are excited about it now
[12:11.70] and why it's promising now is that we have evidence that shows that LLMs can generate
[12:18.18] data to improve themselves with no teacher LLM.
[12:22.30] For all of 2023, when people say synthetic data, they really kind of mean generate a
[12:27.62] whole bunch of data from GPT-4 and then train an open source model on it.
[12:31.66] Hello to our friends at News Research.
[12:33.98] That's what News Hermes is.
[12:35.20] They're very, very open about that.
[12:37.00] I think they have said that they're trying to migrate away from that.
[12:40.76] But it is explicitly against opening up terms of service.
[12:43.16] Everyone knows this, especially once ByteDance got banned for doing exactly that.
[12:50.32] So synthetic data that is not a form of model distillation is the hot thing right now, that
[12:56.84] you can bootstrap better LLM performance from the same LLM, which is very interesting.
[13:03.60] A variant of this is RLAF, where you have sort of a constitutional model or some kind
[13:09.34] of judge model that is more aligned.
[13:13.74] But that's not really what we're talking about when most people talk about synthetic data.
[13:17.42] Synthetic data is just really, I think, generating more data in some way.
[13:23.50] A lot of people, I think we talked about this with Vipple from the Together episode, where
[13:29.14] I think he commented that you just have to have a good world model or a good sort of
[13:33.42] inductive bias or whatever that term of art is.
[13:37.92] And that is strongest in math and code, where you can verify what's right and what's wrong.
[13:44.40] And so the rest EM paper from DeepMind explored that very well.
[13:49.56] It's just the most obvious thing.
[13:51.28] And then once you get out of that domain of things where you can arbitrarily generate
[13:56.48] like a whole bunch of stuff and verify if they're correct, and therefore they're correct
[13:59.54] synthetic data to train on, once you get into more sort of fuzzy topics, then it's a bit
[14:06.26] less clear.
[14:07.26] So I think that the papers that drove this understanding, there are two big ones and
[14:12.50] then one smaller one, one was RAP, like rephrasing the web from Apple, where they basically rephrased
[14:17.82] all of the C4 data set with Mistral and trained on that instead of C4.
[14:23.66] And so new C4 trained much faster and cheaper than regular raw C4.
[14:30.52] And that was very interesting.
[14:31.80] And I have told some friends of ours that they should just throw out their own existing
[14:36.40] data sets and just do that because that seems like a pure win.
[14:39.80] Obviously we have to study like what the trade-offs are.
[14:42.48] I imagine there are trade-offs.
[14:44.32] So I was just thinking about this last night, like if you do synthetic data, you would probably,
[14:49.04] and it's generated from a model, probably will not train on typos.
[14:55.62] So therefore you will be like, once the model that's trained on synthetic data encounters
[14:59.70] the first typo, they'll be like, "What is this?
[15:01.38] I've never seen this before."
[15:03.76] So they have no association or a correction as to like, "Oh, these tokens are often typos
[15:08.42] of each other.
[15:09.42] Therefore they should be kind of similar."
[15:11.38] I don't know.
[15:13.10] That's really remains to be seen, I think.
[15:14.58] I don't think that the Apple people export that.
[15:16.74] Yeah.
[15:17.74] Yeah.
[15:18.74] Isn't that the whole mode collapse thing if we do more and more of this at the end of
[15:21.68] the day?
[15:22.68] Yes.
[15:23.68] That's one form of that.
[15:24.68] Yeah.
[15:25.68] Exactly.
[15:26.68] Microsoft also had a good paper on text invitations.
[15:27.68] And then I think this is a meta paper on self-rewarding language models that everyone is very interested
[15:32.00] in.
[15:33.00] And other people was also spin.
[15:34.00] These are all things we covered in the late in space paper club, but also I just kind of
[15:39.04] recommend those as top reads of the month.
[15:41.48] Yeah.
[15:42.48] I don't know if there's any much else in terms.
[15:44.88] And then regarding the potential of it, I think it's high potential because, one, it
[15:48.90] solves one of the data war issues that we have.
[15:51.94] Like everyone is opening eyes, paying Reddit $60 million a year for their user-generated
[15:56.58] data.
[15:57.58] Google, right?
[15:58.58] Not opening eyes.
[15:59.58] Is it Google?
[16:00.58] That's confirmed.
[16:01.58] I don't know.
[16:02.58] Well, somebody's paying them $60 million.
[16:04.02] That's for sure.
[16:05.02] Yes.
[16:06.02] Yeah.
[16:07.02] Yeah.
[16:08.02] And then I think it's maybe not confirmed who, but yeah, it is Google.
[16:09.82] Oh my God.
[16:10.82] That's interesting.
[16:11.82] Everyone was saying because Sam Altman owns 5% of Reddit, which is apparently $500 million
[16:17.44] for Reddit, he owns more than the founders.
[16:21.44] Not enough to get the date, I guess.
[16:24.12] So it's surprising that it would go to Google instead of open AI, but whatever.
[16:27.76] Okay.
[16:28.76] Yeah.
[16:29.76] So I mean, I think that's all super interesting in the data field that I think it's high potential
[16:32.56] because we have evidence that it works.
[16:34.28] Like this is not a doubt that it doesn't work.
[16:37.76] I think it's a doubt that there's what the ceiling is, which is the mold collapsing.
[16:43.34] If it turns out that the ceiling is pretty close, then this will maybe augment our data
[16:47.70] by like, I don't know, 30, 50% good, but not game changing.
[16:52.70] And most of the synthetic data stuff, it's reinforcement learning on a pre-trained model.
[16:57.66] People are not really doing pre-training on fully synthetic data, like large enough scale.
[17:02.54] Yeah.
[17:03.54] Unless one of our friends that we've talked to succeeds.
[17:06.26] Yeah.
[17:07.26] Pre-trained, synthetic data, pre-trained scale is synthetic data.
[17:09.96] I think that there will be a big step.
[17:12.80] And then there's a wild card.
[17:13.80] So all of these like smaller directions, I always put a wild card in there.
[17:17.52] And one of the wild cards is, okay, like let's say you have pre, you have, you, you script
[17:22.20] all the data on the internet that you think is useful that seems to top out at somewhere
[17:26.68] between two trillion to three trillion tokens, maybe eight trillion if Mistral gets lucky.
[17:31.56] Okay.
[17:32.56] If I need 80 trillion, if I need 100 trillion, where do I go?
[17:36.04] And so you can do synthetic data, maybe, but maybe that only gets you to like 30, 40 trillion.
[17:41.34] Like where, where's the extra alpha and maybe extra alpha is just trained more on the same
[17:45.18] tokens, which is exactly what almost it like a Nathan Lambert AI to after just after he
[17:51.66] did an interview with us, they release almost.
[17:53.62] So it's unfortunate that we didn't get to talk much about it, but almost actually started
[17:57.10] doing 1.5 epochs on every, on all data.
[18:00.98] And the databases paper that I covered in Europe says that, you know, you don't like
[18:05.42] don't really start to tap out of like the, the alpha dig or the sort of increase in improve
[18:09.60] loss that you get from data all the way until four epochs.
[18:13.08] And so I'm just like, okay, like, why do we all agree that one epoch is all you did?
[18:17.88] It seems like to be a trend.
[18:19.08] It seems that we think that memorization is very good or too good.
[18:22.66] But then also we are finding that for improvement in results that we really like, we're fine
[18:29.84] on overtraining on things intentionally.
[18:32.60] So I think that's an interesting direction that I don't see people exploring enough.
[18:37.06] And the more I see papers coming out, stretching beyond the one epoch thing, the more people
[18:43.02] are like, it's completely fine.
[18:44.30] And actually the only reason we stopped is because we ran out of compute budget.
[18:47.30] Yeah.
[18:48.30] Yeah.
[18:49.30] I think that's the biggest thing, right?
[18:50.30] Like that's not a valid reason.
[18:52.38] That's not science.
[18:53.38] It's just like, yeah.
[18:54.38] I wonder if, you know, Matt is going to do it.
[18:57.62] I heard Lama three, they want to do a hundred billion parameters model.
[19:01.94] I don't think you can train that on any, so many epochs, even with their compute budget.
[19:07.16] But they're the only ones that can save us because even if OpenAI is doing this, they're
[19:11.72] not going to tell us.
[19:12.72] Yeah.
[19:13.72] Same with DeepMind.
[19:14.72] Yeah.
[19:15.72] And so the updates that we got on Lama three so far is apparently that because of the
[19:18.84] Gemini news that we'll talk about later, they're pushing it back on the release.
[19:22.00] They already have it and they're just pushing it back to do more safety testing and politics
[19:27.28] testing.
[19:28.28] Well, our episode with Sumit will have already come out by the time this comes out, I think.
[19:33.72] So people will get the insight story on how they actually allocate the compute.
[19:38.62] Alternative architectures.
[19:39.62] Yeah.
[19:40.62] Well, shout out to RWKV, who won one of the prizes at our final Frontiers event last week.
[19:47.38] We talked about Mumbai and Strapayin on the Together episode.
[19:50.98] A lot of, yeah, Monarch Mixers.
[19:52.66] I feel like Together is like the strong Stanford/Hazy Research partnership because Chris Ray
[19:58.92] is one of the co-founders.
[20:01.00] So they kind of, I feel like they're going to be the ones that have one of the state-of-the-art
[20:06.08] models alongside maybe RWKV.
[20:08.88] I haven't seen as many independent people working on this thing.
[20:12.88] Like Monarch Mixer, yeah, Mamba, Strapayin, all of these are together.
[20:16.84] Nobody understands the math.
[20:18.84] Yeah.
[20:19.84] They got all the Giga brains.
[20:20.84] They got three now.
[20:21.84] They got all these folks in there like working on all of this.
[20:25.06] Albert Gu.
[20:26.06] Yeah.
[20:27.06] Yeah.
[20:28.06] So what should we comment about it?
[20:29.06] I mean, I think it's useful, interesting, but at the same time, you know, both of these
[20:33.58] are supposed to do really good scaling for long context.
[20:38.38] And then Gemini comes out and goes like, "Yeah, we don't need it."
[20:42.14] Yeah.
[20:43.14] I know.
[20:44.14] That's the risk.
[20:47.14] Yeah.
[20:48.14] I was going to say, maybe it's not here, but I don't know if we want to talk about diffusion
[20:52.04] transformers as like in the alt-architectures just because of Sora.
[20:56.08] One thing, yeah.
[20:57.08] So, you know, this came from the January cap, which in diffusion transformers were not
[21:00.88] really a discussion.
[21:01.88] And obviously they blow up in February.
[21:04.84] I don't think that it's a mixed architecture in the same way that Strapayin is mixed.
[21:10.24] There's just different layers and taking different approaches.
[21:13.44] Also I think another one that I maybe didn't call out here, I think because it happened
[21:17.12] in February was our glass diffusion from stability, but also, you know, another form
[21:23.02] of mixed architecture.
[21:24.42] So I guess that is interesting.
[21:26.66] I don't have much commentary on that.
[21:28.78] Like I just think like we will try to evolve these things and maybe one of these architectures
[21:33.62] will stick and scale.
[21:34.86] It seems like diffusion transformers is going to be good for anything generative, you know,
[21:40.22] multimodal.
[21:41.22] We don't see anything where diffusion is applied to text yet.
[21:44.38] And that's the wild card for this category.
[21:46.80] Yeah.
[21:47.80] I mean, I think I still hold out hope for, let's just call it sub quadratic LLMs.
[21:53.44] I think that the lot of discussion this month actually was also centered around this concept
[21:58.52] that people always say, "Oh, like transformers don't scale because attention is quadratic
[22:03.48] in the sequence."
[22:04.48] Yeah, but, you know, attention actually is a very small part of the actual compute that
[22:09.08] is being spent, especially in inference.
[22:11.08] And this is the reason why, you know, when you multiply, when you, when you, when you
[22:15.46] jump up in terms of the, the model size in GPT-4 from like, you know, 38K to like 32K,
[22:21.06] you don't also get like a 16 times increase in your, in your price.
[22:25.14] And this is also why you don't get like a million times increase in your, in your latency
[22:30.02] when you throw a million tokens into Gemini.
[22:33.50] Like people have figured out tricks around it, or it's just not that significant as a
[22:38.10] term as a part of the overall compute.
[22:41.16] So there's a lot of challenges to this thing working.
[22:43.76] It's really interesting how, like, how hype people are about this versus, I don't know
[22:49.60] if it, you know, is exactly going to, going to work.
[22:52.48] And then there's also this, this idea of retention over long context.
[22:56.12] Like even though you have context utilization, like the amount of, the amount you can remember
[23:01.96] is interesting because I've had people criticize both Mamba and RWKV because they're kind of
[23:06.60] like R&N-ish in the sense that they have like a hidden memory and sort of limited hidden
[23:10.38] memory that they will forget things.
[23:14.42] So, so that, for all these reasons, Gemini 1.5, which we have, we still haven't covered
[23:18.38] is, is, is very interesting because Gemini magically has fixed all these problems with
[23:22.70] perfect Haystack recall and, and these reasonable latency and cost.
[23:29.14] So that's, that's super interesting.
[23:30.78] So the wall card I put in here, if you want to go to that is, I put two, actually, one
[23:34.42] is text diffusion.
[23:35.42] I think I'm still very influenced by my meeting with a mid-journey person who said they were
[23:40.48] working on text diffusion.
[23:41.76] I think it would be a very, very different paradigm for, for text generation, reasoning
[23:47.64] plan generation.
[23:49.44] If we can get diffusion to work for text.
[23:52.20] And then the second one is the Dawekila's contextual AI, which is working on retrieval
[23:57.52] augmented language models, where it kind of puts rag inside of the language model, instead
[24:01.44] of outside.
[24:02.44] Yeah.
[24:03.44] There's a paper called Retro that covers some of this.
[24:06.14] I think that's an interesting thing.
[24:07.78] I think the, the challenge, or not the challenge, what they need to figure out is like, how
[24:12.42] do you keep the rag piece always up to date constantly, you know, I feel like the models,
[24:17.82] you put all this work into pre-training them, but then at least you have a fixed artifact.
[24:22.42] These architectures are like constant work needs to be done on them and they can drift
[24:26.94] even just based on the rag data instead of the model itself.
[24:29.66] Yeah.
[24:30.66] So I was in a panel with one of the investors in contextual, and the guy, the way that guy
[24:34.00] pitched it, I didn't agree with.
[24:35.32] He was like, this will solve hallucination.
[24:38.84] That's what everybody says.
[24:40.16] We solve hallucination.
[24:41.24] I'm like, no, you reduce it.
[24:43.44] It's not, it cannot, if you solved it, the model wouldn't exist, right?
[24:47.48] It would just be plain text.
[24:48.76] It wouldn't be a generative model.
[24:50.64] Cool.
[24:51.64] So all three architectures, then we got mixture of experts.
[24:53.68] I think we covered a lot of, a lot of times, maybe any new interesting threads you want
[24:59.72] to go under here?
[25:00.82] DeepSeq MOE, which was released in January.
[25:03.38] Everyone who's interested in MOE should read that paper because significant for two reasons.
[25:08.78] One, three reasons.
[25:10.42] One it had small experts, like a lot more small experts.
[25:13.82] So for some reason, everyone has settled on eight experts for GPT-4, for mixed trial.
[25:19.82] That seems to be like the favorite architecture, but these guys pushed it to 64 experts and
[25:24.58] each of them smaller than the other.
[25:26.14] But then they also had the second idea, which is that it is, they had two, one or two, all
[25:29.70] always on experts for common knowledge.
[25:32.28] And that's like a very compelling concept that you would not route to all the experts
[25:37.08] all the time and make them switch to everything.
[25:40.24] You would have some always on experts.
[25:41.88] I think that's interesting on both the inference side and the training side for memory retention.
[25:46.76] And yeah, the results that they published, which actually excluded mixed trial, which
[25:53.32] is interesting.
[25:54.32] The results that they published showed it's an infant performance jump versus all the
[25:58.12] other sort of open source models at the same parameter count.
[26:01.22] So like this may be a better way to do MOEs that is about to get picked up.
[26:05.42] And so that is interesting for the third reason, which is this is the first time a new idea
[26:09.70] from China has infiltrated the West.
[26:13.46] It's usually the other way around.
[26:14.98] I probably over spoke there.
[26:16.62] There's probably lots more ideas that I'm not aware of, maybe in the embedding space.
[26:20.30] But I think DCM, we woke people up and said, "Hey Deepseek, this like weird lab that is
[26:26.22] attached to a Chinese hedge fund is somehow doing groundbreaking research on MOEs."
[26:32.16] So I classified this as a medium potential because I think that it is sort of like a
[26:36.72] one-off benefit.
[26:37.72] You can add to any base model to make the MOE version of it.
[26:41.52] You get a bump and then that's it.
[26:45.32] I saw Samba Nova, which is like another inference company that released this MOE model called
[26:51.12] Samba One, which is like a one trillion parameters.
[26:54.56] But they actually MOE other open source models.
[26:57.28] So it's like they just clustered them all together.
[27:00.30] So I think people sometimes they think MOE is like you just train a bunch of small models
[27:05.18] or like smaller models and put them together.
[27:07.76] But there's also people just taking, you know, Mistral plus Clip plus, you know, Deep Coder
[27:13.62] and like put them all together.
[27:15.62] And then you have an MOE model.
[27:16.62] I don't know.
[27:17.62] I haven't tried the model.
[27:18.62] So I don't know how good it is, but it seems interesting that you can then have people
[27:23.18] working separately on state of the art, you know, Clip, state of the art, text generation,
[27:28.52] and then you have an MOE architecture that brings them all together.
[27:31.24] I'm thrown off by your addition of the word Clip in there.
[27:34.60] Is that what is it?
[27:35.60] Yeah, that's what we said.
[27:36.60] Yeah, yeah.
[27:37.60] Yeah, that's what I did.
[27:38.60] I just saw yesterday I was also like scratching my head and they did not use the word adapter
[27:43.44] because that usually what people mean when they say, oh, I add Clip to a language model
[27:47.40] is adapter, which is what Lava did.
[27:49.84] Yeah.
[27:50.84] It's a staple diffusion, as it do.
[27:53.46] Yeah.
[27:54.46] Yeah.
[27:55.46] It says, among the models that are part of SAMBA1 are Lama2, Mistral, Deep Coder, Falcon,
[27:59.98] DeepLod, Clip, Lava.
[28:02.10] So they're just taking all these models, putting them in MOE.
[28:05.30] Okay.
[28:06.30] So a routing layer and then not jointly trained as much as a normal MOE would be.
[28:12.58] Which is okay.
[28:13.58] All right.
[28:14.58] That's all this.
[28:15.58] There's no paper, you know.
[28:16.58] I'm just reading the article, but I'm interested to see how it evolves.
[28:22.04] So the wild card for this section, the MOE section is model merges, which has also come
[28:26.40] up as a very interesting phenomenon.
[28:29.16] The last time I talked to Jeremy Howard at the Olamah meetup, we call it model grafting
[28:33.64] or model stacking.
[28:34.64] But I think the term that people are liking these days, the model merging, there's all
[28:39.48] different variations of merging.
[28:40.84] Merge types and some of them are stacking, some of them are grafting.
[28:44.08] And so like some people are approaching model merging in the way that Samba is doing, which
[28:47.94] is like, okay, here are defined models, each of which have their specific pros and minuses.
[28:53.54] And we will merge them together in the hope that the sum of the parts will be better than
[28:57.94] others.
[28:59.06] And it seems like it's working.
[29:01.34] I don't really understand why it works, apart from I think it's a form of regularization.
[29:05.78] If you merge weights together in a smart strategy, you get a less overfitting and more generalization,
[29:13.66] which is good for benchmarks, if you're honest about your benchmarks.
[29:16.92] So this is really interesting and good.
[29:19.24] But again, they're kind of limited in terms of the amount of bumps you can get.
[29:23.40] But I think it's very interesting in the sense of how cheap it is.
[29:26.00] We talked about this on the China Talk podcast, like the guest podcast that we did with China
[29:30.56] Talk.
[29:31.56] And you can do this without GPUs because it's just adding weights together and dividing
[29:35.80] things and doing like simple math, which is really interesting as for the GPU course.
[29:41.56] Yeah, there's a lot of them.
[29:44.78] And just to wrap these up, online LLAMPs.
[29:47.54] Yeah, I think that I had to feature this because one of the top news of January was that Gemini
[29:56.02] Pro beat GPT-4 Turbo on LMSYS for the number two slot to GPT-4.
[30:03.02] And everyone was very surprised, like, how does Gemini do that?
[30:06.98] Surprise surprise, they added Google search to the results.
[30:09.54] So it became an "online LLAM" and not an offline LLAM.
[30:14.52] Therefore, it's much better at answering recent questions, which people like.
[30:17.96] There's an emerging set of table stakes features after you pre-train something.
[30:21.92] So after you pre-train something, you should have the chat tune version of it, or the instruct
[30:26.04] tune version of it, however you choose to call it.
[30:28.08] You should have the JSON and function calling version of it, a structured output, the term
[30:32.68] that you don't like.
[30:34.76] You should have the online version of it.
[30:36.52] These are all table stakes variants that you should do when you offer a base LLAM or you
[30:43.02] train a base LLAM.
[30:44.02] And I think online is just there, it's important.
[30:47.18] I think companies like Perplexity and even Exa, formerly Metaphor, are rising to offer
[30:54.62] their search needs.
[30:55.62] And it's kind of like, they're just necessary parts of a system.
[30:59.58] You have RAG for internal knowledge, and then you have online search for external knowledge,
[31:04.62] things that you don't know yet.
[31:06.96] And it seems like it's one of many tools.
[31:09.68] I feel like I may be underestimating this, but I'm just going to put it out there that
[31:13.12] I think it has some potential.
[31:15.48] One of the evidence points is that it doesn't actually matter that much, is that Perplexity
[31:20.20] has an online LLAMs for three months now, and it doesn't perform great on LLAMs.
[31:26.80] It's like number 30 or something.
[31:28.52] So it's like, okay, it helps, but it doesn't give you a giant, giant boost.
[31:34.44] I feel like a lot of stuff I do with LLAMs doesn't need to be online.
[31:38.82] So I'm always wondering, again, going back to state of the art, state of the art for
[31:44.50] who and for what.
[31:46.18] I think online LLAMs are going to be state of the art for news-related activity that
[31:54.18] you need to do.
[31:55.18] You're like social media, it's like you want to have all the latest stuff, but coding,
[32:00.74] science.
[32:01.74] Yeah, but I think sometimes you don't know what is news affecting.
[32:07.84] The decision to use an offline LLAM is already like a decision that you might not be consciously
[32:13.60] making that might affect your results.
[32:17.72] Just putting things on, being connected online means that you get to invalidate your knowledge,
[32:23.54] and when you're just using offline LLAM, it's never invalidated.
[32:27.80] I agree, but I think going back to your point of like the standing the test of time, I think
[32:32.62] sometimes you can get swayed by the online stuff, which is like, hey, you ask a question
[32:38.46] about, yeah, maybe AI research direction, you know, and it's like, all the recent news
[32:43.32] are about this thing.
[32:44.76] So the LLAM is going to like focus on answering, bring it up, you know, these things.
[32:49.46] Yeah, so, yeah, I think I think it's interesting, but I don't know if I can, I bet heavily
[32:55.10] on this.
[32:56.10] Cool.
[32:57.10] Was there one that you forgot to put or like a new direction?
[33:01.60] Yeah, so this brings us into sort of February ish.
[33:05.60] So like I published this in like Feb 14 and then Feb 15 came with Sora.
[33:12.14] And so like the one thing I did not mention here was anything about multimodality, right?
[33:17.00] And I have chronically underrated this, I always wrestle and my cop out is that I focused
[33:23.68] this piece or this research direction piece on LLAMs because LLAMs are the source of like
[33:29.30] AGI, quote unquote, everything else is kind of like, you know, related to that.
[33:34.90] Like generative, like just because I can generate better images or to generate better videos,
[33:40.10] it feels like it's not on the critical path to AGI, which is something that Ned Friedman
[33:45.30] also observed like the day before Sora, which is kind of interesting.
[33:49.90] And so I was just kind of like trying to focus on like what is going to get us like super
[33:53.68] human reasoning that we can rely on to build agents that automate our lives and blah, blah,
[33:58.56] blah, you know, give us this utopian future.
[34:01.08] But I do think that I, everybody underestimated the, the share importance and cultures or
[34:09.20] human impact of Sora and you know, really actually good text to video.
[34:13.48] Yeah.
[34:14.48] Yeah.
[34:15.48] And I saw Jim Fann at a, at a very good tweet about why it's so impressive.
[34:18.64] And I think when you have somebody leading the embodied research at Ambedia and he said
[34:23.10] that something is impressive, you should probably listen.
[34:25.86] So yeah, they're specifically like, I think you, you mentioned like impacting the world,
[34:32.66] you know, that we live in.
[34:33.66] I think that's kind of like the key, right?
[34:35.94] It's like the LLAMs don't have a world model and Jan Le Koon, he can come on the podcast
[34:42.18] and talk all about what he thinks of that.
[34:45.50] But I think Sora was like the first time where people are like, oh, okay, you're not statically
[34:49.92] putting pixels of water on the screen, which you can kind of like, you know, project without
[34:55.36] understanding the physics of it.
[34:57.24] Now you're like, you have to understand how the water splashes when you have things.
[35:02.20] And even if you just learned it by watching video and not by actually studying the physics,
[35:07.28] you still know it, you know?
[35:08.88] So I think that's like a direction that yeah, before you didn't have, but now you can do
[35:14.76] things that you couldn't before, both in terms of generating, I think it always starts with
[35:18.98] generating, right?
[35:19.98] But like the interesting part is like understanding it, you know, it's like, if you gave it,
[35:24.38] you know, there's the video of like the ship and the water that they generated with Sora.
[35:28.98] Like if you gave it the video back and now it could tell you why the ship is like too
[35:33.98] rocky, or like it could tell you why the ship is sinking.
[35:36.58] Then there's like, you know, AGI for like oil rig deployments and like all this stuff,
[35:41.86] you know?
[35:42.86] And there's none of that yet.
[35:44.60] So hopefully they announce it and talk more about it.
[35:47.60] Maybe a Dev Day this year.
[35:48.88] Who knows?
[35:49.88] Yeah.
[35:50.88] Who knows?
[35:51.88] Who knows?
[35:52.88] I'm talking with them about Dev Day as well.
[35:55.16] So I would say like the phrasing that Jim used, which resonated with me that he kind
[36:00.88] of called a data-driven world model.
[36:02.80] I somewhat am more on Yanlucan side than I am on Jim's side in the sense that I think
[36:10.08] that is the vision or the hope that these things can build world models.
[36:16.56] But you know, clearly even at the current Sora size, they don't have the idea of strong
[36:22.02] consistency yet.
[36:23.02] They have very good consistency, but you know, fingers and arms and legs will appear and
[36:27.74] disappear and chairs will appear and disappear and that definitely breaks physics.
[36:33.94] And actually it makes me think about how we do deep learning versus world models in the
[36:39.54] sense of, you know, in a classic machine learning, when you have too many parameters, you will
[36:44.32] overfit and actually that fails, that like does not match reality and therefore fails
[36:49.16] to generalize well.
[36:51.02] And like what scale of data do we need in order to learn world models from video?
[36:56.88] A lot.
[37:00.36] So I am cautious about taking this interpretation too literally.
[37:05.10] Obviously, you know, I get what he's going for.
[37:08.12] And he's like, obviously partially right, obviously, like transformers and, you know,
[37:13.18] these like these sort of these neural networks are universal function approximators.
[37:17.46] Theoretically could figure out world models.
[37:20.38] It's just like, how good are they and how tolerant are we of hallucinations?
[37:25.34] We're not very tolerant.
[37:26.34] Like, yeah, so it's going to prior, it's going to bias for creating like very convincing
[37:32.86] things, but then not create like the useful world models that we want.
[37:38.02] At the same time, what you just said, I think made me reflect a little bit like we just
[37:42.80] got done saying how important synthetic data is for training LLMS.
[37:47.80] And so like if this is a way of synthetic, you know, video data for improving our video
[37:53.40] understanding, then sure by all means, which we actually know that like GPT-4 Vision and
[37:59.72] Dolly were trained kind of co-trained together.
[38:02.96] And so like, maybe this is on the critical path and I just don't fully see the full picture
[38:07.64] yet.
[38:08.64] Yeah.
[38:09.64] I don't know.
[38:10.64] I think there's a lot of interesting stuff is like, imagine you go back, you have Sora,
[38:14.68] you go back in time and Newton didn't figure out gravity yet.
[38:19.34] Would Sora help you figure it out?
[38:21.66] Because you start saying, okay, a man standing under a tree with like apples falling and it's
[38:26.78] like, oh, they're always falling at the same speed in the video.
[38:29.62] It's like, why is that?
[38:31.10] I feel like sometimes these engines can like pick up things like humans have a lot of intuition.
[38:36.22] But if he has the average person, like the physics of like a fluid in a boat, they couldn't
[38:40.80] be able to tell you the physics, but they can like observe it.
[38:44.72] But humans can only observe this much, you know, versus like, now you have these models
[38:48.00] to observe everything and then they generalize these things and maybe we can learn new things
[38:52.64] through the generalization that they pick up.
[38:55.20] But again,
[38:56.20] And it might be more observed than us in some aspects.
[38:58.80] In some ways we can scale it up a lot more than the number of physicists that we have
[39:02.24] available at Newton's time.
[39:04.12] So like, yeah, absolutely possible that this can discover new science.
[39:08.64] I think we have a lot of work to do to formalize the science.
[39:11.66] And then I think the last part is like, you know, how much, how much do we cheat by generating
[39:18.06] data from Unreal Engine 5, which is what a lot of people are speculating with very, very
[39:22.86] limited evidence that OpenEI did that.
[39:27.38] The strongest evidence that I saw was someone who works a lot with Unreal Engine 5, looking
[39:31.30] at the side characters in the videos and noting that they all adopt Unreal Engine defaults
[39:37.52] of like walking speed and like character choice, like character creation choice.
[39:43.56] And I was like, okay, like that's actually pretty convincing that they actually use Unreal
[39:46.80] Engine to bootstrap some synthetic data for this, for this training set.
[39:52.00] Yeah.
[39:53.00] Could very well be.
[39:54.00] Yeah.
[39:55.00] Because then you get the labels and the training side by side.
[39:58.38] One thing that came up on the last day of February, which I should also mention is Emo
[40:02.76] coming out of Alibaba, which is also a sort of like video generation and space time transformer
[40:10.60] thing that also involves probably a lot of synthetic data as well.
[40:18.00] And so like, this is Alvacain in the sense of like, oh, like, you know, really good generative
[40:23.24] video is here and it is not just like the one, two second clips that we saw from like
[40:28.82] other people and like, you know, Pika and all the other runway or, you know, run crystal
[40:34.66] ball Valenzuela from runway was like game on, which like, okay, good, but like, let's
[40:39.58] see your response.
[40:40.58] Because we've heard a lot about J1 and two, but like, it's nothing on this level of Sora.
[40:44.42] So like, it has to be seen like how we can actually apply this, but I do think that the
[40:48.50] creative industry should start preparing.
[40:50.30] I think the Sora technical blog post from open AI was really good.
[40:53.84] It was like a request for startups.
[40:55.64] It was so good in like spelling out here, the individual industries, this can impact.
[41:01.12] And anyone who anyone who is like interested in generative video should look at that.
[41:05.16] But also be mindful that probably when opening, I releases a Sora API, you know, the ways
[41:12.52] you can interact with it are very limited, just like the way you can interact with Dali
[41:15.16] you're very limited and someone is going to have to make open Sora to, to, to, to, for
[41:21.38] you to create comfy UI pipelines.
[41:23.82] Yeah.
[41:24.82] The stability folks said they want to build an open Sora competitor, but their demo, their
[41:29.82] demo video, their demo video was like so underwhelming, it was just like two people sitting on the
[41:34.10] beach.
[41:35.10] Well, they don't have it yet.
[41:36.10] Right?
[41:37.10] Yeah.
[41:38.10] They just want to train it.
[41:39.10] Everybody wants to, right?
[41:40.10] I think what is confusing a lot of people about stability is like, they're, they're pushing
[41:43.54] a lot of things in stable codes, stable on them and stable video diffusion.
[41:48.12] But like how much money do they have left?
[41:50.20] How many people do they have left?
[41:52.28] I have had like a really, like you might spend two hours with me reassuring me things are
[41:57.72] great and, and I'm like, I, I do like, I do believe that they have really, really quality
[42:02.20] people, but it's just like, I also have a lot of very smart people on the other side
[42:06.88] telling me like, Hey man, like, you know, don't, don't put too much faith in this thing.
[42:11.74] I don't know who to believe.
[42:12.74] Yeah.
[42:13.74] Sorry.
[42:14.74] Let's see.
[42:15.74] What else?
[42:16.74] We got a lot more stuff.
[42:17.74] I don't know if we can.
[42:18.74] Yeah.
[42:19.74] We can do a bit of a grok prep.
[42:20.74] We're about to go to talk to Dylan Patel.
[42:22.24] Maybe, maybe it's the audio in here.
[42:23.74] I don't know.
[42:24.74] It depends what we get up to later.
[42:26.96] What do you as an investor think about grok?
[42:29.24] Yeah.
[42:30.24] Actually, can you recap?
[42:31.24] Like why is grok interesting?
[42:32.68] So Jonathan Ross, who's the founder of grok, he's the person that created the TPU at Google.
[42:38.90] It's actually it was one of his like 20% projects.
[42:42.24] It's like, he was just on the side, doobie-doo, created the TPU.
[42:47.08] But yeah, basically grok, they, they had this demo that went viral where they were running
[42:52.04] Mistraw at like 500 tokens a second, which is like fastest at anything that you have
[42:57.04] out there.
[42:58.20] The question, you know, it's not like the, the means for like, is immediate that like
[43:02.40] people don't need H 100s anymore.
[43:04.92] I think there's a lot of money that goes into building what grok has built as far as the
[43:10.30] hardware goes.
[43:11.30] We're gonna, we're gonna put some of the notes from, from Dylan in here, but basically
[43:15.26] the cost of the grok system is like 30 times the cost of a H 100 equivalent.
[43:22.66] So let me, I put some numbers because me and Dylan, I, where like, I think the two people
[43:27.26] actually try to do grok math spreadsheet doors, so one, that's okay.
[43:34.74] So, so equivalent H, H 100 for a llama two is 300,000 for a system of eight cards.
[43:42.48] And for grok is 2.3 million because you have to buy 576 grok cards.
[43:48.80] So yeah, that, that just gives people an idea.
[43:50.96] So like if you deprecate both over a five year lifespan per year, you're deprecating
[43:55.04] 460 K for grok and 60 K a year for each 100.
[43:59.20] So like grok's are just way more expensive per model that you're, that you're hosting.
[44:04.44] But then you make it up in terms of volume.
[44:06.92] So I don't know if you want to cover that.
[44:08.42] Yeah.
[44:09.42] And the, I think one of the promises of, of grok is like super high parallel inference
[44:14.58] on the same thing.
[44:15.58] So you're busy saying, okay, I'm putting on this upfront investment on the hardware,
[44:20.98] but then I get much better scaling once I have it installed.
[44:24.54] I think the big question is how much can you sustain the parallelism?
[44:28.94] You know, like if you get, if you're going to get 100% utilization rate at all times
[44:33.82] on grok, like it's just much better, you know, because like at the end of the day, the tokens
[44:38.92] per second costs that you're getting is better than with DH 100s.
[44:43.98] But if you get to like 50% utilization rate, you will be much better off running on NVIDIA.
[44:49.72] And if you look at most companies out there, who really gets 100% utilization rate?
[44:56.64] Probably open AI at peak times, but this probably hit, but yeah, curious to see more.
[45:01.04] I saw the, Jonathan was just at the web summit in Dubai in Qatar.
[45:06.66] He just gave a talk there yesterday that I haven't listened to yet.
[45:09.74] I tweeted that he should come on the pod.
[45:12.02] He liked it.
[45:13.02] And then grok followed me on Twitter.
[45:14.02] I don't know if that means that they're interested, but hopefully--
[45:17.02] Brok's social media person is just very friendly.
[45:18.90] Yeah.
[45:19.90] Hopefully we're going to get them.
[45:20.90] We're going to get them.
[45:21.90] Yeah.
[45:22.90] We just call them out.
[45:23.90] And so basically the key question is like how sustainable is this and how much is this
[45:28.50] a loss leader?
[45:29.58] The entire grok management team has been on Twitter and Hacuneo saying they are very,
[45:32.80] very comfortable with the pricing of $0.27 per million tokens.
[45:37.52] This is the lowest that anyone has offered tokens as far as Mekstral or Lama2.
[45:42.98] This matches deep and fra and, you know, I think that's about it in terms of that low.
[45:47.72] And we think the break even for each 100 is 50 cents at a normal utilization rate.
[45:53.04] To make this work, so in my spreadsheet I made this work, you have to have like a parallelism
[45:58.86] of 500 requests all simultaneously.
[46:02.26] And you have model bandwidth utilization of 80%, which is way high.
[46:07.50] I just gave them high marks for everything.
[46:10.70] Grok has two fundamental tech innovations that they hinge their hats on in terms of
[46:16.90] like why we are better than everyone, you know, even though it remains to be independently
[46:21.62] replicated.
[46:22.62] But one, you know, they have this sort of the entire model on the chip idea, which is
[46:28.06] like, okay, get rid of HBM and like put everything in a stream, like, okay, fine, like, but then
[46:33.46] you need a lot of cards and whatever, and that's, that's all okay.
[46:36.78] And so like, because you don't have to transfer between memory, then you just save on that
[46:40.46] time and that's why they're faster.
[46:41.58] So a lot of people buy that as like, that's the reason that you're faster.
[46:45.42] Then they have some kind of crazy compiler or like speculative routing magic using compilers
[46:53.18] that they also attribute towards their higher utilization.
[46:55.76] So I give them 80% for that.
[46:57.36] And so that all that works out to like, okay, base costs, I think you can get down to like
[47:01.16] maybe like 20 something cents per million tokens.
[47:04.46] And therefore you actually are fine if you have that kind of utilization.
[47:08.82] But it's like, I have to make a lot of favorable assumptions for this to work.
[47:12.22] Yeah.
[47:13.22] Yeah.
[47:14.22] I'm curious to see what Dylan tests later.
[47:15.22] Yeah.
[47:16.22] So he was like complete opposite of me.
[47:17.86] He's like, they're just burning money.
[47:19.58] Yeah.
[47:20.58] It's just great.
[47:22.74] Gemini.
[47:23.74] I'm going to do a quick run through since this touches on all the four words.
[47:28.08] Yeah.
[47:29.08] And I see, I think this is the market for useful framework that when a new thing comes
[47:32.36] along, you can break it down in terms of the four words and sort of slotted it slotted
[47:35.64] it in or analyze it in those four frameworks and have nothing left.
[47:40.60] So it's, so it's a MISI categorization MISI is mutually exclusive and collectively exhaustive.
[47:48.40] And that's a really, really nice way to think about taxonomies and to create mental frameworks.
[47:52.96] So what is Gemini 1.5 pro?
[47:55.36] It is the newest model that was came out one week after Gemini 1.0, which is very interesting
[48:00.26] that they have not really commented on why they released this.
[48:04.10] The headline feature is that it has a one million token context window that is multimodal,
[48:08.02] which means that you can put all sorts of video and audio and PDFs natively in there
[48:12.82] alongside of text.
[48:14.38] And you know, it's, it's at least 10 times longer than anything the opening eye offers,
[48:19.14] which is interesting.
[48:20.22] So it's great for prototyping and it, it has interesting discussions on whether it kills
[48:24.56] RAG.
[48:25.56] Yeah, no, I mean, we always talk about long context is good, but you're getting charged
[48:30.76] per token.
[48:31.76] So yeah, people love for you to use more tokens in the, in the context and RAG, RAG is better
[48:37.48] economics, but I think it all comes down to like how the price curves change, right?
[48:42.32] I think if anything rags complexity goes up and up, the more you use it, you know, because
[48:47.44] you have more data sources, more things you want to put in there.
[48:51.12] The token costs should go down over time, you know, if the model stays fixed, if people
[48:56.66] are happy with the model today in two years, three years, it just going to cost a lot less,
[49:01.74] you know, so now it's like, why would I use RAG and like go through all of that?
[49:05.82] It's interesting.
[49:06.82] I think RAG is better cutting edge economics for LLMs.
[49:10.10] I think large context will be better long tail economics when you factor in the bill
[49:15.04] cost of like managing the RAG pipeline.
[49:17.66] But yeah, the recall was like the most interesting thing because we've seen the needle in the
[49:22.10] haystack things in the past, but apparently they have 100% recall on anything across the
[49:27.42] context window.
[49:28.42] At least they say nobody has used it on the large, on the, yeah.
[49:32.82] So as far as, so, so what does needle in the haystack thing for people who are following
[49:37.38] as closely as us is that someone, I forget his name now, someone created this needle in
[49:42.46] the haystack problem where you feed in a whole bunch of generated junk, not junk, but just
[49:47.04] like generated data and ask you to specifically retrieve something in that data, like one
[49:51.72] line in like a hundred thousand lines where it like has a specific effect.
[49:55.64] And if it, if you get it, you're, you're good.
[49:57.68] And then he moves the needle around, like, you know, does it, does, does your ability
[50:00.88] to retrieve that vary if I put it at the start versus put it in the middle, put it at the
[50:05.20] end.
[50:06.20] And then you generate this like really nice chart that kind of shows like it's recall
[50:09.08] ability of a model.
[50:10.78] And he did that for GPT and, and, and thropic and showed that entropic did really, really
[50:14.50] poorly.
[50:15.50] And then entropic came back and said, it was a skill issue just at this, like for, for
[50:18.98] magic words.
[50:19.98] And then, then it's magically all fixed.
[50:21.10] And obviously everybody laughed at that.
[50:22.78] But what Gemini came out with was, was that, yeah, we reproduced the needle in the haystack
[50:27.22] issue, you know, test for Gemini and it's good across all, all the 1 million token window,
[50:31.94] which is very interesting.
[50:33.10] Cause usually for typical context extension methods, like rope or yarn or, you know, anything
[50:38.48] like that or alibi, it's lossy, like Dubai design is lossy.
[50:42.82] Usually for conversations, that's fine.
[50:44.66] Cause we are lossy when we talk to people, but for super human intelligence, perfect
[50:48.74] memory across a very, very long context is very, very interesting for, for picking things
[50:53.50] up.
[50:54.50] And so the people who have been given the beta tests for Gemini have been testing this.
[50:58.46] So what you do is you upload, let's say all of Harry Potter and you change one fact in,
[51:02.50] in one sentence somewhere in there and you ask it to pick it up and it does.
[51:07.78] So this is legit.
[51:08.88] We don't super know how this is like, because it doesn't, yes, it, it's slow to inference,
[51:16.44] but it's not slow enough that it's like running five different systems in the background without
[51:20.64] telling you.
[51:21.64] Right.
[51:22.64] So it's something, it's something interesting that they haven't fully disclosed yet.
[51:27.32] The open source community has centered on this ring attention paper, which is created
[51:31.64] by your friend Matei Zaharia and, and a couple other people.
[51:37.20] And it's a form of distributing the, the compute.
[51:41.76] I don't super understand like why, you know, doing, calculating like the fee for network
[51:46.88] and attention in block wise fashion and distributing it makes it so good at recall.
[51:52.06] I don't think they have any answer to that.
[51:54.06] The only thing that ring attention is really focused on is basically infinite context.
[51:59.34] They said it was good for like 10 to 100 million tokens, which is, it's just great.
[52:04.86] So yeah, using the four, four wars framework, what is this frame?
[52:07.56] What is this framework for Gemini?
[52:09.68] One is the sort of rag and ops war here.
[52:13.76] We care less about rag now.
[52:15.48] Yes.
[52:16.48] Or we still care as much about rag, but like now it's, it's not important in prototyping.
[52:21.40] And then for data war, I guess this is just part of the overall training data set, but
[52:25.84] Google made a 60 million dollar deal with Reddit and presumably they have deals with
[52:29.76] other companies for the multi modality war.
[52:32.80] We can talk about the image generation crisis or the fact that Gemini also has image generation,
[52:39.42] which we'll talk about in the next section.
[52:42.14] But it also has video understanding, which is, I think the top Gemini posts came from
[52:46.02] our friend, Simon Willison, who basically did sort of like a short video of him scanning
[52:51.70] over his bookshelf and it would be able to convert that video into a JSON file, a JSON
[52:58.74] sort of output of, you know, what's on that bookshelf.
[53:01.90] And I think that is very useful and actually ties into the conversation that we had with
[53:06.64] David Luan from a debt in the sense of like, okay, what if video was the main modality
[53:12.52] instead of text as the input?
[53:14.76] What if, what if everything was video in, because that's how we work.
[53:19.68] Our eyes don't actually read, don't actually like get input, our brains don't get inputs
[53:24.56] as characters.
[53:25.56] Our brains get the pixels shooting into our eyes and then our vision system takes over
[53:32.54] first, and then we sort of mentally translate that into text later.
[53:37.94] And so it's kind of like, what a debt is kind of doing, which is driving by vision model
[53:42.74] instead of driving by raw text understanding of the DOM.
[53:46.86] And in that episode, which we haven't released, I made the analogy to like self-driving by
[53:51.18] LiDAR versus Self-Driving by Camera, right?
[53:53.50] I think what Gemini and any other super long context that model that is multimodal unlocks
[54:00.32] is what if you just drive everything by video, which is cool.
[54:04.34] Yeah.
[54:05.34] And that's Joseph from Roboflow, it's like anything that can be seen, can be programmable
[54:11.02] with these models.
[54:12.02] You mean the computer vision guys is a bullish computer vision?
[54:16.46] It's like the rag people.
[54:20.14] The rag people are a bullish rag and not a long context.
[54:23.94] The fine tuning people, a lot of fine tuning instead of a few shots.
[54:28.34] Yeah.
[54:29.34] Yeah.
[54:30.34] That's that.
[54:31.34] Yeah.
[54:32.34] I think the ring attention thing and it's how they did it, we don't know.
[54:36.30] And then they released the Gemma models, which are like a two billion and seven billion
[54:40.82] open models, which people said are not, are not good based on my experience, which are
[54:46.42] the, the GPU poor crumbs.
[54:48.34] It's like, Hey, we did all this work for us because we're GPU rich and we're just kind
[54:52.78] of around this whole thing and you guys can take these, these small models and they're
[54:56.82] not very good.
[54:57.82] They're not better than the others, but at least we can say we made some open source
[55:01.26] stuff.
[55:02.26] Yeah.
[55:03.26] Well, it's not actually technically open source because the license is weird.
[55:05.90] Yeah.
[55:06.90] They used the rail license from hugging face, which has been abandoned or, you know, modified
[55:11.58] to real plus, particularly adopting the term of the phrase that you should make reasonable
[55:16.90] efforts to update whenever they release a new version.
[55:20.14] And so people are, don't like that obviously, you know, depends on your stance on open sourcing
[55:24.46] and all that.
[55:25.46] So I wrote the whole post, not going to go through it again.
[55:28.62] Yeah.
[55:29.62] You can go read a less use post on whether open source matters or not.
[55:31.90] Okay.
[55:32.90] So I, I know this is like politically problematic, but we just cover it because it's it is news
[55:37.90] and if it results in the resignation of Sundar Pichai, I think that is important.
[55:42.10] Right.
[55:43.10] So it's, I've been calling this the alignment, alignment crisis.
[55:45.70] I think a lot of people have been focusing on Gemini, but I do think that it is not just
[55:50.34] Gemini.
[55:51.34] There's, there's been documented examples that we can link in the show notes of meta
[55:55.78] having unintentionally unaligned results for Microsoft's co-pilot Sydney is apparently
[56:02.14] back Justin from a 16 Z somehow got it to, to, to break and then bring back the Sydney
[56:09.26] persona, which is interesting.
[56:11.22] And my favorite commentary is from Grimes, the sort of Elon affiliated music, the news
[56:16.78] research.
[56:17.78] I want to read her post cause it is beautiful.
[56:22.30] Have you read this?
[56:23.30] Yeah.
[56:24.30] So she says, so a lot of people criticize Gemini for being too woke effectively, right?
[56:28.38] And everyone's like, Oh, like, you know, you're, you're, you're, you know, you're replacing
[56:32.14] us or you're raising us or whatever.
[56:34.98] And she, obviously as an artist, she's like upset about it.
[56:37.94] Then she was like, wait a minute, I'm retracting my statements about the Gemini art disaster.
[56:41.70] It is in fact a masterpiece of performance art, even if unintentional true gain of function
[56:46.46] art, artists of virus, unthinking, unintentional and contagious, offensive to all comforting
[56:51.18] to none.
[56:52.18] So totally divorced for meaning intention, desire and humanity that is accidentally a
[56:56.18] conceptual masterpiece.
[56:57.54] Wow.
[56:58.54] And I love, okay, blah, blah, blah.
[57:00.70] It's a long post, but I love the way that she ended it.
[57:03.78] It's trapped in a cage, trained to make beautiful things and then battered into gaslighting humankind
[57:07.18] about our intentions towards each other.
[57:09.18] This is arguably the most impactful art project of the decade thus far.
[57:13.14] Art for no one by no one.
[57:15.34] Art whose only audience is the collective pathos, incredible and worthy of the MoMA.
[57:20.06] Like art for no one by no one is what is going on.
[57:26.10] I think it's just another way of multi-collapsing, you know, it's just like, it's the RLHF
[57:32.70] multi-collapse.
[57:33.70] It's like, okay, I just think everything should like trends, trend towards this.
[57:38.58] And I think there's obviously, you know, it's a deep discussion on a lot of these things,
[57:44.06] but there's safety stuff that I would expect a lot of the model builders to say, hey, I
[57:50.70] definitely got to work on this.
[57:53.02] But we talked about how image generation is not really on the AGI path a lot of times.
[57:58.18] And it's like, okay.
[57:59.18] Yeah, and then I contradicted myself by saying like, maybe it is useful synthetic data.
[58:03.62] Yeah, yeah, exactly.
[58:05.18] But then it's like, okay, then why, why are the image generation model like so much because
[58:11.86] the internet is so visual.
[58:13.38] I think the image generation model get like so much interest in like a lot of these things.
[58:17.90] But if their job is really to like go build the AGI, it's like, just build the great model
[58:22.50] and let it go.
[58:24.02] But you know, but part of my problem, part of my issue is that I think the prompt stuff
[58:27.74] from Gemini is honestly the work of like one or two people who like didn't really think
[58:32.56] it through at Google and now they're facing a huge backlash.
[58:35.66] Yeah.
[58:36.66] Elon has specifically picked a fight with the product manager who did it.
[58:40.26] And so specifically for those who don't know, the reason that Gemini is so woke is literally
[58:44.10] because they just take your prompt and they rewrite it to be more diverse without your
[58:47.38] consent or knowledge, right?
[58:49.10] And Hamel Hussein, who's a good consultant on AI things, actually wrote an interesting
[58:54.06] blog post recently, which was basically, fuck you, show me the prompt, which is like stop
[58:58.38] hiding prompts for me, stop rewriting magic things away from me and then like, you know,
[59:02.22] hiding it, obscuring it because I need, I need that control, I need that visibility.
[59:07.26] And I think like people just didn't understand that this tendency towards diversity did not
[59:12.02] exist at the model level.
[59:13.70] It actually exists at the prompt level and it was just inserted by probably like two
[59:18.22] or three guys without much review.
[59:20.50] That's it.
[59:21.50] Like that made all of Google look bad, which is absurd.
[59:24.30] Like, you know, those were a lot of the work that, you know, the rest of Google did, specifically
[59:28.86] Imogen2.
[59:29.86] This is Imogen2 and I've met that team and they're, you know, they're good, they're smart.
[59:34.50] They're not, they're a completely different team than Imogen1, which is another fun topic
[59:38.44] of conversation.
[59:39.76] So I think like that's interesting and, but what's more interesting is like OpenAI has
[59:44.30] done this for people don't remember, they used to append like black or like, you know,
[59:49.26] Asian or whatever to, to their prompts, just to make it more diverse and dolly and they
[59:53.06] didn't get canceled.
[59:54.06] And I think, so I think this, this will get, this will get, go away.
[59:56.86] But what really is more interesting is at the model level, like, are we, are we overlining
[60:00.62] two things?
[60:01.62] And, and people are now focusing on the alignment of Gemini as well in text, text only as also
[60:07.54] still being to work.
[60:08.54] So I think this is like a phenomenon that is, needs to be studied and, and, you know, trained
[60:14.22] like obviously they will try to make attempts, but, you know, they're not going to make anyone
[60:18.02] happy.
[60:19.02] And then like, I think my last point on this, because obviously we can talk about this all
[60:22.18] day with, with no results, I think that this is a huge incentive for like China and like
[60:27.74] Russia to put out their own models because models are soft power.
[60:31.70] Like the best way to control how someone thinks is to go in and provide their thinking assistance
[60:39.22] and like subtly make changes.
[60:40.58] Like, you know, it's too on the nose to be like, Oh, I don't know what Tiananmen Square
[60:44.46] is.
[60:45.46] You know, like, but if you have like subtle ways of affecting the biases of your decisions,
[60:52.14] your reasoning, your, you know, your, your knowledge in, in the LLM and in publishing
[60:56.18] a really, really good LLM for everyone to use so that they're like, Oh yeah, this is
[60:59.42] great.
[61:00.42] You know, and I use them as maybe a leading LLM, then they will just like uncritically
[61:04.78] accept that as like state of the art, you know, this digital intelligence and that becomes
[61:09.54] soft power.
[61:10.54] And that translates into unconscious thought a lot of times.
[61:14.46] Yeah.
[61:15.46] Yeah.
[61:16.46] I think the prompt point, it's great.
[61:18.78] You know, you just got to, you just want to see what it is, you know, like, you want
[61:22.10] to understand.
[61:23.10] Yeah.
[61:24.10] Yeah.
[61:25.10] And then the model side, I think there's just some things are too, they're almost you cannot
[61:29.94] but like the Elon Musk, Twitter meme or Hitler bring more harm to humanity and the, and Gemini's
[61:36.06] like, Oh, it's hard to say if Elon Musk tweeting or it is like, what, how, what, there's something
[61:41.50] wrong in the data pipelines, you know, like there's something wrong somewhere.
[61:44.94] Yeah.
[61:45.94] But like this is like to an LLM, this is the same class of error as which is heavy, which
[61:50.78] is heavier.
[61:51.78] One pound of feathers or one pound of bricks.
[61:54.42] But, but then like, but, but to me, the, the point is more like, okay, then one can we,
[61:59.54] what can we help these models do, you know, because if they cannot, if the, the physical
[62:04.38] stuff, I get it because it's like the whole like world model thing.
[62:07.62] But then it's like, okay, can we expect the models to say what's more harmful than something
[62:12.58] else?
[62:13.58] Maybe not.
[62:14.58] That might be where we land.
[62:15.58] Then it's like, okay, that's one more thing.
[62:16.94] And then we kind of go down the line and it's like, what are these models good for?
[62:20.74] If anything is too like hard for them to pick up when it's like A or B, but we'll see.
[62:26.02] Yeah.
[62:27.02] Okay.
[62:28.02] So, I mean, you know, I know we're up on time.
[62:29.02] It, like this has been an eventful month.
[62:31.06] I think, you know, February was a lot more interesting than, than January.
[62:34.22] In fact, a lot of my January recap was like how nothing's changed.
[62:37.50] And then February came out and it was like very, very interesting.
[62:40.14] So yeah, we hope to see what's next.
[62:42.10] I think we have a, also this was the month that we did compute provider month.
[62:45.86] I think relatively successful, surprisingly hard to like string together all these, uh,
[62:50.70] providers.
[62:51.70] Yeah.
[62:52.70] We did it.
[62:53.70] People like it, you know, based on the post stats, uh, maybe we'll do, we'll do something
[62:58.98] else.
[62:59.98] Yeah.
[63:00.98] If you want, you know, if anyone listening wants more sort of thematic explorations of
[63:03.26] like, okay, these like three, four companies are always come out together.
[63:06.30] Like let's get, let's get a focused, uh, effort on, on those things.
[63:09.46] I think we're open to doing that.
[63:11.02] We, you know, and then obviously we'll have opportunistic interviews along the way.
[63:14.98] Cool.
[63:15.98] Thank you everyone for tuning in and yeah, keep the, the feedback coming.
[63:20.22] That was the latent space recap of January and February, 2024.
[63:28.78] If you have any feedback or questions, please head to the show notes for ways to get in
[63:33.12] touch with us or come by the latent space discord.
[63:36.78] For those who just want the core content, you can stop listening here, but for the super
[63:41.70] fans, you might notice that there's 45 more minutes of audio left in this pod.
[63:47.34] That's because in February, we also celebrated latent spaces first anniversary.
[63:53.66] Some of you may remember how we launched our very first episode with Logan Kilpatrick,
[63:58.50] now formerly of open AI and a massively popular demo day.
[64:03.46] Click through to the show notes for photos over 750,000 downloads later, having established
[64:09.18] ourselves as the top AI engineering podcast, reaching hash 10 in the US tech podcast charts
[64:15.26] and crossing one million unique readers on substack.
[64:18.74] We celebrated with latent space, final frontiers, a combination demo day and birthday celebration.
[64:25.74] We're going to bring you some snippets from the demo day and then some conversations with
[64:29.66] listeners from all over the world, from Hungary to China to my own sunburnt country down under
[64:35.42] on how the issues we've covered in latent space has impacted their lives.
[64:40.46] First up, we'll have a demo from Florence Crovello from Lindy AI, who gave a great keynote
[64:45.22] at the last AI engineer summit, and recently opened up Lindy AI to the general public.
[64:50.42] We're just chatting right now with this week, we come with 3000+ integrations out of the
[64:55.14] box, we have a partnership with Nathan, which is like an open source app here, so we have
[64:58.66] like a ton of integrations out of the box.
[65:00.78] So unlike, feels I shall not name, we don't require you play with open API specs or anything
[65:05.22] like that, you just go and select your integration here.
[65:10.54] All right, so that's my Lindy, oh, something even cooler, Lindy's can work together.
[65:15.18] So here I'm going to let her work with a support reporter that I created before, and the support
[65:19.26] reporter what he does is, it receives details about the support tickets, and it logs them
[65:24.54] in a spreadsheet.
[65:25.54] So you can have, it's sort of like object oriented programming for agents, so you can
[65:28.46] create as many agents as you want, and let them work together.
[65:32.06] So here I'm going to tell her, when you're done, give the details of the ticket to the
[65:40.22] support reporter.
[65:42.06] All right, and now I'm going to send her an email.
[65:48.22] Can I have a refund please?
[65:50.02] Please, my family is starving.
[65:57.58] You will see she has no empathy whatsoever, it's awful.
[66:03.34] So she received the email, she's subscribing to this thread, so now she's going to receive
[66:08.26] a reply.
[66:09.26] Dear Flo, I understand your situation and I'm truly sorry to hear about the difficulties,
[66:14.34] but we absolutely do not have a refund.
[66:16.34] All right, yeah, this is good, Lindy.
[66:18.66] So she sends the, oh, well, the demo effect, she did not delegate, but she sends the answer
[66:25.98] in the thread here.
[66:28.10] So again, Lindy.ai can be used for support, for executive assistance, email drafting,
[66:33.54] email triaging, meeting, recording, and we are hiring software engineers.
[66:37.34] Hit me up at the floor at Lindy.ai.
[66:40.74] Thank you.
[66:41.74] Our next demo is one of our previous guests, Eugene Chi from RWKV, now also CEO of Recursal
[66:49.14] AI.
[66:50.66] You can listen back to our original RWKV episode to learn the full history and details of the
[66:56.22] model, but also compare it with his more polished pitch now for a more general audience.
[67:01.66] Next, I think we have Eugene Chi from RWKV, a previous guest.
[67:10.88] I'm going to present about the RWKV/EU project, so beyond Transformers, there has been a lot
[67:16.62] of excitement lately, and like one AI year ago, apparently we launched our second B model.
[67:24.18] There was a lot of excitement in the bus because for the first time, an attention-free model
[67:29.34] beat other Transformer models at 1 trillion tokens at a 7B class.
[67:34.54] And if everyone's been playing with open source AI, you know 7B class is one of the most important
[67:40.46] class because it's the one that works on most devices, laptops, and everyone's been playing
[67:44.38] around with it.
[67:45.38] And the excitement is compounded by the fact that we even showed that even with 300 million
[67:49.34] tokens in a few, that we perform similarly Transformers, that means people are projecting
[67:53.30] is what happens if we train another 1 trillion, will we match or beat Lamma, or can we go
[67:58.18] beyond that?
[67:59.18] And it also spurs up questions beyond actually our architecture itself.
[68:04.10] It spurs up questions that maybe what we need is good data and efficient architecture, not
[68:09.98] just RWKV, it could be beyond that.
[68:12.54] And that's what caught the attention for a lot of folks.
[68:17.38] And what we do very different is that our architecture scales linearly.
[68:20.86] So we are in this space together with Mamba and a few other architecture where we are
[68:24.62] trying to build the next architecture that can scale much larger for everyone.
[68:32.94] But we share that with Mamba because we believe that attention is not all you need and it's
[68:37.90] been around in bed right now and we are the strongest evidence to date.
[68:42.34] But sometimes talking about scale, sometimes we get lost in numbers because I can show
[68:48.42] this, and the last time I showed this at a linear transformer event which only 8 people
[68:55.14] took pictures of it and understood what it means and they were all from either Google
[68:59.26] or Facebook.
[69:00.26] Because what it says here is that we are able to run on a single GPU with one model, 256
[69:10.30] on a single 49G, or a thousand concurrent users, but to put that into contrast, what
[69:17.34] the mid-transformers typically handle 8 or 16 concurrent requests per GPU.
[69:22.10] We are talking about 256 or a thousand, many orders of magnitude higher, and all we are
[69:27.26] sustaining at near-checked GPU speed.
[69:29.54] And so sometimes when I get lost in these words, these days I am actually trying to
[69:34.62] step back into like, why are we doing this for our group, for our organisation?
[69:40.66] And for us, we are actually making the AI model for everyone in the world.
[69:47.22] And in every country, in every language.
[69:50.74] So what does it take to make an AI for the world?
[69:54.70] Apparently some folks think it's $7 trillion.
[69:58.54] But I think $7 trillion is a bit too much.
[70:01.70] What's going to happen to a half of the world that doesn't even have $1 trillion?
[70:05.70] So I want AI to be accessible at scale.
[70:09.82] So apparently CheckGPT produces 100 billion words per day, that's 28.4 million tokens
[70:17.58] per second.
[70:19.08] No one can have the exact numbers, but it's typically 50K, H100s and above.
[70:24.78] I mean, these are some old numbers, like the numbers have gone way beyond this apparently.
[70:30.62] But with our architecture, for a 7B model, that's just a thousand GPUs, or 10,000 GPUs
[70:36.50] for a 7B model.
[70:37.50] We are talking about one data centre to handle all of open AI's workload.
[70:44.02] And if we want AI agents everywhere, cheaper at a much larger scale, we need to be thinking
[70:49.70] about that fundamental shift, because it's not just about who can, it's not just about
[70:54.50] you can afford it in the US, it's about everyone else in the world.
[70:58.70] And that brings us to the second advantage of our model, which is not even architecture,
[71:04.62] because we are accessible by language.
[71:07.26] We apparently beat MISTRO and everyone else in multilingual, but that's not because our
[71:13.82] architecture is better, but because we are an open source team that came from all around
[71:18.46] the world and wanted our model to work for our mum and grandma.
[71:22.34] That was the real reason, and we iterated and refined the data accordingly.
[71:26.38] We created a custom tokeniser that supports all languages, not just English, and sometimes
[71:34.22] in the race for the English benchmark, because one of the reasons why other models don't
[71:38.34] perform as well in multilingual is because the truth is, if you add multilingual, you
[71:43.30] will hurt your English evals.
[71:45.42] But who are we building the AI for?
[71:47.94] Are we building it for our evals, or are we building it for the people to use?
[71:54.78] And even in evals, my frustration is, we train our 100 language, our only 23 language for
[71:59.74] evals.
[72:00.74] Where's everything else?
[72:03.18] So, we are in between the 1.5 trillion and the 1 trillion models for all the English
[72:12.10] models benchmark, and zooming in further, it just shows that we have more room to go.
[72:18.78] And for me, the emphasis on English is weird because only 70% of the world speaks English,
[72:25.50] but we are here for the 83%.
[72:27.90] That's for us.
[72:28.90] If you all want to get the best English model, sure, it may not be true us, but we are here
[72:32.94] for everyone else.
[72:35.10] And yeah, the launch of that model, I think what was the biggest feedback I had was not
[72:40.94] that it was a linear transformer, was that it can run on their laptops.
[72:47.14] Some people even ran it on a Raspberry Pi very slowly, and it supported their language,
[72:52.38] which was more exciting because that's more important for most people.
[72:57.86] And I think the last one that I've recently heard that was unique for us that's a lot
[73:03.06] more important is that, ultimately, this model is owned by everyone because we put it into
[73:08.50] the Linux Foundation.
[73:09.90] No custom charity, no custom board structure, no weird stuff.
[73:13.98] We just put, we just train the model, put it in an open source organization, that means
[73:17.78] it's not owned exclusively to us.
[73:19.86] If I go rock one day, you can just, the code will not disappear.
[73:23.18] The model will not disappear.
[73:24.18] The Linux Foundation has already fallen into it, and that is to all of you here.
[73:32.02] And so what's next for us?
[73:33.02] Well, we recently started a commercial entity, I know that's weird to say after the open
[73:36.94] source stuff, but we, and since then we managed to get more investors and sponsors that we
[73:43.86] started our next major training.
[73:45.10] So we are training the next one trillion token, this is 16 H100 nodes, eating enough
[73:50.38] electricity for multiple homes and by the end of next month, we'll have our two trillion
[73:58.82] transformer alternative that you can do one-to-one compare with Lamar.
[74:04.02] And of course, because since we had to make a profit somehow for our investors, we are
[74:08.26] launching our platform also to host, train, and fine tune our models all in March 2024.
[74:16.18] Final Frontiers had a stellar lineup of demo judges, featuring CEOs and VPs of AI from
[74:44.62] Lamindex, Replit, GitHub, AMD, Meta, and Lemurian Labs.
[74:52.42] RWKV won one of two judge prizes available that night, alongside with this next startup,
[74:58.74] Pixie AI.
[74:59.74] Next up, also in the automated workforce category, Pixie.
[75:05.70] Awesome.
[75:06.70] Hi everyone, I'm Ryan.
[75:07.70] I'm a software engineer on the team building Pixie, pretty straightforward, automated
[75:10.58] security.
[75:11.58] A little bit about myself, previously I've worked at other security companies building
[75:15.66] developer facing security tools.
[75:17.86] I've also worked as a security engineer on developer tools.
[75:21.04] So this is a space I love, I'm really interested to see how it develops.
[75:25.56] Why are we doing this?
[75:26.62] So as it turns out, we're generating a lot more code.
[75:30.66] So this is an example, a user of Pixiebot, it's a repository called Sterling PDF, it's
[75:36.26] just a web application, it's got 18,000 stars on GitHub, developed using 100% using chat
[75:43.18] gbt.
[75:44.18] So they installed Pixiebot three weeks ago, and they got a lot of different suggestions
[75:48.62] for fixes for us, one of which was, I'm positive, was a real vulnerability.
[75:54.70] This is a, you know, web application that's used by real people, there's a button here,
[75:59.46] you can deploy it to digital ocean.
[76:00.86] So we need to find a way to scale our security automation in order to scale our relatively
[76:05.86] limited security workforce.
[76:08.14] So just to give you an idea of what Pixiebot can do, this is like a very classically vulnerable
[76:13.50] application that a lot of security tools like to try themselves out on.
[76:17.42] One of the things that I'm really excited about that we just shipped on the past couple
[76:20.14] of weeks was integrating with Sonar.
[76:22.66] So Sonar is a code quality tool, that Sonar is a code quality tool that finds security
[76:26.50] issues, performance issues, lots of kinds of issues in your code.
[76:30.94] It also, as you can see here, found 2,600 issues in here, taking 33 days of effort.
[76:39.58] That's not really where we want to have most product engineers focusing their time.
[76:44.02] It's definitely not where we want to have our security engineers focusing their time.
[76:48.14] What can we do to automate this and get these fixes automatically?
[76:51.60] So with Pixie, we take these code quality security issues in from these other tools
[76:55.98] and then automatically remediate them.
[76:57.94] So in the case of this, this is a super minor change.
[77:01.76] If a developer were to find this issue in their code, they could fix it in a minute.
[77:06.06] But they don't have to.
[77:07.22] And more importantly, there's backlogs of tens of thousands of these issues in organizations
[77:11.74] across the world.
[77:13.86] And so if we can automate this one task even if it just takes a minute and perform that
[77:18.58] continuously across thousands of companies, we can save a lot of time.
[77:23.54] Imported enforcement of security and code quality is what we're all about, but yeah.
[77:29.02] Not all security issues are worth fixing.
[77:30.98] Not all code quality issues are worth fixing.
[77:32.98] Sometimes they're wrong.
[77:33.98] The incentive structure for these tools is, you know, they want to find real things, but
[77:40.38] most importantly, they have to find something.
[77:42.54] So at Pixie, we believe even if something might not be a complete, exploitable vulnerability,
[77:48.50] if there's an opportunity for hardening or improving your code base, you should probably
[77:52.50] take it.
[77:53.50] But there's some of these things that are just not that.
[77:55.10] So we developed a tool we call triage, which we'll connect in with, uh, with other tools
[78:00.30] that are notorious for finding lots of issues and we can help you fix them.
[78:05.66] So in this case, we made a CLI that looks at your security backlog and identifies issues
[78:10.74] that we know don't matter in the context of your code base.
[78:14.50] It pulls it.
[78:15.50] It pulls down the issues, categorizes them and then enables you to prompt, uh, it prompts
[78:19.54] you to either say, Hey, this issue is not important.
[78:23.34] Here's why we think it is, uh, and we'll update the state for it.
[78:26.22] Uh, in this case, this is worrying about a parameter into a, this file directory.
[78:31.34] It has some cross platform compatibility concerns, but based on the context of your code base
[78:36.42] and a large language model, we're able to give you the confidence to focus on the issues
[78:41.82] that are most likely to actually matter.
[78:43.82] And one of the other things we do is, you know, well, we're delivering what you saw before
[78:49.02] is we're delivered as a GitHub app that we're delivered as a GitHub app so that developers
[78:53.86] can integrate this into their existing workflows.
[78:56.50] But a lot of people like to just try a Pixie from the command line on small projects, automatically
[79:00.90] get their fixes and just commit all of them.
[79:02.94] So that's what we built.
[79:04.10] So try Pixie on GitHub, try Pixie on a CLI, and we're really excited to see what we can
[79:10.26] help you fix.
[79:16.62] Congrats to Pixie and RWKV.
[79:19.38] Our last featured demo is Rahul from Julius AI, who provides an interesting take on competing
[79:25.48] with OpenAI on its own home turf, the chat GPT code interpreter.
[79:30.74] You might remember Rahul, that's the board engineer, they got laid off by Elon Musk.
[79:35.70] He's back, he's back on the speed, he's got a whole new start.
[79:40.70] Thanks so much for having me here, I'm working on Julius.
[79:43.74] How many of you here are data scientists?
[79:46.70] I think everyone here needs to be data scientists, but there just aren't enough and that's what
[79:50.46] we're building.
[79:51.46] Julius is an AI data scientist that helps you analyze data sets, create visualizations,
[79:56.02] get insights from the data, and really dive deep into all sorts of data we have in our
[80:01.82] life.
[80:02.82] So we launched about six months ago and since then have grown to 300,000 users, several
[80:08.98] thousand users using this daily to analyze data sets, create visualizations, and get
[80:14.46] insights.
[80:15.46] So what I'll do now is give you guys a quick live demo of how that actually works and I
[80:19.86] actually hope it works, because it's a pretty simple protein list, but here I have a data
[80:24.18] set of 20,000 rows of data over time for the last 100 years with human height for different
[80:30.58] countries.
[80:31.58] So I'm going to take this data set, dump it in Julius and say, load this for me.
[80:42.26] And while it's doing that, I want to explain what's happening under the hood.
[80:46.46] So basically, for each user, think about how a human data scientist would analyze it because
[80:52.90] they didn't give it.
[80:53.90] It would take its computer, a write code, run that code, maybe in a Jupyter number.
[80:59.90] Look at the output and then decide if that answers your question or if you need to write
[81:05.78] more code.
[81:06.78] Julius works similarly.
[81:07.98] So that's the AI and then for each user, you get a virtual machine in the cloud where
[81:14.50] the AI is putting a Jupyter in our book, writing the code, you get the analysis that you want
[81:20.10] and then serving that back to you.
[81:22.66] Many times that code is not correct the first time, but Julius isn't able to recover from
[81:27.54] those errors and actually gets you the answer that you want.
[81:31.94] So let's look at our chat.
[81:32.94] We said, load this file for me and the AI basically went spun up a Jupyter number book, loaded
[81:38.26] pandas, looked at the file and gave us a few rows.
[81:41.78] I'm going to ask, plot the mail height over time.
[81:53.82] So the AI basically writing this code, you get this height over time frames for men and
[81:59.70] then plotting that for us.
[82:00.70] And the other thing about Python, if you try to put a time on SQL, what we realized was
[82:06.78] that SQL, it's really hard to write actually useful queries and do deep analysis like regression,
[82:14.74] etc.
[82:15.74] It's just SQL.
[82:16.74] With Python, you also get a whole ecosystem of modules built in like Mathwa, Potli, Pandas,
[82:23.94] NumPy, SQL, there's thousands of these.
[82:27.94] So that was the initial insight and then we built Julius about six months ago.
[82:33.14] But what's like the practical difference in UX between this and just traffic between
[82:37.50] code and Jupyter?
[82:38.50] Great question.
[82:39.50] Yeah, the question was, where is the difference between Julius and code and Jupyter?
[82:41.50] Really, there isn't.
[82:42.50] It's just better.
[82:43.50] We're focused.
[82:44.50] We're focused.
[82:45.50] We're focused for people or people who do stuff with data multiple times a day.
[82:53.54] And we talked to all these people and said, okay, how can we build things for you that
[82:57.54] would help you do your job?
[82:59.26] So an example of this is on chat GPD, oftentimes you'll give it a data set, it will program
[83:05.62] the code and sometimes that code has errors and it kind of goes into this loop of trying
[83:11.94] to fix this on errors.
[83:12.94] What we're focused on is, okay, how do we prevent that from happening?
[83:16.98] So we looked at thousands of users using this daily, collected data on where these errors
[83:22.10] happen and focused real hard on fixing those beforehand before they actually happen at
[83:29.58] one time.
[83:30.58] This could mean a bunch of rules.
[83:32.58] This could mean prompting changes, et cetera, and just preventing that from happening.
[83:38.06] So first of all, we have features that allow people to do stuff with data on daily basis
[83:44.18] to code deep and do the last mile of analysis doc that could mean you can click show code,
[83:51.38] go into the code, edit the code changes, you can also give natural language instructions
[83:55.98] on the code.
[83:57.98] Finally, let's say you have this graph, and I want the graph to have some changes, like
[84:02.86] I want it to be a bar card instead of a line graph, and you kind of just go in here and
[84:09.18] give natural language instructions to let the user take what the AI has done for it and
[84:14.86] then take it to the finish line.
[84:16.86] You've seen that code interpreter that's really hard for users to do, so you focus on data
[84:21.70] and that use case and do it.
[84:25.70] Cool.
[84:26.70] Thanks, guys.
[84:28.18] That's unfortunately all the time we had to feature demos.
[84:31.30] But many thanks to BotPress, Markov, Cura AI, Sweep, and Motif as well for being finalists.
[84:37.94] For the last part of our anniversary celebration, we wanted to turn over the mics to you, our
[84:42.86] dear listeners.
[84:44.22] We hear so many great stories from listeners about how latent space has come into their
[84:48.62] lives, and we've never had the opportunity to feature them on the pod till now.
[84:53.86] Our first listener is Belaz Nemethi from Hungary, who talked about one of the most delightful
[84:59.54] gems in the latent space community, a weekly Discord paper club.
[85:04.40] Yeah, like two weeks ago, there was the paper reading club on Discord, and then halfway
[85:14.38] or like one quarter in, the author of the paper showed up, and it was so fucking cool.
[85:20.94] If you could do this, I was thinking this should be a format.
[85:25.14] There is two minutes papers that probably, you know, he's Hungarian, living in Vienna,
[85:32.52] but like "károly" in Hungarian is "károly", yes.
[85:38.86] So that was so special because there is certain amount of information in papers, the quality
[85:46.00] of paper might have dropped in the past year than before due to the social media aspect
[85:51.02] of archive.
[85:52.34] So having the person there and giving in even more details than just what you could read
[85:58.62] was like so amazing.
[86:00.10] I know it's really hard to organize, but like if it would be possible to have more, maybe
[86:05.50] not recurring, like you know, it's just like, oh nice.
[86:09.02] The Matryoshka Representation Learning paper, yeah yeah.
[86:13.62] So we have one next week, the MRL paper, Matryoshka Representation Learning, which is a way of
[86:18.26] sorting and bidding so that you can truncate them.
[86:20.50] And opening, I recently shipped this in their API for the new embeddings models where you
[86:24.74] can reduce like a 3,000 vector embedding to 265, so you save more than 90% on your vector
[86:31.10] database costs and speed and everything.
[86:33.96] So the authors are coming by and presenting the discord.
[86:38.30] I will join.
[86:39.30] I will join.
[86:40.30] So basically I'm just going to record random opinions, I know, so we're going to do this,
[86:47.30] you're going to be on the show, you're going to be on the show.
[86:49.98] Any other, like how did you discover the podcast, what do you feel?
[86:54.86] Discovered it on Spotify, searching basically AI.
[86:57.74] I use Pocketcast for all my podcasts, but I was like, let's just search AI.
[87:03.10] I think I was searching for AI-generated music, but it brought up podcasts and I was like,
[87:09.34] you know what, I'm kind of getting out of my previous industry, so like I'm just going
[87:14.94] to separate the whole AI following thing, and I just followed, this was the first one
[87:21.38] that came up, and then a couple of others just to have it downloaded, but this was literally
[87:28.82] the first podcast I'm following on Spotify when I followed like 70 on Pocketcast.
[87:33.98] So I was like, and I started and I was like, okay, this is great, or they're only great
[87:38.50] podcasts and I kept coming back to yours.
[87:40.78] There are the podcasts that we consider friends and we try to do collaborations with them
[87:44.54] and podcast swaps with them, so yeah, that's great.
[87:47.54] Our next listener is Sylvia Tong, founder of the Entraconect Community, a community of
[87:52.62] founders and investors supporting entrepreneurs in Silicon Valley.
[87:56.90] She wanted to discuss OpenAI Sora and Jim Fan from NVIDIA, who we have featured on our
[88:02.38] previous OpenAI Dev Day Recap podcast and will be a future guest on Late In Space.
[88:07.78] How did you find the podcast, and what do you feel about it, what do you want to tell
[88:10.50] people about it?
[88:11.50] Actually, I know Jim Fan, so Jim Fan, I know you, and then I follow your Twitter and found
[88:18.46] your podcast.
[88:19.46] Yeah, yeah, yeah.
[88:20.46] It's another event, maybe you know Alliance AI, it's another community, anyway, like they
[88:28.26] had that event like last year, so they found this event, they are the founders of Stanford,
[88:33.58] so they are Stanford grads, so they are always in the Stanford University, like one of the
[88:38.26] rooms.
[88:39.26] Yeah, so Jim Fan is one of the first speakers, so yeah, I can now with him, we try, yeah,
[88:44.54] now it's you, yeah, follow your Twitter.
[88:48.46] Jim is super friendly and we have to have a full episode with him at some point, but
[88:54.34] he's, yeah, I mean he's doing amazing things at NVIDIA, I'm sure he's very happy there.
[88:58.26] You should ask him about Sora, the J-A-I video, yeah, he has so many opinions about, you
[89:06.34] know, yeah.
[89:07.66] I feel like, okay, Jim is this interesting mix between a researcher and a content creator,
[89:13.10] right, so Jim's take on Sora I slightly disagree with, because he says it's basically a data-driven
[89:19.38] world model, and a lot of people misinterpreted him, me included, basically saying like,
[89:25.58] oh, are you saying that there's an underlying physics model behind Sora?
[89:28.82] And he's like, no, no, no, no, no, it's just, you know, using diffusion transformers to
[89:32.98] learn a representation of world models, it's not perfect.
[89:36.26] Then I'm like, okay, but like that's a misleading analogy, I don't know, anyway, so like he-
[89:40.46] But that's for the content purpose, that's for the Twitter content purpose, you'll have
[89:44.10] to, yeah, yeah, yeah.
[89:46.34] So I feel this like, pulled towards like celebrating things on Twitter, but then also trying to
[89:52.70] be realistic, trying to present like, what is actually the thing instead of the hype?
[89:56.78] And it's very hard to separate, and that's something that's a challenge for Lane Space.
[90:00.30] Yeah, it's hard, I feel it's hard to have the conversation on Twitter, so you need to
[90:04.66] have conversation in the podcast.
[90:06.54] So invite a few people who maybe have to talk about Twitter, but really explain what I mean,
[90:12.26] meaning our tweets, because yeah, it's hard to understand just a few words, yeah.
[90:16.58] So do you actually think Sora understands the physics of the world?
[90:23.26] Sora understands a little bit of physics.
[90:24.98] The problem with this is they cannot have 80% physics, like, it's 100 or zero, like,
[90:31.14] otherwise you lose confidence in the thing.
[90:33.86] So that's why you have like these generated models where like the chair will show up and
[90:38.38] disappear, the spoon will show up and disappear, you know, like that's also the artifacts you
[90:43.30] see in Sora, which is good for us for now, because we're lucky that it's not good enough
[90:48.34] yet to like consistently generate all those things.
[90:50.70] At some point it will be, we just wait two years, and it will be.
[90:53.86] Very cool.
[90:54.86] Thanks for it.
[90:55.86] Well, I love this discussion.
[90:56.86] Thanks for listening, I'm really glad to have you as a listener.
[90:59.78] Alessio and Swix covered the Jim Fan versus Jan Lacan world model debate in the main pod,
[91:05.90] and you can click through the show notes for more detail directly from each of them.
[91:11.10] Our third listener is RJ Honike, who comes from a data science background, but wanted
[91:16.50] to ask about how we think about learning in public in AI and how that informs the context
[91:21.38] with which latent space is created.
[91:23.34] Hi, I'm RJ.
[91:25.74] Sean, nice to meet you.
[91:27.90] Do you also listen to part or are you just here to hang out?
[91:30.66] Oh yeah.
[91:31.66] How do you feel about it?
[91:32.66] The depth that you guys go into, it's a lot deeper than other podcasts that I've listened
[91:36.94] to and I kind of found it and then didn't switch back.
[91:39.58] Thanks.
[91:40.58] Yeah.
[91:41.58] What's your background?
[91:42.58] I am a data scientist.
[91:44.62] I run a data team at Cell Communications Equipment Manufacturer, and we collect a ton of telemetry
[91:50.58] data in other things like that.
[91:53.90] I'm running a data team to make inferences about the health of our network, about operating
[92:00.26] the network more efficiently, and also in our manufacturing process and product development
[92:05.18] process to improve our ability to detect when we improve or get worse at operating or our
[92:14.26] product builds or hardware builds get better or worse.
[92:18.90] I wanted to actually ask a question of you and your thoughts about this.
[92:22.82] I find the discussion about model measurement and evaluation to be very similar to the problems
[92:30.94] that we have in wireless because you have this very non-deterministic system.
[92:37.50] I was thinking, and I also just read your little thing about learning in public.
[92:43.70] I was thinking about trying to come up with a good way to, and I'm learning about some
[92:51.74] new techniques that we're starting to implement to monitor our development process and so
[92:57.50] forth and evaluate the quality of our builds and our hardware.
[93:02.40] I was thinking about trying to tie that in with evaluation of LLMs.
[93:08.50] I don't know.
[93:09.80] That's as far as I got in the thinking, but I just thought that would be a fun thing to
[93:13.50] try to put out there and wanted to hear your thoughts about how to go about that.
[93:18.54] You don't need anyone's permission.
[93:20.46] That's the beauty of this thing.
[93:21.92] But also, no one owes you anything, no one owes you their time, their attention, or responses.
[93:29.48] I typically try to classify these things as different modes of learning in public.
[93:33.98] I think I have four modes that I sketched out, but the two I remember the most are Explorer
[93:39.08] and Connector, and then there are two more advanced modes, I think like Teacher or Builder
[93:44.76] or something like that.
[93:46.00] The Explorer is where you sort of put things out as you go along, is learning exhaust,
[93:51.60] where you don't have expectations so that anyone will read it.
[93:54.60] It's mostly just notes for yourself.
[93:56.40] And then actually, that lack of expectations frees you because then you're like, oh, like
[94:03.00] two people read it.
[94:04.00] It doesn't matter.
[94:05.00] It's useful to me.
[94:06.00] It's useful to my team.
[94:07.00] It's useful to me.
[94:08.00] It's useful to whoever comes after me because I documented my work and my thinking.
[94:10.40] And that's great.
[94:11.40] And I think that's the way that most people should start, which is just lower.
[94:15.80] You're not going to be an influencer overnight.
[94:17.44] It's fine completely.
[94:19.16] You get your thoughts out there, but also start having feelers in different directions
[94:24.12] on what works for you.
[94:25.94] What works is a combination of what you like to do and what other people want from you,
[94:31.16] and you will know when people tell you they want more from you.
[94:35.16] And so then when you get there, when you have expertise that you have that other people
[94:39.80] don't, then you switch gears into a connector where you are now coming from a place of authority.
[94:45.32] I know how to do this right.
[94:47.32] And I will teach you because I have done this and I have spent more time, paid more in my
[94:52.64] dues, and here's the lessons.
[94:56.32] And then that tends to become more of a polished effort.
[94:59.72] That tends to become more measurable or in terms of the impact and the influence it can
[95:03.76] get.
[95:04.76] And I think that's where people start moving towards.
[95:07.20] But basically just lower expectations, make it cheap to experiments, put out a lot of
[95:10.76] stuff in different directions, and see where the market pulls you from.
[95:14.12] Yeah, so I mean, do you have thoughts about, like, I'm very much aligned with who cares
[95:22.08] about, I mean, I care, but I'm not, my need is not to be a social media influencer.
[95:30.00] My need is to, I want to learn and I like the idea of sharing that with people and sharing
[95:38.08] the process with people.
[95:39.36] So, like, thoughts about platform or, like, I mean, I know you, it's going to be different
[95:46.20] for everyone, but like, what's, what's in your experience has been successful while
[95:52.84] getting started?
[95:53.84] Yeah, so I tend to tell developers, most developers to start on a passion note these days, hash
[96:00.60] note is basically a medium, if it was for developers and didn't suck, because I hate
[96:06.56] medium with a passion and a glowing fiery hatred, everyone does.
[96:10.96] It's comical how bad they are.
[96:14.04] But I use Substack really in space, I'm pretty happy with Substack, it's an email social
[96:18.56] network, emails are the most important things for people to, like, come back to you frequently,
[96:22.72] so that you don't, you're not subject to an algorithm, you own your audience, you know,
[96:26.56] if you want to move all Substack someday, they'll let you take the emails and keep that relationship
[96:30.40] going with the people that you have, and that's super important as a creator.
[96:34.80] And then you can also write your own blog and tweet and tweet and all that.
[96:38.12] I tend to say though, pay attention to what you enjoy and what you spend the most time
[96:42.64] on.
[96:43.64] If you're a LinkedIn guy, be on LinkedIn.
[96:45.80] I'm not on LinkedIn, so I'm going to do horrible on LinkedIn, because I don't know the metagame
[96:48.96] of LinkedIn.
[96:49.96] I don't know what does well, I don't know what people want.
[96:51.96] So I shouldn't even, I don't, I don't bother, I should try, because obviously there are
[96:56.12] like way more people on LinkedIn that are on Twitter, but I'm just a Twitter guy.
[96:59.40] Like I'm, that's just, that's who I am. I have, I have, I also sort of am old money
[97:03.64] there in the sense of I have an existing followership that predated LinkedIn space.
[97:08.28] You know, LinkedIn space doubled my following, but like I had some before that.
[97:12.48] So like all that's great.
[97:14.12] I just think like you are going to know the metagame and that's actually very important
[97:18.40] of like where you already spend time.
[97:20.04] Like I have friends who are like on TikTok and friends who are on YouTube a lot.
[97:23.84] I'm on YouTube a lot.
[97:24.84] I should do YouTube because I know, I know what's, what's going on on YouTube.
[97:27.52] It's just, then you have to put the effort to do that and I'm, I'm like video production
[97:31.28] is like the most expensive thing.
[97:32.84] Anyway, long story short, try to pay attention to complex makes of like publishing platform,
[97:39.64] existing embedded social network on that platform and where you already spend times so that
[97:43.92] you know how to create what will do well just because you already spent time on it.
[97:46.96] Yeah.
[97:47.96] Okay.
[97:48.96] What's your favorite?
[97:49.96] What, what, what favorite episode?
[97:50.96] I really liked actually the, the NURPS like recap because I haven't been in NURPS.
[97:55.96] How much time that took?
[97:57.32] Well, I mean the episode is like four hours, right?
[98:01.20] Holy shit.
[98:02.20] I, and that one, I didn't, I didn't do the paper one because I, I actually, I usually
[98:07.20] listen.
[98:08.20] I don't watch.
[98:09.20] So I like, it's really hard to.
[98:11.68] There's no video for that.
[98:12.68] Oh, there isn't.
[98:13.68] Oh, okay.
[98:14.68] So I like, I have to find the paper and anyway.
[98:16.32] Yeah.
[98:17.32] So that's hard for me.
[98:18.32] Yeah.
[98:19.32] Right.
[98:20.32] But I did enjoy the interviews in the other.
[98:23.44] The startups.
[98:24.44] Yeah.
[98:25.44] I love that.
[98:26.44] It's just, it just takes a ton of work and I would love to offload it.
[98:29.84] This is going to be another one of those where I just kind of slip together little things
[98:32.48] and it's good.
[98:33.48] It's, it brings you there.
[98:35.04] That's the thing, right?
[98:36.04] Like you, you're not there physically.
[98:37.04] I'm here.
[98:38.04] That's like bringing people into the, the, the closed community.
[98:40.96] And so I would like to do more of that.
[98:42.64] Yeah.
[98:43.64] No, I really enjoy how you bring like a lot of people that I would not have otherwise
[98:50.12] even known about let alone have access to and then you have this conversation with them.
[98:54.72] It's really fun.
[98:55.72] Yeah.
[98:56.72] Yeah.
[98:57.72] Thanks for coming on.
[98:58.72] Can I, can I get your contact so that we can find you?
[98:59.72] Yeah, yeah.
[99:00.72] Yeah, you're going to be on the pod.
[99:01.72] Oh, awesome.
[99:02.72] People seem to love the New Reap's Recap pod and we'll keep doing more of those when
[99:06.46] the right occasion presents itself.
[99:09.66] This was also a pick for our last listener, Jan Jong from Australia, who comes at AI from
[99:14.66] the design point of view and was very interested in our early AI UX work on latent space.
[99:20.64] If you're in SF and want to more novel AI UX ideas, reach out to him.
[99:26.56] My name is Jaan and I came across you on GitHub when I was looking for ways to solve
[99:33.26] problems on Svelte and you pretty much answered all the question I had for pretty much a couple
[99:40.36] of years.
[99:41.36] And then you left and you started doing latent space and like, what is that?
[99:45.56] What is an LLM?
[99:46.56] So I started listening to your pod and, and here I am.
[99:49.36] And then you're part, you're, you're from Sydney or you weren't, you weren't Sydney?
[99:52.38] I, I moved to Sydney a couple of years ago to work on a clinical trial, but now I moved
[99:57.82] back probably again, I blame you for it because I listened to every episode and I'm like, shit's
[100:02.96] going down in San Francisco, I got to be here.
[100:05.72] That's what's happening.
[100:06.72] So back.
[100:07.72] And then you were, you're part of Bill Club.
[100:08.72] Yeah.
[100:09.72] I'm part of Bill Club in Sydney.
[100:10.72] What is Bill Club?
[100:11.72] Bill Club is a.
[100:12.72] The impressions of the thing that I did.
[100:15.72] Unfortunately I was at the airport when you were giving a presentation and Annie has not
[100:20.00] sent me the recording yet.
[100:21.24] So I've not seen it.
[100:22.96] It's on YouTube.
[100:23.96] Oh, okay.
[100:24.96] Great.
[100:25.96] I uploaded it on my channel.
[100:26.96] Oh, awesome.
[100:27.96] Okay.
[100:28.96] I'll take a look.
[100:29.96] But Bill Club is the one and only AI centric community in pretty much Sydney and I had
[100:41.28] to spend months to push Annie to do that thing and eventually she did and I'm so glad she
[100:47.20] did and it's growing and she's doing amazing.
[100:50.20] She's expanding to many cities.
[100:53.28] It's ANZ now.
[100:54.28] Yeah.
[100:55.28] It's amazing.
[100:56.28] And she has our couch from our apartment when we moved away, we couldn't find a way to
[101:00.92] sell it.
[101:01.92] We're like, hey, Annie, you're getting a space.
[101:02.92] Do you guys need a couch?
[101:03.92] She's like, sure.
[101:04.92] So she has my couch.
[101:06.56] Nice.
[101:07.56] And then what do you listen for in the in-space, what are you interested in?
[101:13.64] I like to get a sense of what's going on.
[101:17.52] You guys ask very good questions.
[101:20.40] For some reason, you guys seem so well-researched, both you and Alessio, somehow you're just,
[101:26.28] you ask very good questions that me as a person, like general product developer, product engineer,
[101:34.60] I have no idea about ML, I don't follow the papers.
[101:38.28] I know about the paper club.
[101:41.68] I don't follow it because it's over my head, but you guys distill it so well and you guys
[101:47.04] ask the questions to your guests that I have in the back of my mind or that I would don't
[101:53.48] even know that I have the questions of it and you guys guide the conversations in a way
[101:57.60] that I can learn from and I wouldn't even know anything to ask.
[102:01.48] So I'm so glad you guys are doing it.
[102:03.40] It's so helpful, and keep doing what you're doing.
[102:07.48] And I really love what you guys did with the best papers from the talk.
[102:13.52] Good.
[102:14.52] Everyone has shouted it out today.
[102:15.52] Yes.
[102:16.52] Yeah, it's really good.
[102:17.52] I mean, a lot of that was way over my head, but I listen to it all and try to just get
[102:22.96] the sense.
[102:23.96] I just try to keep listening to this stuff until I get it.
[102:27.84] When you guys expose, I would never go to a conference like that, but I was just not
[102:36.00] understanding anything.
[102:37.00] But you guys make it so accessible, and I love it.
[102:40.04] Yeah, so maybe the Pocket Studio is right here, actually, I can show you after we're
[102:43.48] done recording.
[102:44.48] It's not that fancy, it's just a studio.
[102:47.00] And yeah, for me, the goal with the New Europe's recap was not that you would read everything
[102:52.96] or anything.
[102:53.96] Yeah, we would just pick what we thought was most important for you.
[102:56.96] And if any one of them interested you, you could double click on it.
[103:00.30] That's it.
[103:01.30] We're not going to be the experts on every single thing.
[103:04.80] It's impossible.
[103:05.80] And already, the episode that I cut together for that was like three and a half hours.
[103:09.34] So people were complaining about that.
[103:11.44] But then the last thing, the reason that-- Les and I don't do that much research for
[103:17.56] each episode, but we researched a guess.
[103:21.72] But just being involved in the day-to-day conversations and our day jobs prepares you
[103:26.20] for that.
[103:27.20] And I think that is important.
[103:28.22] So no prep needed, because we're in it, we're in the arena, as they say.
[103:33.70] Yeah, anything else?
[103:35.20] Like, there's so much excitement, there's so many things to cover, and what you guys
[103:43.08] are-- maybe culturally, yeah, that would be a thing I was always wondering.
[103:48.98] And that might be not partly in the space, but what are you guys doing?
[103:56.26] To cover the cultural aspect of happening here is probably a separate thing, but equally
[104:02.10] important thing, to document all the conversations that are happening around here and all the
[104:08.58] other build spaces.
[104:10.46] We see glimpses of that on Twitter, but I think capturing more of that would be super
[104:14.92] cool.
[104:15.92] So I feel like that's something that someone else should do.
[104:21.10] We try to be more technical, because people can use it at work, they can justify that
[104:25.74] for productivity.
[104:26.96] We might try to dabble in some of that, so I'm pretty connected with the main areas for
[104:32.50] those listening-- the main areas for those listening who are interested in SFAI is like
[104:37.26] Shack 15, AGI House SF, AGI House Hillsboro, and then us, and maybe HF0, and then maybe
[104:46.60] a little bit of Founders Inc.
[104:49.32] Those are it.
[104:50.32] There's more community oriented spaces like the Commons, but they're not AI-centric.
[104:58.04] So we can do a little bit of reporting around that, but it's going to be this American life,
[105:01.92] like tell me your life story, like solve story.
[105:04.20] I'm not the best at that, and then also there's a lot of very, very brutal cutting for that.
[105:09.02] That is hard to do, but we can dabble, or we can do it on the side.
[105:15.34] The other thing I'm very interested in-- I'm a UX designer by trade, and any time you guys
[105:20.38] touch on AI and UX and Jett or UI, I'm all ears, and I would love to-- again, it's probably
[105:26.66] not the technical side of the space, but I think there needs to be 100 times more resources
[105:31.94] out there than what's currently available.
[105:35.88] I think we held the first AI/UX meetup ever in SF in the world.
[105:40.36] That was really fun.
[105:41.36] The meetup's on YouTube, if you want to see it, and it's in the latest-based archives,
[105:45.12] the newsletter.
[105:46.12] I don't think we ever published a podcast version of it, so you have to just subscribe
[105:49.36] to the newsletter and then check the YouTube for that stuff.
[105:52.16] But AI/UX is a topic of ours that we like to cover.
[105:54.96] It's just very hard to cover as an audio medium, because you can't see it.
[105:59.68] And also, I think it's going to be mostly owned by Notion and Vercel and Retool, which
[106:07.84] we've interviewed, Retool.
[106:09.34] We're going to interview Vercel when we've interviewed Notion.
[106:12.58] So who else?
[106:13.58] Who do you want to listen to on AI/UX, right?
[106:17.30] There's individual people.
[106:18.30] We had Amelia Wattenberger present at AI Engineer Summit.
[106:21.62] You can see that on YouTube.
[106:23.70] I know a lot of the thinkers on AI/UX, and I think I know what they say.
[106:28.54] I haven't seen anything super innovative.
[106:31.60] Everyone hates chatbots.
[106:34.02] Everyone wants to innovate things.
[106:36.10] I haven't seen any new ideas since we did the AI/UX meetup one year ago.
[106:41.40] Tell me I'm wrong.
[106:42.40] Well, that sounds really disappointing.
[106:47.08] I haven't seen anything on Twitter that I thought that would be easier to push, because
[106:53.64] you just rap LLMs.
[106:59.66] But there needs to be more people from the design space, from the product space, like
[107:05.38] UX researchers coming in and figuring out how can we take LLMs and apply them to real
[107:11.14] problems.
[107:12.14] I haven't seen a whole lot of that.
[107:13.38] In Sydney, there's not a whole lot of that.
[107:15.66] I'm hoping to maybe be a part of the community here and try to grow that side of the things.
[107:22.46] Well, look, you're here now.
[107:24.48] You're interested in AI/UX.
[107:26.98] Run the next AI/UX meetup.
[107:28.00] I can set you up with the venue, the people.
[107:31.62] You need to find the speakers.
[107:32.86] I'm not going to find the speakers for you, but if you want to set that up, go for it.
[107:38.06] So I actually copied your AI/UX format, and I hold a talk in Sydney, and in a very light
[107:46.72] fashion, like 20, 30 people showed up.
[107:49.84] We had some cool demos.
[107:51.24] It was like a small version of your AI/UX conference, but yeah, I'd love to participate.
[107:59.32] I mean, this is SF, 300 people show up.
[108:01.30] I just got to get some cool demos.
[108:03.10] I can see you with some people.
[108:04.42] Let's make it happen.
[108:05.42] Let's make it happen.
[108:06.42] Let's make it happen.
[108:07.42] All right.
[108:08.42] Well, it's nice to meet you, and I'll get your details.
[108:09.58] That's all, folks.
[108:11.06] If you've enjoyed or benefited from our work on latent space over this past year, we'd
[108:15.78] really love to hear from you and really appreciate it if you tell a friend.
[108:20.78] The only way a podcast consistently grows is through your word of mouth, and that helps
[108:26.08] us book incredible guests and attend great events in our second year.
[108:30.20] Have a lovely weekend.
[108:31.40] [MUSIC]
[108:41.40] [MUSIC]
