[by:whisper.cpp]
[00:00.00]Hey everyone, welcome to the Late In Space podcast.
[00:09.30]This is Alessio, partner in CTO & Residence at Decibel Partners.
[00:13.00]And I'm joined by Makoho's Swicks, founder of SmallAI.
[00:16.00]Hello, we're back in the remote studio with Jason Liu from Instructor.
[00:20.00]Welcome, Jason.
[00:21.00]Hey there, thanks for having me.
[00:23.00]Jason, you are extremely famous.
[00:25.00]So I don't know what I'm going to do introducing you.
[00:28.00]You're one of the Waterloo clan.
[00:30.00]There's like a small cadre of you that's just completely dominating machine learning.
[00:34.00]Actually, can you list like Waterloo alums that you're like, you know are just dominating and questioning it right now.
[00:39.00]So like, John from like RaiSana is doing his inversion models, right?
[00:45.00]I know like Clive Chen from Waterloo.
[00:48.00]When I started the data science club, he was one of the guys we're like joining in and just like hanging out in the room.
[00:52.00]And now he was at Tesla working with Carpathian, now he's at OpenAI.
[00:56.00]Yeah, he's in my climbing club.
[00:58.00]Oh hell yeah.
[00:59.00]Yeah, I haven't seen him in like six years now.
[01:01.00]To get in the social scene in San Francisco, you have to climb.
[01:04.00]So both in career and in rocks.
[01:07.00]So you started the data science club in Waterloo.
[01:09.00]We can talk about that.
[01:10.00]But then also spent five years at Stitchfix as an MLE.
[01:13.00]You pioneered the use of OpenAI's LLMs to increase stylish efficiency.
[01:17.00]So you must have been like a very, very early user.
[01:19.00]This was like pretty early on.
[01:21.00]Yeah, I mean this was like GPT-3.
[01:24.00]Okay, so we actually were using transformers at Stitchfix before the GPT-3 model.
[01:29.00]So we were just using transformers recommendation systems.
[01:31.00]At that time, I was very skeptical of transformers.
[01:34.00]I was like, why do we need all this infrastructure?
[01:36.00]We can just use like matrix factorization.
[01:38.00]When GPT-2 came out, I fine-tuned my own GPT-2 to write like rap lyrics.
[01:42.00]And I was like, okay, this is cute.
[01:43.00]Okay, I got to go back to my real job, right?
[01:45.00]Like, who cares if I can write a rap lyric?
[01:47.00]When GPT-3 came out, again, I was very much like,
[01:50.00]why are we using like a post request to review every comment a person leaves?
[01:54.00]Like, we can just use classical models.
[01:56.00]So I was very against language models for like the longest time.
[01:59.00]And then when chat GPT came out,
[02:01.00]I basically just wrote a long apology letter to everyone at the company.
[02:04.00]I was like, hey guys, you know, I was very dismissive of some of the technology.
[02:07.00]I didn't think you would scale well and I am wrong.
[02:10.00]This is incredible.
[02:11.00]And I immediately just transitioned to go from computer vision,
[02:14.00]recommendation systems to LLMs.
[02:16.00]But funny enough, now that we have RAG,
[02:18.00]we're kind of going back to recommendation systems.
[02:21.00]Speaking of that, I think Alessio is going to bring up the next one.
[02:23.00]Yeah, I was going to say we had Brian Bishop from X on the podcast to overlap Stitch Fix.
[02:28.00]Yeah, he was like one of my main users of the recommendation framework
[02:31.00]that I had built out at Stitch Fix.
[02:33.00]Yeah, we talked a lot about REXIS.
[02:35.00]So it makes sense.
[02:36.00]So now I have adopted that line, RAG is REXIS.
[02:39.00]And you know, if you're trying to reinvent new concepts,
[02:42.00]you should study REXIS first,
[02:43.00]because you're going to independently reinvent a lot of concepts.
[02:45.00]So your system was called flight.
[02:47.00]It's a recommendation framework with over 80% adoption
[02:50.00]servicing 350 million requests every day.
[02:52.00]Wasn't there something existing at Stitch Fix?
[02:54.00]Why did you have to write one from scratch?
[02:56.00]No, so I think because at Stitch Fix,
[02:59.00]a lot of the machine learning engineers and data scientists
[03:01.00] were writing production code.
[03:03.00]So every team's systems were very bespoke.
[03:06.00]It's like this team only needs to do like real-time recommendations
[03:09.00]with small data.
[03:10.00]So they just have like a fast API app with some like pandas code.
[03:13.00]This other team has to do a lot more data.
[03:15.00]So they have some kind of like spark job that does some batch ETL
[03:18.00]that does a recommendation,right?
[03:20.00]And so what happens is each team writes their code differently.
[03:23.00]And I have to come in and like refactor their code.
[03:25.00]And I was like, oh man,I'm refactoring four different code bases
[03:28.00]four different times.
[03:29.00]Wouldn't it be better if all the code quality was my fault?
[03:32.00]Alright,let me just write this framework for everyone else to use it.
[03:35.00]And now one person can maintain five different systems
[03:38.00]rather than five teams having their own bespoke system.
[03:41.00]And so it was really a need of just sort of standardizing everything.
[03:44.00]And then once you do that,you can do observability
[03:47.00]across the entire pipeline and make large sweeping improvements
[03:50.00]in this infrastructure,right?
[03:52.00]If we notice that something is slow,we can detect it on the operator layer.
[03:56.00]Just hey,hey,like this team,you guys are doing this operation
[03:59.00]it's lowering our latency by like 30%.
[04:01.00]If you just optimize your python code here,we can probably
[04:05.00]make an extra million dollars.So let's jump on a call
[04:07.00]and forget this out.And then a lot of it was doing
[04:09.00]all this observability work to figure out what the heck is going on
[04:12.00]and how to optimize this system from not only just a code perspective.
[04:15.00]So like harassingly Oregon saying like we need to add cash in here.
[04:18.00]We're doing duplicated work here.Let's go clean up the systems.
[04:21.00]Yeah,got it.One more system that I'm interested in finding out more about
[04:25.00]is your similarity search system using Clip and GPT-3
[04:29.00]embedding in FICE,where you said over $50 million in annual revenue.
[04:33.00]So of course they all gave all that to you,right?
[04:35.00]No,no,no.I mean,it's not going up and down,but you know,
[04:38.00]I got a little bit,so I'm pretty happy about that.
[04:40.00]But there,you know,that was when we were doing fine tuning like
[04:44.00]resnets to do image classification.
[04:46.00]And so a lot of it was given an image if we could predict
[04:50.00]the different attributes we have in the merchandising
[04:52.00]and we can predict the index embeddings of the comments
[04:55.00]then we can kind of build a image vector or image embedding
[04:59.00]that can capture both descriptions of the clothing and sales of the clothing.
[05:03.00]And then we would use these additional vectors
[05:05.00]to augment our recommendation system.
[05:07.00]And so with the recommendation system really was just around
[05:10.00]like what are similar items,what are complementary items,
[05:12.00]what are items that you would wear and a single outfit
[05:15.00]and being able to say on a product page,let me show you
[05:18.00]like 15,20 more things.
[05:20.00]And then what we found was like,hey,when you turn that on
[05:22.00]you make a bunch of money.
[05:23.00]Yeah,so okay,so you didn't actually use GPT-3embeddings
[05:26.00]you fine tuned your own,because I was surprised
[05:28.00]that GPT-3 worked off the shelf.
[05:30.00]Okay,because I mean,at this point we would have
[05:32.00]3 million pieces of inventory over like a billion interactions
[05:35.00]and users and clothes
[05:37.00]any kind of fine-taining would definitely up the form
[05:39.00]like some off the shelf model.
[05:41.00]Cool,I'm about to move on from Stitch Fix
[05:43.00]but,you know,any other like fun stories from the Stitch Fix
[05:45.00]that you want to cover?
[05:46.00]No,I think that's basically it.
[05:48.00]I mean,the biggest one really was the fact that
[05:50.00]I think for just four years I was so bearish on language models
[05:53.00]and just NLP in general,I was just like,none of this really works.
[05:55.00]Like,why would I spend time focusing on this?
[05:57.00]I gotta go do the things that makes money.
[05:59.00]Recommendations,bounding boxes,image customization.
[06:02.00]Yeah,now I'm like prompting an image model.
[06:04.00]Oh,man,I was wrong.
[06:05.00]So,my Stitch Fix question would be,you know,
[06:08.00]I think you have a bit of a drip and I don't.
[06:10.00]You know,my primary wardrobe is free start-up
[06:12.00]conference t-shirts.
[06:14.00]Should more technology brothers be using Stitch Fix?
[06:17.00]What's your fashion advice?
[06:20.00]Oh,man,I mean,I'm not a user of Stitch Fix,right?
[06:23.00]It's like,I enjoy going out and like touching things
[06:27.00]and putting things on and trying them on,right?
[06:29.00]I think Stitch Fix is a place where you kind of go
[06:31.00]because you want the work offloaded.
[06:33.00]I really love the clothing I buy
[06:35.00]where I have to like,when I land in Japan
[06:37.00]I'm doing like a 45-minute walk
[06:39.00]up a giant hill to find this weird denim shop.
[06:42.00]That's the stuff that really excites me.
[06:44.00]But,I think the bigger thing that really captures
[06:46.00]is this idea that narrative matters a lot
[06:48.00]to human beings,okay?
[06:50.00]And I think the recommendation system,that's really hard to capture.
[06:53.00]It's easy to use AI to sell like a $20 shirt,
[06:56.00]but it's really hard for AI to sell like a $500 shirt.
[06:59.00]But people are buying $500 shirts,you know what I mean?
[07:01.00]Like,there's definitely something that we can't really capture
[07:04.00]just yet that we probably will figure out how to
[07:07.00]in the future.
[07:08.00]Well,he'll probably output in JSON,which is
[07:10.00]what you're going to turn to next.
[07:12.00]Then you went on a sabbatical to South Park Commons
[07:14.00]in New York,which is unusual
[07:16.00]because it's based on NSF.
[07:18.00]Yeah,so,basically in 2020,really,
[07:20.00]I was enjoying working a lot
[07:22.00]and so I was like building a lot of stuff.
[07:24.00]This is where we were making like the tens of millions of dollars
[07:26.00]doing stuffand then I had a hand injury
[07:28.00]and so I really couldn't code anymore
[07:30.00]for a year or two years.
[07:32.00]And so I kind of took sort of half of it as medical leave.
[07:34.00]The other half I became more of like a tech lead
[07:36.00]just like making for the systems or like lights were on.
[07:39.00]And then when I went to New York,
[07:41.00]I spent some time there and kind of just like wound down
[07:44.00]the tech work,you know,did some pottery,did some jiu-jitsu
[07:47.00]and after GBD came out,I was like,
[07:49.00]Oh,I clearly need to figure out what is going on here
[07:52.00]because something feels very magical.
[07:54.00]I don't understand it.
[07:56.00]So I spent basically like five months just prompting
[07:58.00]and playing around with stuff.
[07:59.00]And then afterwards it was just my starter friends
[08:01.00]going like,Hey Jason,you know,
[08:03.00]my investors want us to have an AI strategy.
[08:05.00]Can you help us out?
[08:06.00]And it's just snowballed and born more
[08:08.00]and become until I was making this my full-time job.
[08:10.00]Yeah,got it.
[08:11.00]You know,you had YouTube University
[08:13.00]and a journaling app,you know,a bunch of other explorations,
[08:16.00]but it seems like the most productive
[08:18.00]or the best-known thing that came out of your time
[08:20.00]there was Instructor.
[08:21.00]Yeah,written on the bullet train in Japan.
[08:23.00]Tell us the origin story.
[08:24.00]Yeah,I mean,I think at some point,
[08:27.00]you know,tools like guardrails and Marvin came out,
[08:29.00]right?
[08:30.00]Those are kind of tools that like use XML
[08:32.00]and Python to get structure data out,
[08:33.00]but they really were doing things
[08:35.00]sort of in the prompt and these were built
[08:37.00]with sort of the instruct models in mind.
[08:39.00]Like,I'd already done that in the past,right?
[08:41.00]Stitchfix,you know,one of the things we did was
[08:43.00]we wouldn't take every crest note
[08:45.00]and turn that into a JSON object
[08:47.00]that we would use to send to our search engine,right?
[08:49.00]So if you said like,I want to,you know,
[08:51.00]skinny jeans that were this size,
[08:53.00]that would turn into JSON that we would send
[08:55.00]to our internal search APIs.
[08:56.00]But it always felt kind of gross.
[08:58.00]A lot of it is just like you read the JSON,
[09:00.00]you like parse it,
[09:01.00]you make sure the names are strings
[09:02.00]and ages are numbers
[09:03.00]and you do all this messy stuff.
[09:04.00]But when function calling came out,
[09:06.00]it was very much sort of a new way of doing things,right?
[09:09.00]Function calling lets you define the schema
[09:11.00]separate from the data and the instructions.
[09:13.00]And what this meant was
[09:15.00]you can kind of have a lot more complex schemas
[09:17.00]and just map them in pidantic
[09:19.00]and then you can just keep those very separate.
[09:21.00]And then once you add like methods,
[09:22.00]you can add validators and all that kind of stuff.
[09:24.00]The one thing I really had with a lot of these libraries,
[09:26.00]though,was it was doing a lot of the string formatting themselves,
[09:29.00]which was fine when it was the instruction tune models,
[09:32.00]you just have a string.
[09:33.00]But when you have these new chat models,
[09:35.00]you have these chat messages,
[09:37.00]and I just didn't really feel like
[09:38.00]not being able to access that for the developer
[09:40.00]was sort of a good benefit that they would get.
[09:43.00]And so I just said,let me write like the most
[09:45.00]simple SDK around the OpenAI SDK,
[09:48.00]so simple wrapper on the SDK,
[09:50.00]just handle the response model a bit
[09:52.00]and kind of think of myself more like requests
[09:55.00]than an actual framework that people can use.
[09:57.00]And so the girl's like,
[09:58.00]Hey,like this is something that you can use
[09:59.00]to build your own framework.
[10:00.00]But let me just do all the boring stuff
[10:02.00]that nobody really wants to do.
[10:03.00]People want to build their own frameworks,
[10:05.00]but people don't want to build like JSON parsing.
[10:08.00]And the retrying and all that other stuff.
[10:10.00]Yeah,right.
[10:11.00]We had this little build this discussion before the show,
[10:13.00]but like that design principle
[10:14.00]of going forward being requests
[10:16.00]rather than being Django.
[10:17.00]Yeah.
[10:18.00]What inspires you there?
[10:19.00]This has come from a lot of prior pain.
[10:21.00]Are there other open source projects
[10:23.00]that inspired your philosophy here?
[10:25.00]Yeah,I mean,I think it would be requests,right?
[10:27.00]Like I think it is just the obvious thing you install.
[10:30.00]If you were going to go
[10:31.00]make like HTTP requests in Python,
[10:33.00]you would obviously import requests.
[10:35.00]Maybe if you want to do more async work,
[10:37.00]there's like future tools,
[10:38.00]but you don't really even think about installing it.
[10:40.00]And when you do install it,
[10:41.00]you don't think of it as like,
[10:42.00]this is a requests app,right?
[10:44.00]Like,no,this is just Python.
[10:46.00]Like the bigger question is like,
[10:48.00]a lot of people ask questions like,
[10:49.00]oh,why isn't requests like in the standard library?
[10:52.00]That's how I want my library to feel,right?
[10:54.00]It's like,oh,if you're going to use the LLM SDKs,
[10:57.00]you're obviously going to install instructor.
[10:59.00]And then I think the second question would be like,
[11:01.00]oh,how come instructor doesn't just go into
[11:03.00]open AI,go into Anthropic?
[11:05.00]Like,if that's the conversation we're having,
[11:06.00]like that's where I feel like I've succeeded.
[11:08.00]Yeah,it's like,yeah,yeah.
[11:09.00]So standard,you may as well just have in the base libraries.
[11:12.00]And the shape of the request
[11:14.00]stayed the same,but initially
[11:16.00]function calling was maybe
[11:17.00]equal structure outputs for a lot of people.
[11:19.00]I think now the models also
[11:21.00]support like JSON mode and
[11:23.00]some of these things and,you know,
[11:25.00]return JSON of my grandma is going to die.
[11:27.00]All of that stuff is maybe to the side.
[11:29.00]How have you seen that evolution?
[11:30.00]Like,maybe what's the meta game today?
[11:32.00]Like,should people just forget about
[11:33.00]function calling for structure outputs
[11:35.00]or where,when is structure output,
[11:37.00]like JSON mode the best versus not.
[11:39.00]We'd love to get any thoughts given that you do this every day.
[11:42.00]Yeah,I would almost say these are like
[11:44.00]different implementations of like
[11:46.00]the real thing we care about is the fact that now we have
[11:48.00]type responses to language models.
[11:50.00]And because we have the type response,
[11:52.00]my ID is a little bit happier.
[11:53.00]I get autocomplete.
[11:54.00]If I'm using the response wrong,
[11:56.00]there's a little red squiggly line.
[11:57.00]Like,those are the things I care about.
[11:59.00]In terms of whether or not like
[12:00.00]JSON mode is better,
[12:01.00]I usually think it's almost worse
[12:03.00]unless you want to spend less money
[12:05.00]on like the prompt tokens
[12:07.00]that the function call represents.
[12:08.00]Primarily because with JSON mode,
[12:10.00]you don't actually specify the schema.
[12:11.00]So sure,like JSON load works,
[12:13.00]but really I care a lot more than just
[12:15.00]specify that it is JSON,right?
[12:17.00]I think function calling gives you a tool to
[12:19.00]specify the fact like,ok,this is a list
[12:21.00]of objects that I want and each object
[12:23.00]has a name or an age and I want the age
[12:25.00]to be above zero and I want to make sure
[12:27.00]it's parsed correctly.That's where
[12:28.00]kind of function calling really shines.
[12:30.00]I need thoughts on single versus
[12:32.00]parallel function calling.
[12:34.00]So I did a presentation at our
[12:36.00]AI inaction discord channel
[12:38.00]and obviously Shogays instructor.
[12:41.00]One of the big things that we have before
[12:43.00]is single function calling.It's like
[12:44.00]when you're trying to extract lists,
[12:46.00]you have to make these funky like properties
[12:48.00]that are list to then actually return
[12:50.00]all the objects.How do you see
[12:52.00]the hack being put on the developer's
[12:54.00]plate versus like more of this stuff
[12:56.00]just getting better in the model.
[12:58.00]And I know you tweeted recently about
[13:00.00]entropic,for example,you know,some
[13:02.00]less they're not list or strings and
[13:03.00]there's like all of these discrepancies.
[13:05.00]I almost would prefer that there was
[13:07.00]always a single function call,but
[13:09.00]obviously there is like the agent's
[13:10.00]workflows that,you know,instructor
[13:12.00] can really support that well,but
[13:14.00]are things that,you know,ot to be done.
[13:16.00]Like you could define,I think maybe
[13:18.00]like 50 or 60 different functions
[13:20.00]in a single API call.
[13:22.00]And,you know,if it was like get the
[13:24.00]weather or turn the lights on or do
[13:26.00]something else,it makes a lot of sense
[13:28.00]to have theseparallel function calls,but
[13:30.00]in terms of an extraction workflow,I
[13:32.00]definitely think it's probably more
[13:34.00]helpful to have everything be a
[13:36.00]single schema.Just because you can
[13:38.00]specify relationships between these
[13:40.00]single chain of thought before you
[13:42.00]generate a list of results,like
[13:44.00]there's like small,like,API
[13:46.00]differences,right,whereif it's
[13:48.00]parallel function calling,if you do one,like
[13:50.00]again,really,I really care about how
[13:52.00]the SDK looks,and says,okay,do I
[13:54.00]always return a list of functions,or do
[13:56.00]you just want to have the actual
[13:58.00]object back out,and you want to have
[14:00.00]autocomplete over that object.Interesting.What's
[14:02.00]kind of the cap for,like,how many
[14:04.00]function definitions you can put in
[14:06.00]where it still works well?Do you have
[14:08.00]anything that doesn't really need to do
[14:10.00]anything that's more than six or seven
[14:12.00]different functions?I think in the
[14:14.00]documentation,they support way more.I
[14:16.00]don't even know if there's any good
[14:18.00]evalves that have,you know,over like two
[14:20.00]dozen function calls.I think if you're
[14:22.00]riding into issues where you have,like,20
[14:24.00]or 50 or 60function calls,I think
[14:26.00]you're much better having those
[14:28.00]specifications saved in a vector
[14:30.00]database,and then have them be retrieved,right.So
[14:32.00]if there are 30tools,like,you should
[14:34.00]basically be,like,ranking them,and then
[14:36.00]other than just,like,shubbing,like,60
[14:38.00]function into a single.Yeah.
[14:40.00]Well,I mean,so,I think this is
[14:42.00]relevant now,because previously,I
[14:44.00]think context limits prevented you
[14:46.00]from having more than a dozen tools
[14:48.00]anyway.And now that we
[14:50.00]have million-token context windows,you
[14:52.00]know,a cloud recently with their new
[14:54.00]function calling release said they can
[14:56.00]handle over 250tools,which is
[14:58.00]insane to me.That's a lot.You're
[15:00.00]saying,like,you know,you don't think
[15:02.00]there's many people doing that.I think
[15:04.00]a sort of agent-like platform where you
[15:06.00]have a bunch of connectors,they wouldn't
[15:08.00]run into that problem.Probably,you're
[15:10.00]right,that they should use a vector
[15:12.00]database and kind of rag their tools.I
[15:14.00]know Zapier has,like,a few thousand,like,8,000,9,000
[15:16.00]connectors that,you know,obviously don't fit
[15:18.00]anywhere.So,yeah,I mean,that,I
[15:20.00]think that would be it,unless you need
[15:22.00]some kind of intelligence thatchains
[15:24.00]things together,which is,I think
[15:26.00]what Alessio is coming back to,right.Like,there's
[15:28.00]this trend aboutparallelfunction
[15:30.00]calling.I don't know what I think about
[15:32.00]multiple tools insequence,but they're not
[15:34.00]inparallel.I haven't explored this atall.I'm
[15:36.00]just,like,throwing this open to you.So,like,what
[15:38.00]doyou think aboutall these newthings.Yeah,it's
[15:40.00]like,you know,do we assume thatall
[15:42.00]functioncalls could happen in anyorder?In
[15:44.00]which case,like,we eithercannassume that
[15:46.00]or wecannassume that,like,things need to
[15:48.00]happen in some kind ofsequence as a dag,right.But if
[15:50.00]it's a dag,really,that's just,like,one json
[15:52.00]object that is the entire dag,ratherthan
[15:54.00]going,like,okay,theorder of thefunction
[15:56.00]thatreturn don't matter.That's definitely
[15:58.00]just not true inpractice,right.Like,ifI have,I
[16:00.00] can do something that's,like,turn the lights on,like,unplug
[16:02.00] the power,then,like,turn the toaster on,or
[16:04.00] something,like,theorder doesn't matter.And
[16:06.00]it's unclear how well you can describe
[16:08.00]the importance of that reasoning to a
[16:10.00]language model yet.I mean,I'm sure
[16:12.00]you can do it with,like,good enough prompting.But
[16:14.00]I just haven't any use case with a function
[16:16.00]sequence,really matters.Yeah,to me,the most
[16:18.00]interesting thing is,the models are better
[16:21.00]atpicking than your ranking is usually.Like,I
[16:24.00] mean,competing a company around system
[16:26.00]integration.And,for example,with one system,there
[16:29.00] are,like,700,80 endpoints.And
[16:31.00]ifyou actually try and do vector
[16:33.00]similarity,it's not that good,because
[16:35.00]the people that wrote the specs didn't
[16:37.00]have a mind,making them,like,semantically
[16:39.00] apart.You know,they're kind of like,oh,create
[16:41.00] this,create this,create this,versus
[16:43.00]when you give it to a model,like,in Opus
[16:45.00]you put them all,it's quite good at picking
[16:47.00]which ones you should actually run.And
[16:49.00]I'm curious to see if the model providers
[16:51.00]actually care about some of those
[16:53.00]worthless,or if the agent company is
[16:55.00]actually gonna build very good rankers to
[16:57.00]kind of fill that gap.Yeah,my money is on
[16:59.00]the rankers,becauseyou can do those so
[17:01.00]easily,right?You could just say,well,given
[17:03.00]the embeddings ofmy search query and
[17:05.00]the embeddings ofthe description,I
[17:07.00]can just train XGBoost and just make
[17:09.00]sure that I have very high,like,MRR,which
[17:11.00]is,like,mean reciprocal rank.And
[17:13.00]so,the only objective is to makesure
[17:15.00]that the tools you use are in the top
[17:17.00]and filtered.Like,that feels super
[17:19.00]straightforward,and you don't have to
[17:21.00]actually figure out how to fine tune a
[17:23.00]language model to do tool selection anymore.Yeah,I
[17:25.00]amagine you either have,like,less than 3
[17:27.00]tools or more than a thousand.I
[17:29.00]don't know what kind of companies
[17:31.00]that,oh,thank god we only have,like,185
[17:33.00]tools.And this works perfectly,right?
[17:35.00]That's right.And before we maybe move on
[17:37.00]just from this,it was interesting to me
[17:39.00]you retweeted this thing about entropic
[17:41.00]function calling,and it was Joshua
[17:43.00]Brown's retweeting some benchmark that
[17:45.00]is,like,oh,my god,entropic function
[17:47.00]calling so good.And then you retweeted
[17:49.00]and then you tweeted it later,and it's
[17:51.00]like,it's so good.And
[17:53.00]what's your flow?How do you
[17:55.00]actually test these things?Because
[17:57.00]obviously the benchmarks are lying,right?Because
[17:59.00]the benchmarks say it's good and you said
[18:01.00]it's bad and I trust you more than the
[18:03.00]benchmark.How do you think about
[18:05.00]that?And then how do you evolve it
[18:07.00]over time?It's mostly just client data.I
[18:09.00]actually have been mostly busy with
[18:11.00]enough client work that I haven't been
[18:13.00]able to reproduce public benchmarks.And
[18:15.00]so I can't even share some of the results
[18:17.00]entropic.But I would just say,like,in
[18:19.00]production,we have some pretty
[18:21.00]interesting schemas,where it's like
[18:23.00]initatively building lists where
[18:25.00]we're doing,like,updates of lists.Like,we're
[18:27.00]doing in-place updates.So,like,upserts
[18:29.00]and inserts.And in those
[18:31.00]situations,we're like,oh,yeah,we have a bunch
[18:33.00]of different parsing errors.Numbers are being
[18:35.00]returned to strings.We were expecting lists
[18:37.00]of objects,but we're getting strings that
[18:39.00]are,like,the strings of JSON,right?So we
[18:41.00]had to call JSON parse on individual
[18:43.00]elements.Overall,I'm,like,super happy
[18:45.00]with theentropic models compared
[18:47.00]to the openAM models.Sonnet is very
[18:49.00]cost-effective.Hikoo is,in function
[18:51.00]calling,it's actually better.But I think they just
[18:53.00]had to sort of file down the edges a little
[18:55.00] bit,where,like,our tests pass,but
[18:57.00]then we actually deployed a production,we get,you
[18:59.00] know,half a percent of traffic
[19:01.00]having issues,where,if you ask for JSON,it'll
[19:03.00] try to talk to you,or if you use function
[19:05.00]calling,you know,we'll have,like,a parse error.And
[19:07.00]so I think that definitely going to be things
[19:09.00]that are fixed in,like,the upcoming weeks.But
[19:11.00] in terms of,like,the reasoning capabilities,I
[19:13.00] mean,it's hard to beat,like,70%
[19:15.00]cost reduction,especially when you're building
[19:17.00] consumer applications,right?If you're building
[19:19.00] something for,like,consultants or private equity,like,you're charging
[19:21.00] $400,it doesn't really matter if it's $1
[19:23.00] or $2.But for consumer apps,it
[19:25.00] makes products viable.If you can go
[19:27.00] from 4 to Sonnet,you might actually be able
[19:29.00] to price it better.Yeah.
[19:31.00]I had this chart about the ELO
[19:33.00] versus the cost of all the
[19:35.00] models.And,you could
[19:37.00] put trend graphs on each of those things
[19:39.00] about,like,you know,higher ELO equals
[19:41.00] higher cost,except for Hikoo.Hikoo kind of just broke
[19:43.00] the lines,or the ISO ELOs,if you want
[19:45.00] to kind of call it.Cool.Before we
[19:47.00] go too far into,you know,your opinions on
[19:49.00] just the overall ecosystem,I want to
[19:51.00] make sure that we map out the surface area
[19:53.00] of Instructure.I would say that
[19:55.00] most people would be familiar with
[19:57.00] Instructure from your talks and your tweets
[19:59.00] and all that.You had the number one
[20:01.00] talk at from the AI Engineering Summit
[20:03.00]Jason Liu and Jerry Liu.Yeah,yeah,yeah,yeah.You have to
[20:07.00] start with Jay and Italy to do well.But
[20:09.00] yeah,until I actually went through your
[20:11.00] cookbook,I didn't realize,like,the surface area.Like,how
[20:13.00] would you categorize the use cases,right?You have
[20:15.00]LLM self-critique,you have knowledge
[20:17.00] graphs in here,you have PII data
[20:19.00] sanitation.How do you characterize to people
[20:21.00] like,what is the surface area of Instructure?Yeah,so
[20:23.00] I mean,this is the part that feels crazy
[20:25.00] because really,the difference is
[20:27.00]LLMs give you strings and Instructure gives
[20:29.00] you data structures.And once you get data structures
[20:31.00] again,you can do every,like,lead code problem
[20:33.00] you ever thought of,right?And so,I think
[20:35.00] there's a couple of really common applications.The
[20:37.00] first one obviously is extracting
[20:39.00]Structure data.This is just be,okay,well,like,I want to
[20:42.00] put in an image of a receipt,I want to
[20:44.00] give it back out a list of checkout items
[20:46.00] with a price and a fee and a coupon code
[20:48.00] or whatever.That's one application.Another
[20:50.00] application really is around
[20:52.00] extracting graphs out.So,one of the things
[20:54.00] we found out about these language models
[20:56.00] is that not only can you define nodes,it's
[20:58.00] really good at figuring out what are nodes
[21:00.00] and what are edges.And so,we have a bunch
[21:02.00] of examples where,you know,not only do I
[21:04.00] extract that,you know,this happens
[21:06.00] after that,but also,like,okay,these two
[21:08.00] are dependencies of another task.And you
[21:11.00] can do,you know,extracting complex
[21:13.00] entities that have relationships.Given
[21:15.00] a story,for example,you can extract
[21:17.00] relationships of families across different
[21:19.00] characters.This is going to be done by
[21:21.00] defining a graph.The last really big
[21:23.00] application really is just around query
[21:25.00] understanding.The idea is that,like,any
[21:27.00] API call has some schema and if you can
[21:29.00] define that schema ahead of time,you can
[21:31.00] use a language model to resolve a request
[21:33.00] into a much more complex request.One
[21:35.00] that an embedding could not do.So,for
[21:37.00] example,I have a really popular post called,like,
[21:39.00]ragg is more than embeddings and
[21:41.00] effectively,you know,if I have a question
[21:43.00] like this,what was the latest thing that
[21:45.00] happened this week?That embeds to nothing,right?
[21:47.00]But really,like,that query should just
[21:49.00] be,like,select all data where the
[21:51.00] date time is between today and today
[21:53.00] minus seven days,right?
[21:55.00]What if I said,how did my writing
[21:57.00] change between this month and last month?
[21:59.00]Again,embeddings would do nothing.
[22:01.00]But really,if you could do,like,a group
[22:03.00] by over the month and a summarize,then
[22:05.00] you could,again,like,do something much
[22:07.00] interesting.And so,this really just calls out the fact
[22:09.00] that embeddings really is kind of,like,the
[22:11.00] lowest hanging fruit.And using something
[22:13.00] like instructor can really help produce a
[22:15.00] data structure.And then you can just
[22:17.00] use your computer science and reason about this data
[22:19.00] structure.Maybe you say,okay,well,I want to produce
[22:21.00] a graph where I want to group by each month
[22:23.00] and then summarize them jointly.You
[22:25.00] can do that if you know how to define this
[22:27.00] data structure.Yeah.In that part,you
[22:29.00] kind of run up against,like,the
[22:31.00]lang chains of the world that used to have
[22:33.00] that.They still do have,like,the
[22:35.00] self-querying.I think they used to call it
[22:37.00] when we had Harrison on in our episode.How do you
[22:39.00] see yourself interacting with the other LLM
[22:41.00] frameworks in the ecosystem?Yeah.I mean,if
[22:43.00] they use instructor,I think that's totally cool.Again,it's,like,it's just
[22:46.00] Python,right?It's,like,asking,like,oh,how does,like,Jango
[22:49.00] interact with requests?Well,you just might make
[22:51.00] a request.get in a Jango app,right?But no one
[22:54.00] would say.I,like,went off of Django because I'm using requests now.They
[22:57.00] should be ideally,like,sort of the wrong
[22:59.00] comparison.In terms of it,especially,like,the agent
[23:02.00] workflows,I think the real goal for me is to go down,like,the LLM
[23:04.00] compiler route,which is,instead of doing,like,a
[23:07.00]react type reasoning loop,I think
[23:10.00] my belief is that we should be using,like,workflows.If we
[23:13.00] do this,then we always have a request and a complete
[23:16.00] workflow.We can fine tune a model that has a better
[23:18.00] workflow,whereas it's hard to think about,like,how do you fine tune
[23:21.00] a better react loop?Yeah.Do you want to always train it to
[23:24.00] have less looping,in which case,like,you wanted to get the right
[23:27.00] answer the first time,in which case,it was a workflow
[23:29.00] to begin with,right?Right.Can you define workflow
[23:31.00] because I used to work at a workflow company,but I'm not
[23:34.00] sure this is a good term for everybody.Oh,yeah,like,I'm
[23:36.00] thinking workflow in terms of,like,the prefect to zap
[23:39.00] your workflow.Yeah.Like,I want to build a DAG.I want you
[23:42.00] to tell me what the nodes and edges are,and then maybe the
[23:45.00] edges are also put in with AI.But the idea is that,like,I
[23:48.00] want to be able to present you the entire plan,and then
[23:51.00] ask you to fix things as I execute it,rather than going,like,Hey,I
[23:55.00] couldn't parse the JSON,so I'm going to try again.I couldn't
[23:58.00] parse the JSON,like,I'm going to try again.And then
[24:00.00] next,you know,you spent,like,$2 on opening AI
[24:02.00] credits,right?Yeah.As well as with the plan,you can just
[24:05.00] say,Oh,the edge between node,like,X and Y does not
[24:09.00] run.Let me just iteratively try to fix that
[24:12.00] component.Fix the one that sticks,go on the next component.
[24:14.00]And obviously,you can get into a world where if you have
[24:17.00] enough examples of the nodes X and Y,maybe you can use,like,a
[24:20.00] vector database to find a good few shot examples.You can do a
[24:24.00] lot if you sort of break down the problem into that workflow
[24:27.00] and execute that workflow,rather than looping and hoping
[24:30.00] the reasoning is good enough to generate the
[24:33.00] correct output.Yeah.You know,I've been hammering
[24:36.00] on Devin a lot.I got access a couple of weeks ago.
[24:39.00]And obviously,for a simple task,it does well.For the
[24:42.00] complicated,like,more than 10,20-hour tasks,I can
[24:45.00] see it in a lot of times.That's a crazy comparison,like,we
[24:47.00] use to type out,like,3,4 loops.You're like,only once
[24:51.00] it gets to,like,our tasks,it's hard.Yeah.Less than an hour,it's
[24:55.00] nothing.That's crazy.I mean,okay.Maybe my
[24:59.00] goal force has shifted.I don't know.That's incredible.Yeah.No,like,I'm
[25:03.00] like,I'm like sub one minute executions.Like,the fact
[25:06.00] that you're talking about 10 hours is incredible.I think
[25:09.00] it's a spectrum.I think I'm going to say this every
[25:11.00] single time I bring up Devin.Like,let's not reward them
[25:13.00] for taking longer to do things.Do you know what I mean?Like,that's
[25:16.00] a metric that is easily abusable.Sure.Yeah.You can definitely
[25:19.00] you know what I mean.But I think it's like,if you can
[25:22.00]monotonically increase the success probability over an
[25:26.00] hour,like,that's winning to me,right?Like,obviously,if you run an hour and you've
[25:30.00] made no progress,like,I think when we were in,like,auto-GBT land,there was that one
[25:34.00] example where it's like,I wanted it to,like,buy me a bicycle over
[25:37.00] night.I spent $700 and I never found the bicycle.Yeah.Yeah.Right.I
[25:41.00] wonder if you'll be able to purchase a bicycle.Because it actually can do
[25:44.00] things in real world,it just needs to suspend to you for often stuff.The point I was
[25:48.00] trying to make was that I can see it turning plants.I think one of the agents
[25:51.00]loop holes,or one of the things that is a real barrier for agents is LLMs really
[25:55.00] like to get stuck into a lane.And,you know,what you're talking about,what I've
[25:58.00] seen Devin do,is it gets stuck in a lane and he would just kind of change
[26:01.00] plans based on the performance of the plan itself.And it's kind of cool.I
[26:05.00] feel like we've gone too much in the looping route,and I think a lot of more
[26:08.00] plans and,like,dags and data structures are probably going to come back to help
[26:12.00] fill in some holes.Yeah.And what's like the interface to that?You know,you see
[26:16.00] it's like an existing,like,state machine kind of thing that,like,connects to the LLMs.
[26:20.00]The traditional DAC player.So,like,do you think we need something new for,like,a.i.dags?
[26:25.00]Yeah.I mean,I think that the hard part is going to be describing visually the fact
[26:30.00] that this DAC can also change over time,when it should still be allowed to be fuzzy.
[26:34.00]I think in,like,mathematics,we have,like,plate diagrams,and,like,marked
[26:37.00] octane diagrams,and,like,you know,recurrent states,and all that,like that.Some of
[26:40.00] that might come into this,like,workflow world,but to be honest,I'm not too sure.
[26:44.00]I think right now,the first steps are just how do we take this DAC idea and break
[26:48.00] it down to modular components that we can,like,prompt better,have few shot examples
[26:52.00] for,and ultimately,like,fine tune against.But in terms of even the UI,it's hard
[26:56.00] to say what,well,likely would.I think,you know,people like Prefect and Zapier
[27:00.00] have a bit of good shot at doing a good job.Yeah.You seem to use Prefect a lot.
[27:04.00]I actually work there at Prefect competitor at Temporal,and I'm also very familiar
[27:07.00] with Daxter.What else would you call out as,like,particularly interesting
[27:11.00] in the AI engineering stack?Man,I almost use nothing.Like,I just use
[27:16.00]everything.And,like,py tests.Like,okay.I think that's basically it.You know,a lot
[27:22.00] of the observability companies have,the more observability companies
[27:26.00] I've tried,the more I just use Postgres.Really?Okay.Postgres for observability?
[27:32.00]But,like,the issue really is the fact that these observability companies isn't
[27:35.00] actually doing observability for the system.It's just doing the LLM thing.
[27:39.00]Like,I still end up using,like,Datadoc,or,like,you know,Sentry to do,like,latency.
[27:43.00]And,so,I just have those system handle it.And,then,the,like,prompt in,prompt out,like,in C token costs.
[27:49.00]I just put that in,like,a Postgres table now.So,you don't need,like,20 funded startups
[27:53.00]building LLM ops?Yeah,but I'm also,like,an old-tired guy,you know what I mean?
[27:58.00]Like,I think,because of my background,it's like,yeah,the Python stuff,I'll write
[28:01.00] myself.But,you know,I will also just use Versal happily.Yeah,yeah.Because I'm just
[28:05.00] not familiar with that world of tooling.Whereas,like,I think,you know,I spent
[28:09.00]3 good years building observability tools for recommendation systems.And,I
[28:13.00] was like,oh,compared to that,instructor is just one call.I just have to put
[28:17.00]time,start,time,end,then count the prompt token,right?Because I'm not doing a very
[28:21.00]complex looping behavior.I'm doing mostly workflows and extraction.Yeah.
[28:25.00]I mean,while we're on this topic,we'll just kind of get this out of the way.You
[28:28.00]famously have decided to not be a venture-backed company.You want to do the consulting route.
[28:32.00]Oh,yes.The obvious route for,you know,someone as successful as instructor is
[28:35.00]like,oh,here's hosted instructor with,like,all tooling.Yeah.You just said you had a whole
[28:38.00] bunch of experience building observability tooling,like,you have the perfect
[28:41.00] background to do this,and you're not.Yeah.Isn't that sick?I think it's sick.I
[28:44.00] mean,I know why,because you want to go free-dive.Yeah.Yeah.Because I think there's
[28:48.00] two things,right?Look,one,if I tell myself I want to build requests,request is not
[28:53.00] a venture-backed startup,right?I mean,one could argue,like,whether a postman
[28:57.00] is,but I think for the most part,it's like,having worked so much,and more
[29:01.00] interested in looking at how systems are being applied,and just having access to
[29:05.00]the most interesting data,and I think I can do that more through a consulting
[29:08.00] business where I can come in and create,go,oh,you want to build perfect memory,you
[29:11.00] want to build an agent,you want to build,like,automations over construction,or,like,
[29:15.00] insurance,and it's a supply chain,or,like,you want to handle writing private equity
[29:20.00] mergers and acquisitions reports based off of user interviews,like,those things are
[29:23.00] super fun.Whereas,like,maintaining the library,I think is mostly just kind of
[29:28.00] like a utility that I try to keep up,especially because if it's not venture-backed,I have
[29:32.00] no reason to sort of go down the route of,like,trying to get a thousand integrations
[29:36.00] in my mind,I just go,like,oh,okay,98% of the people use OpenAI,I'll
[29:40.00] support that,and if someone contributes another platform,that's great,I'll
[29:43.00] merge it in.Yeah,I mean,you only addedenthropic support this year.
[29:47.00]Yeah,yeah.You couldn't even get an API key until,like,this year,right?
[29:51.00]That's true.And so,ok,if I add it,like,last year,I was trying to,like,double the
[29:55.00] code base to service,you know,half a percent of all downloads.Do you think the
[29:58.00]market share will shift a lot,now that Anthropic has,like,a very,very competitive offering?
[30:02.00]I think it's still hard to get API access.I don't know if it's fully GA now,if
[30:08.00] it's GA,if you can get commercial access really easily.I got commercial after,like,two weeks
[30:13.00] to reach out to their sales team.Two weeks.Yeah,so,it's not too bad.There's a call list here,and
[30:17.00] then anytime you run into rate limits,just,like,ping one of the Anthropic staff members.
[30:21.00]Yeah,then maybe we can,like,cut that part out,so I don't need to,like,you know.No,it's cool,it's cool.
[30:24.00]Fall it's news,but it's a common question.Surely,just from the price perspective,it's gonna
[30:28.00] make a lot of sense.If you are a business,you should totally consider Sonnet.The cost savings
[30:34.00] is just gonna justify it,if you actually are doing things at volume.And yeah,I think the
[30:38.00] SDK is,like,pretty good.But to back to the instructor thing,I just don't think
[30:41.00] it's a billion-dollar company.And I think if I raise money,the first question
[30:44.00] is gonna be,like,how are you gonna get a billion-dollar company?And I would just go,like,man,like,if I
[30:48.00] make a million dollars as a consultant,I'm super happy.I'm,like,more than a static.
[30:52.00]I can have,like,a small staff of,like,three people.Like,it's fun.And I think a lot of my happiest
[30:57.00] founder friends are those who,like,raised a tiny seed round,became profitable.They're making,like,
[31:02.00]70,000MRR.And they're,like,we don't even need to raise the seed round.Let's just keep it,like,between
[31:07.00] me and my co-founder,we'll go traveling,and it'll be a great time.I think it's a lot of fun.
[31:11.00]I'll write Pete to the seed investor in the company.Yeah.I think that's,like,one of the things
[31:16.00] that people get wrong sometimes,and I see this a lot.They have an insight into,like,some new
[31:21.00] tech.Like,say hello to MCI.And they build some open-source stuff.And it's,like,I should just raise
[31:26.00] money and do this.And I tell people a lot.It's,like,look,you can make a lot more money
[31:30.00] than something else than doing a startup.Like,most people that do a company could make a lot
[31:34.00] more money just working somewhere else than the company itself.Do you have any advice for folks
[31:39.00] that are maybe in a similar situation?They're trying to decide,oh,should I stay?Am I,like,high-paid
[31:44.00] fang job?And just tweet this on the side?And do this on GitHub?Should I go be a consultant?
[31:50.00]It seems like a lot of work.It's,like,you got to talk to all these people,you know.There's a lot to unpack.
[31:55.00]I think the open-source thing is just,like,well,I'm just doing it purely for fun.And I'm doing it
[31:59.00] because I think I'm right.But part of being right is the fact that it's not a venture-back startup.
[32:04.00]Like,I think I'm right because this is all you need,right?So I think a part of the philosophy
[32:09.00] is the fact that all you need is a very sharp blade to sort of do your work.And you don't
[32:13.00] actually need to build,like,a big enterprise.So that's one thing.I think the other thing,too,that
[32:17.00]I've been thinking around,just because I have a lot of friends at Google that want to leave right now,it's,like,man,like,what we lack is not money or skill.
[32:23.00]Like,what we lack is courage.You just,like,you just have to do this hard thing,and you have to do it scared anyways,right?
[32:29.00]In terms of,like,whether or not you do want to do a founder,I think that's just a matter of
[32:32.00]optimality.But I definitely recognize that the,like,expected value of being a founder is still quite low.
[32:39.00]Right.I know as many founder breakups,and as I know,friends who've raised a seed round this year.
[32:44.00]Right.And,like,that is,like,the reality.And,like,you know,even in,from that perspective,it's been tough,where it's,like,oh,man,like,a lot of incubators
[32:51.00]want you to have co-founders,now you spend half the time,like,fundraising,and then trying to,like,meet co-founders,and find co-founders,
[32:57.00]rather than building the thing.This is a lot of time spent out doing things I'm not really good at.
[33:02.00]I do think there's a rising trend in solo founding.You know,I am a solo.I think that something,like,30% of,like,forget what the exact
[33:11.00]estat is,something,like,30% of starters that make it so,like,series B or something,actually are solo
[33:15.00]founder.I feel,like,this must have co-founder idea,mostly comes from YC,and most,everyone else copies it,and then
[33:23.00]planning your company's breakup over co-founder breakups.And,I bet it would be,like,I wonder how much of it is the people
[33:28.00]who don't have that much,like,and hope there's not a diss to anybody,but it's,like,you sort of,you go through
[33:33.00]the incubator route,because you don't have,like,the social equity,you would need to sort of,like,send an email to
[33:38.00]Lacoy,and be,like,hey,I'm going on this ride.You want to take it on the rocket ship,right?Like,that's very hard to sell.
[33:43.00]My message if I was to raise money is,like,you've seen my Twitter.My life is sick.I've decided to make it much worse by
[33:49.00]being a founder,because this is something I have to do.So,do you want to come along?Otherwise,I want to fund it myself.
[33:55.00]Like,if I can't say that,like,I don't need the money,because I can,like,handle payroll,and,like,hire an intern,and get an assistant.
[34:01.00]Like,that's all fine.But,I really don't want to go back to Meta.I want to,like,get two years to,like,try to find a problem
[34:07.00]we're solving.That feels like a bad time.Yeah.Jason is,like,I wear a YSL jacket on stage at AI
[34:13.00]Engineer Summit.I don't need your accelerator money.And boots.You don't forget the boots.That's true.That's true.You have really good boots.
[34:20.00]Havily good boots.But,I think that is a part of it,right?I think it is just,like,optionality,and also,just,like,I'm a lot older now.
[34:26.00]I think 22-year-old Jason would have been probably too scared,and now I'm,like,too wise.But,I think it's a matter of,like,if you raise money,you have to have a plan
[34:34.00]of spending it.And I'm just not that creative with spending that much money.Yeah.I mean,to be clear,you just celebrated your 30th birthday.
[34:41.00]Happy birthday.Yeah.It's an album.We're going to Mexico next week.A lot older is relative to some of the folks I think.
[34:48.00]Seeing on the career tips,I think this works out a great post about our user world to get into AI.As on one of your tweets
[34:55.00]in January 23,you applied to,like,Figma,Notion,Coherent,Tropic,and all of them rejected you because you didn't have enough
[35:01.00]LLM experience.Yeah.I think at that time,it would be easy for a lot of people to say,Oh,I kind of missed the boat.You know,I'm too late.
[35:08.00]Not going to make it.You know.Any advice for people that feel like that?
[35:13.00]Like,the biggest learning here is actually from a lot of folks in Jiu Jitsu.They're like,Oh,man,like,is it too late to start Jiu Jitsu?
[35:18.00]Like,I'll join Jiu Jitsu once I get in more shape,right?It's like,there's a lot of,like,excuses.And then you say,Oh,like,why should I start now?
[35:26.00]I'll be like 45 by the time I'm any good.It'll be 45 anyways.Like,time is passing.Like,if you don't start now,you start tomorrow,you're just,like,one more day behind.
[35:35.00]If you're worried about being behind,like,today is,like,the soonest you can start,right?And so you've got to recognize that,like,maybe you just don't want it,and that's fine,too.
[35:43.00]Like,if you wanted it,you would have started.I think a lot of these people,again,probably think of things on a too short time horizon,but,again,you know,you're going to be old anyway,so you may as well just start now.
[35:53.00]You know,one more thing on,I guess,the career advice slash,like,sort of,logging.You always go viral for this post that you wrote,on advice to young people and the lies you tell yourself.
[36:02.00]Oh,yeah,yeah,yeah.You said that you were writing it for your sister,what?Like,why is that?
[36:06.00]Yeah,yeah,she was,like,bummed out about going to college and,like,stressing about jobs,and I was,like,Oh,I really want to hear.Okay.
[36:13.00]And I just kind of,like,text this for you the whole thing.It's crazy.It's got,like,50,000 views.I mean,your average tweet has more.
[36:20.00]But that thing is,like,a 30-minute read now.
[36:24.00]Yeah,yeah.So there's lots of stuff here,which I agree with.You know,I'm also of occasionally indulging the sort of life reflection phase.
[36:31.00]There's the how to be lucky.There's the how to have high agency.I feel like the agency thing is always a trend in SF,or just in text circles.
[36:39.00]How do you define having high agency?
[36:41.00]I'm almost,like,pass the high agency phase now.Now,my biggest concern is,like,okay,the agency is just,like,the norm of the vector.
[36:49.00]What also matters is the direction,right?It's,like,how pure is the shot?Yeah,I mean,I think agency is just a matter of,like,having courage and doing the thing.That's scary,right?
[36:59.00]You know,if you want to go rock climbing,it's,like,do you decide you want to go rock climbing,then you show up to the gym,you rent some shoes,and you just fall 40 times?
[37:05.00]Or do you go,like,Oh,like,I'm actually more intelligent.Let me go research the kind of shoes that I want.Okay,like,there's flatter shoes and more inclined shoes.
[37:13.00]Like,which one should I get?Okay,let me go order the shoes on Amazon.I'll come back in three days.Like,Oh,it's a little bit too tight.Maybe it's too aggressive.I'm only a beginner.Let me go change.
[37:22.00]No,I think the high-agent person just,like,goes and,like,falls down 20 times,right?Yeah,I think the higher agency person is more focused on,like,process metrics versus outcome metrics.
[37:32.00]Right?Like,from pottery,like,one thing I learned was,if you want to be good at pottery,you shouldn't count like the number of cups or bowls you make.You should just weigh the amount of clay you use,right?
[37:42.00]Like,the successful person says,Oh,I want to do a hundred pounds of clay,right?The less agency person is like,Oh,I've made six cups,and then after I've made six cups,like,there's not really,what do you do next?No,just pounds of clay,pounds of clay.
[37:53.00]Seeing with the work here,right?I say,Oh,you just got to write the tweets,like,made the commits,contribute open source,like,write the documentation.There's no real outcome.It's just a process.And if you love that process,you just get really good at the thing you're doing.
[38:04.00]Yeah.So,just to push back on this,because obviously,I mostly agree.How would you design performance review systems?
[38:11.00]Because you were effectively saying,we can count nicer code for developers,right?Like,did you?
[38:18.00]No,I don't think that would be the actual,like,I think if you make that an outcome,like,I can just expand a for loop,right?I think,okay.So,for performance review,this is interesting because I've mostly thought of it from the perspective of science and not engineering.
[38:31.00]I've been running a lot of engineering stand-ups,primarily because there's not really that many machine learning folks.The process outcome is like experiments and ideas,right?
[38:39.00]Like,if you think about outcomes,what you might want to think about an outcome is,oh,I want to improve the revenue or whatnot.But that's really hard.But if you're someone who is going out,like,okay,like,this week,I want to come up with,like,three or four experiments.I might move the needle.Okay,nothing worked.
[38:51.00]To them,they might think,oh,nothing worked.Like,I suck.But to me,it's like,wow,you've closed off all these other possible avenues for,like,research.Like,you're going to get to the place.So,you're going to figure out that direction really soon.
[39:02.00]There's no way you'd try thirty different things and none of them work.Usually,like,ten of them work,five of them work really well,two of them work really,really well.And one thing was,like,the nail in the head.So,agency lets you sort of capture the volume of experiments.And,like,experience lets you figure out,like,oh,that other half,it's not worth doing,right?
[39:19.00]I think experience is going to go,half these pumping papers don't make any sense,just use chain of thought and just,you know,use a for loop.That's basically it,right?It's like,usually performance for me is around,like,how many experiments are you running?How often times are you trying?Yeah.
[39:31.00]So,when do you give up on an experiment?Because a stitch fix,you're going to give up on language models,I guess,in a way,as a tool to use.And then,maybe the tools got better.You were right at the time,and then the tool improved.I think there are similar paths in my engineering career,where I try one approach,and at the time it doesn't work,and then the thing changes.But then I kind of soured on that approach,and I don't go back to it.
[39:52.00]Yeah,how do you think about that loop?So,usually when I'm coaching folks,and they say,like,oh,like,these things don't work,I'm not going to pursue them in the future.Like,one of the big things,like,hey,the negative result is a result,and this is something worth documenting.Like,this is an academia,like,if it's negative,you don't just,like,not publish,right?But then,like,what do you actually write down?Like,what you should write down is,like,here are the conditions,this is the inputs and the outputs we tried the experiment on.And then,one thing that's really valuable is basically writing down,under what conditions would I revisit these experiments.
[40:20.00]These things don't work because of what we had at the time.If someone is reading this two years from now,under what conditions will we try again?That's really hard,but again,that's,like,another skill you kind of learn,right?It's like,you do go back,you do experiments,you figure out why it works now.I think a lot of it here is just,like,scaling worked.Rap lyrics,you know,that was because I did not have high enough quality data.If we phase shift and say,okay,you don't even need training data.Oh,great,then it might just work.Different domain.
[40:47.00]Do you have anything in your list that is,like,it doesn't work now,but I want to try it again later.Something that people should.Maybe keep in mind,you know,people always,like,AGI when,you know,when are you going to know the AGI is here.Maybe it's less than that,but any stuff that you tried recently that didn't work that you think will get there.
[41:03.00]I think the personal assistants and the writing I've shown to myself is just not good enough yet.So I hired a writer and I hired a personal assistant.So now I'm going to basically,like,work with these people until I figure out,like,what I can actually,like,automate,and what are,like,the reproducible steps.But,like,I think the experiment for me is,like,I'm going to go pay a person,like,thousand of dollars a month to,like,help me improve my life,and then let me get them to help me figure,like,what are the components,and how do I actually modularize something to get it to work.Because it's not just,like,o-auth Gmail calendar and,like,notion.It's not just,like,it's,like,another skill.It's,like,another skill.
[41:30.00]It's not just,like,o-auth Gmail calendar and,like,notion.It's a little bit more complicated than that,but we just don't know what that is yet.Those are two,sort of,systems that,I wish gb4 or opus was actually good enough to just write me an essay,but most of the essays are still pretty bad.Yeah,I would say,you know,on the personal assistant side,Lindy is probably the one I've seen the most.Flow was a speaker at the summit.I don't know if you've checked it out,or any other,sort of,agents assistant startup.
[41:55.00]No,not recently.I haven't tried,Lindy.They were not GA.I was considering it.Yeah,yeah.A lot of it now,it's,like,oh,like,really,what I want you to do is take a look at all of my meetings and,like,write,like,a really good weekly summary email for my clients.To remind them that I'm,like,you know,thinking of them and,like,working for them,right?Or it's,like,I want you to notice that,like,my Monday is,like,way too packed,and,like,block out more time.And also,like,you know,that people to do the reschedule and then try to opt in to move them around.And then I want you to say,oh,Jason should have,like,a 15 minutes.
[42:24.00]Prep break after form back to back.Those are things that now I know I can prompt them in,but can it do it well.Before,I didn't know that's what I wanted to prompt for us.Defragging a calendar and adding breaks so I can,like,eat lunch.Yeah,that's the AGI test.
[42:39.00]Exactly.Compassion,right?I think one thing that,yeah,we didn't touch on it before,but I think it was interesting.You had this tweet a while ago about prompts should be code.And then there were a lot of companies trying to build prompt engineering tooling.Kinda.
[42:53.00]Trying to turn the prompt into a more structured thing.What's your thought today?Now you want to turn the thinking into DAGs,like,do prompts should still be code?Any updated ideas?
[43:03.00]Ah,it's the same thing,right?I think,you know,with Instructor,it is very much,like,the output model is defined as a code object.That code object is sent to the LLM and in return you get a data structure.So,the outputs of these models,I think,should also be code objects.And the inputs somewhat should be code objects.
[43:20.00]But I think the one thing that Instructor tries to do is separate instruction data and the types of the output.And beyond that,I really just think that most of it should be still,like,managed pretty closely to the developer.
[43:31.00]Like,so much of it is changing that if you give control of these systems away too early,you end up ultimately wanting them back.Like,many companies I know that reach out are ones where,like,oh,we're going off of the frameworks because now that we know what the business outcomes we're trying to optimize for,these frameworks don't work.
[43:47.00]Yeah,because we do rag,but we want to do rag to,like,sell you supplements or to have you,like,schedule the fitness appointment.The prompts are kind of too big in the systems to really pull them back out and,like,start doing upselling or something.It's really funny,but a lot of it ends up being,like,once you understand the business outcomes,you care way more about the prompt.
[44:06.00]Actually,this is fun.In our prep for this call,we were trying to say,like,what can you as an independent person say that maybe me and Alessio cannot say or,you know,someone who wants to get a company say.
[44:15.00]What do you think is the market share of the frameworks?The land chain,the llama index,the everything.
[44:20.00]Oh,massive.Because not everyone wants to care about the code,right?I think that's a different question to,like,what is the business model and are they going to be,like,massively profitable businesses,right?
[44:31.00]Making hundreds of millions of dollars,that feels,like,so straightforward,right?Because not everyone is the prompt engineer,like,there's so much productivity to be captured in,like,backoffice,off to automations,right?
[44:42.00]It's not because they care about the prompts that they care about managing these things.Yeah,but those would be sort of low-code experiences,you know?
[44:48.00]Yeah,I think the bigger challenge is,like,okay,100 million dollars,probably pretty easy.It's just time and effort,and they have the manpower and the money to sort of solve those problems.
[44:58.00]Again,if you go to the VC route,then it's,like,you're talking about billions,and that's really the goal.That stuff,for me,it's,like,pretty unclear.
[45:05.00]But again,that is to say,that,like,I sort of am building things for developers who want to use Instructure to build their own tooling,in terms of the amount of developers there are in the world,versus,downstream consumers of these things,or even just think of how many companies will use,like,the Adobe's and the IBM's,right?
[45:19.00]Because they want something that's fully managed,and they want something that they know will work.And if the incremental 10% requires you to hire another team of 20 people,you might not want to do it.
[45:28.00]I think that kind of organization is really good for those,like,bigger companies.
[45:32.00]And now,I just want to capture your thoughts on one more thing,which is,you said you wanted most of the prompts to stay close to the developer.And Hamel Hussein wrote this post,which I really love,called,FU Show Me The Prompt.
[45:43.00]Yeah.
[45:44.00]I think he cites you in one of those part of the blog post.And I think DSPy is kind of,like,the complete antithesis of that,which is,I think it's interesting,because I also hold the strong view that AI is a better prompt engineer than you are.And I don't know how to square that.Wondering if you have thoughts.
[45:57.00]I think something like DSPy can work because there are,like,very short-term metrics to measure success,right?It is,like,did you find the PII?Or,like,did you write the multi-hop question the correct way?
[46:12.00]But in these workflows that I've been managing,a lot of it,are we minimizing churn and maximizing retention?
[46:18.00]Yeah,that's a very long loop.It's not really,like,a uptuna,like,training loop,right?Like,those things are much more harder to capture.So we don't actually have those metrics.
[46:26.00]And,obviously,we can figure out,like,is the summary good,but,like,how do you measure the quality of the summary?It's,like,that feedback loop ends up being a lot longer.And then,again,when something changes,it's really hard to make sure that it works across these,like,newer models,or,again,like,changes to work for the current prompts.
[46:44.00]Like,when we migrate from,like,anthropic to open AI,like,there's just a ton of changes that are,like,infrastructure-related,not necessarily around the prompt itself.Yeah,cool.
[46:53.00]Any other engineering startups that you think should not exist before we wrap up?
[46:58.00]No,I mean,oh my gosh.I mean,a lot of it,again,is just,like,every time investors,like,how does this make a billion dollars?Like,it doesn't.I'm gonna go back to just,like,tweeting and holding my breath underwater.Yeah,like,I don't really pay attention too much to most of these,like,most of the stuff I'm doing is around,like,the consumer of,like,LLM calls.Yep.
[47:18.00]I think people just want to move really fast,and they will end up pick these vendors,but I don't really know if anything has really,like,blown me out the water.Like,I only trust myself.But that's also a function of just being an old man.Like,I think,you know,many companies are definitely very happy with using most of these tools anyways.But I definitely think I occupy a very small space in the AI engineering ecosystem.Yeah.
[47:39.00]I would say one of the challenges here,you know,you call about the dealing in the consumer of LLM's space.I think that's what AI engineering differs from ML engineering,and I think a constant disconnect or cognitive dissonance in this field in the AI engineers that have sprung up is that they're not as good as the ML engineers.They're not as qualified.I think that,you know,you are someone who has credibility in the MLE space,and you are also a very authoritative figure in the AI e-space.And
[48:08.00]I think so.I think you've built the de facto leading library.I think yours,I think instructors should be part of the standard lib,even though I tried to not use it.Like,basically also end up rebuilding instructor,right?Like,that's a lot of the back and forth that we had over the past two days.I think that's the fundamental thing that we're trying to figure out.Like,there's very small supply of MLEs.Not everyone's going to have that experience that you had,but the global demand for AI is going to far outstrip the existing MLEs.So what do we do?Do we force everyone to go through the standard MLE curriculum?
[48:37.00]MLE curriculum or do we make a new one?
[48:39.36]I got some takes.
[48:40.80]I think a lot of these app layer startups should not be hiring MLEs because they end up turning.
[48:46.56]Yeah, they want to work at opening high.
[48:48.44]They're just like, hey guys, I joined and you have no data.
[48:51.84]And like all I did this week was take some typescript build errors and like figure out why we don't have any tests.
[48:59.00]And like what is this framework X and Y?
[49:01.40]Like how do you measure success?
[49:02.76]What are your business outcomes?
[49:03.84]Oh no, okay, let's not focus on that.
[49:05.60]Great, I'll focus on these typescript build errors.
[49:09.04]And then you're just like, what am I doing?
[49:10.52]And then you kind of sort of feel really frustrated.
[49:12.60]And I already recognize that because I've made offers to machine learning engineers.
[49:18.08]They've joined and they've left in like two months.
[49:21.32]And the response is like, yeah, I think I'm going to join a research lab.
[49:24.00]So I think it's not even that, like I don't even think you should be hiring these MLEs.
[49:27.56]On the other hand, what I also see a lot of is the really motivated engineer that's doing more engineering
[49:34.20]is not being allowed to actually like fully pursue the AI engineering.
[49:37.60]So they're the guy who built the demo, it got traction, now it's working.
[49:40.72]But they're still being pulled back to figure out why Google Calendar integrations are not working
[49:45.08]or like how to make sure that, you know, the button is loading on the page.
[49:48.16]And so I'm sort of like in a very interesting position where the companies want to hire MLE.
[49:53.32]They don't need to hire, but they won't let the excited people who've caught the AI engineering bug
[49:57.84]could go do that work more full time.
[50:00.00]This is something I'm literally wrestling with this week.
[50:02.84]As I just wrote something about it, this is one of the things I'm probably going to be recommending in the future
[50:06.40]is really thinking about like, where is the talent coming from?
[50:08.96]How much of it is internal?
[50:10.12]And do you really need to hire someone who's like writing pie torch code?
[50:14.00]Yeah, exactly.
[50:15.24]Most of the time you're not, you're going to need someone to write instructor code.
[50:20.60]And like, I feel goofy all the time, just like prompting.
[50:23.24]It's like, oh man, I wish I just had a target data set that I could like train a model against.
[50:27.16]Yes.
[50:27.68]And I can just say it's right or wrong.
[50:29.24]Yeah.So, you know, I guess what Lanespace is, what the AI engineer world's fair is, is that we're trying to create
[50:35.48]and elevate this industry of AI engineers, where it's legitimate to actually take these
[50:40.36]motivated software engineers who want to build more in AI and do creative things in AI
[50:44.48]to actually say you have the blessing, and this is legitimate, sub-special to your software engineering.
[50:49.72]I think there's been a mix of that product engineering.
[50:52.44]I think a lot more data science is going to come in versus machine learning engineering.
[50:55.92]Because a lot of it now is just quantifying.
[50:57.96]Like, what does the business actually want as an outcome?
[51:01.16]The outcome is not rag-app.
[51:02.56]The outcome is like reduced churn.
[51:04.64]You wouldn't need to figure out what that actually is.
[51:06.20]I had to measure it.
[51:06.96]Yeah.All the data engineering tools still apply.
[51:09.28]BI layers,sematic layers, whatever.
[51:12.32]Yeah.
[51:12.96]Cool.We'll have you back again for the world's fair.
[51:15.88]We don't know what you're going to talk about, but I'm sure it's going to be amazing.
[51:19.12]You're a very, very polished speaker.
[51:20.48]The title is written.
[51:21.68]It's just a, "Pydantic is still all you need."
[51:26.24]I'm worried about having too many all-you-need titles, because that's obviously very trendy.
[51:30.20]So, you have one of them, but I need to keep a lid on everyone saying their thing is all you need.
[51:35.28]But yeah, we'll figure it out.
[51:36.44]Pydantic is not my thing.
[51:37.68]So, what else?
[51:38.52]I think that's why it works.
[51:40.56]It's true.
[51:41.28]Cool.Well, it was a real pleasure to have you on.
[51:43.24]Of course.
[51:43.76]Everybody should go follow you on Twitter and check out Instructure.
[51:46.68]There's also Instructure.js, which I'm very happy to see.
[51:49.36]And what else? Anything else to plug?
[51:51.76]Useinstructor.com.
[51:52.84]We got a domain name now.
[51:54.04]Nice.
[51:54.84]Nice.Awesome.
[51:55.80]Cool.Cool.
[51:56.92]Thanks for your time.
[51:57.72]Thanks.
[51:58.72](音乐)
