Note: There a live version on Youtube and Spotify if you prefer audios š
Antoine
I think the best way to start the interview, is by letting you introduce yourself and talk about your work in rendering!
Inigo
Yeah as you said Iāve been working mostly on rendering since the beginning of 90ās. I was born in San Sebastian small city in the Basque country between Spain and France. I worked in virtual reality in the early 2000ās in Belgium in Brussels thatās where I learned some French back then it was with huge Silicon Graphics machines and you needed like a hundred thousand dollar equipment to do VR but that was what we got then. I went to work at Pixar where I did work on offline renderings. I learned a lot about lighting and Composition and more of the artistic side of computer graphics not so much on the algorithmic stuff. Then, I worked in Oculus where I did a bit of both where I was doing film and we were trying to discover the language of cinematography in a real-time context and in a virtual reality environment but I was doing both low-level mean and high-level art direction and things and then I created a painting program called Quill for painting in virtual reality and animating and then I became the lead for that project for about four years at, first oculus and then Facebook which is our parent company so some other
In my spare time Iāve been doing lot of computer graphics as well first in the demo scene which is this underground community of computer graphics and artists people in Europe mostly and I joined in 1998 and I was focused mostly on making computer graphics with very small executables where you try to generate as much data visuals,textures,meshes and sound from as little amount of code as possible and then, related to that I created Shadertoy about seven years ago, although the first version was ten years ago. I made sure that I kept that spirit of doing a lot of things with very little amount of code but I made it in a more less underground way where I make it well exposed so that people can have a username and they can associate a work to their name not necessarily a nickname and also where the code is free and available right from the beginning as soon as you publish a piece the code is there for everyone to analyze and then Iāve been also doing a lot of educational content where I have a website where I publish all ideas or experiments. Well not all of them because I donāt have time to write every single thing I do but I try to publish about all the major things that I find them are some discoveries never big ones but little discoveries sometimes are things I learned and I put it all online for people to learn.
\
Antoine
On your website thereās a lot of stuff, youāve been publishing since the late nineties. Your are an electrical engineer on your academic formation, how much does it serve you? I know a lot of people ask you this question and interview but I just wanted to discuss on this.
Inigo
Itās a good question because everything I know about computer graphics and programming itās something I learned on my own by either reading or looking at somebody elseās code which in turn in turn motivated me creating ShaderToy to kind of give back again to the community because I learned everything from the Internet but the math I was studying there didnāt include anything related to number theory or permutations, combinatoric none of that. I geometry of course and calculus, mostly Fourier transforms Laplace transforms calculus and things like that so it does help because anything that has to do with data compression or things like that I can work on those environments. I guess the main thing that studying not computer graphics helped me is to approach things differently I suppose because I never learned how to do anything in a book so I had to find my way and that means that first I had to reinvent the wheel many times you know like I had to discover by myself things that I could have just read about but that meant I got used to discovering new things and giving a try giving a code to problems instead of just looking at how to solve Ć
It also meant that I found variations to some solutions that normally wouldnāt be implemented the way I did so overall I think I am in a place where I am because I donāt have that formal background in computer graphics but I do have it in electrical engineering.
Iām ready to try things in different ways, I guess of problems differently, I just land on the solution that everyone has landed, I found and I hit some little gems here and there.
Antoine
Youāve talked about ShaderToy which is your main side-project. I would like to discuss the graphic culture and general rendering and then after weāre gonna go to some more geeky and nerdy stuff. Shadertoy is a platofrm I use daily, and realized there is a lot of love and huge respect for sharing. Was it the idea when creating the platform 10 years ago?
Inigo
It was since day -1! I had first created a small ShaderToy and it was called ThatWay in 2009 with the first WebGL implementation that Firefox released. I reached out to people I knew and I got them to send me shaders by email and I would publish them myself in that website. I was hunting somewhere from my main websites URL but I had to ask people to send me the code by themselves.
Then two years later a friend of mine, Ricardo caballo, Spanish as well, demosceners as well who made ThreeJS , made his own version of ShaderToy his own which was called GLSL Sandbox and that got very popular because people would upload shaders on their own without having to send him emails like they had to do with my side. The problem with his page was I think that there was no attribution, no one knew who was the owner of the shader. I thought that was a pity that the sense of community was lost it was just anonymous people sending often times duplicated shaders without opportunities to talk about the shader, discuss about it. So thatās when I joined with my actual partner and colleague, partner in crime for this business which is Paul Jeremias who works at Pixar now and together we built the new ShaderToy, with community in mind. You need to enter your name and your email to register, we donāt use it for anything, but for letting you know that someone commented on your Shader. This way you can build a solution of shader and you would be able to build a reputation like āThis is my name, these are my shadersā. So yeah the idea was to build a community for sure! The fear was will people respond to this proposal will they want to join in when there are so many other communities already! You have the demoscene community with their own websites and portals, you have a trillion things but it did work mostly I think because the first round of shaders were injected by people that I hand-picked before the website was public so I emailed 20 colleagues from the game industry, the film industry, from the demoscene and I asked them for a favor which was to publish two or three shaders in the website. So by the time we opened and went public there was already a good collection of very interesting shaders and I think that was the hook to start getting everyone else to join in.
Antoine
Thatās a very nice backstory and does it still evolves nowadays?
Inigo
Paul and I, which we meet every Tuesday night: we have dinner and then we we add new features or fix bugs in shader toy so yeah itās a slow pace in that itās not our job itās not a full-time thing, there is no ambition to grow it or sell it or monetize it or making it into a business although it might have to happen at some point if we want to give better features to the users like storing their own data and so on we might have to pay servers which means we might have to start maybe thinking about how to monetize.
Maybe not for everyone but maybe for some users, I donāt know anyway because we donāt have big plans for it or we donāt depend on it to make a living we are taking it easy and enjoying the ride versus having to rush. In that sense we add features as we please and as we find them interesting and the other thing is that there are many features that some hardcore users are desperate to have myself includedā¦
Antoine
What kind of features these people want?
Inigo
We all want to have more control on the multipass system when you write a shader in different passes. We want to control not only wich resolution they are, but also what pixel format they have: integer vs float, or how often they refresh do every pass does run thus every pass run every frame or every other frame or it runs three times per frame? So all those controls have to happen if we want to improve the algorithms but there are other features which are less interesting or less sexy but which are more useful for the broader user base of ShaderToy: like being able to follow creator that you like or getting to get notifications when they upload a new shader or sending messages across users so they can discuss and get in touch for a business or whatever they want to do. So thereās always this tension between what I want as a user and what I want as a product manager.
Antoine
You also have worked at Pixar on offline rendering system, was it Path tracing rendering back then?
Inigo
At the time it was notā¦ I think Pixar switched to full path traced and ray-tracing rendering just three years ago. I left five years ago so no at the time it was rasterization.
Antoine
āOh so all the movies like Cars, they were made using rasterization?ā
Inigo
Yeah yeah yeah, itās a variant variation of rasterization where itās not exactly the same as how the GPU rasterizes things which is mostly in screen space. They split measures into tiny tiny micro polygons as they call them in object space and then they splat those into the screen but itās rasterization meaning you cannot just cast a ray to query global information about the scene like āis there a shadow here or is there an object blocking the light?ā so although you could do it I was very expensive. 99% of things were just shadow maps based or point clouds based to bake ambient occlusion and things like that. I think they kept using rasterization until The Good Dinosaur, wich was rasterized but some shadows were ray traced and then Monster Inc changed to path tracing.
Antoine
You worked a lot on environnement in your career, your work on creating terrain is mindblowing! What is your creative process when creating large environnement and how does it defers from creating characters?
Inigo
On the terrains or landscapes versus characters I suppose which is how you normally divide every team that ismaking a game or film. Iāve traditionally been doing sets or landscapes of terrains or environments because itās the easiest for me. I grew up in the Basque Country in the Northern Spain, there is a lot of mountains I spent my childhood either hiking or skiing so I know how those things look When I make a landscape, I always work from memory I donāt look at reference pictures or anything! Not because I shouldnāt which is the right thing to do but just for the [came of it], I do it like a painting.
Then characters is something I have never done before a year ago I guess. I took it as a challenge, of course I can keep learning and doing better at landscapes a lot of my landscapes out there are not the best at all became easy enough to do that even refining my process and getting better that it would be a bit of a refinement thing where I would get slightly better with every the more effort I put. I would get smaller and smaller quantities of getting better it would be like a diminishing returns thing. While making characters: itās an open field which I have never touched so I can do a lot of progress very quickly! Again not that Iām gonna reach a super high level but I can get better than I am right now \very quickly so thatās exciting because itās a new thing for me.
I have to learn what a face looks like, what are the components that make a face how they move the expressions so yeah Iām taking it a bit of a challenge. I made that yellow jumping guy which is like 5 blobbs you know itās not very complex but it was animated which I had never done before! I made a boy as well in ShaderToy, procedurally with ellipsoids and parabolas and a few other things and heās there youāre staring at the camera, but he doesnāt blink. So I have a yellow guy which is very simple but is animated and I have a surfer boy who is complex but not animated and now I want to do both. I want to do something complex and animate it. Thatās the shader Iām working on now when I have a bit of time. The process in that case unlike the landscapes and the terrains which I can make from memory is that for characters I definitely have to look online to see how about that other people do characters: both for films, for caricatures and so on. So I try to you watch a few Youtube Videos and see what they have to say about it. Iām also not very systematic as in āIām going to spend one week now learning about character designā because I like to put my hands to work and get things going and discovering things by my own by
But every now and then I do go online to learn a bit. I would add that thereās a third type of shaders or pictures or paintings or whatever you want to call them, that I do which are just abstract! They are not character or landscape but just something that is inspired by some geometrical configuration that I found while trying to discover some equation or whatever!
Then if thatās the case, if I Iām working on a cubic equation or the SDF of a triangle, whatever it is, then chances are I might take that little discovery and apply it in a shader which just tries to be beautiful or stylish but uses that mathematical concepts. And normally they end up being very very abstract. I also do those ones, although Iām not that good at that!
Antoine
When you work on the SDF function, do you consider it being more art or math?
Inigo
It is a combination of both actually! We could specify for the listeners what an SDF is. Itās a formula and a mathematical expression that encodes the shape and the geometry of some figure like a triangle or a star or a circle. So instead of defining those shapes with vertices of polygons as you would do normally in traditional computer graphics the SDF is a purely mathematical representation, no polygons, it is just an equation. You code those and through different algorithms becomes a shape on the screen. Unlike what some of the people following me on ShaderToy assume, I am not that good at math.
Antoine
What do you mean?
Inigo
Iām not so good at math as in I donāt see in 3d very well actually so there are people who you give them a Rubikās Cube and they can solve it or they can understand how you should solve it and they can do it or if you ask me whatās the intersection of a plane and a cone when the plane is more than 45 degrees, itās not immediately obvious to me that thatās a hyperbola or a parabola or an ellipse. I have to write down the things and work out the maths and then I have an answer. Itās not like I can see that much, but a little bit of course.
So when I have to do an SDF most of the times I do the drawing and soon enough I convert it to actual equation that I put in the paper.
The very first times I would go through the process of deriving everything through equations and getting a very complex equation and then trying to simplify it to runs efficiently in the computer. But now with experience I have learn to find the shortcuts before itās too late, meaning I can find the symmetries of the problem. For example, you know like a triangle if itās a isocele triangle then the right side and the left side are the same therefore you donāt need to compute both you can just compute one of them and copy that result or the maths to the other side. So those shortcuts I can take them right away now just from experience like you know when you have done a thing a billion timesā¦ The is the trick basically.
Antoine
Do your find yourself working a lot the SDF on paper nowadays, or you pretty much derive them all
Inigo
Yeah, every other week or so I have a new SDF and the reason is that I Iām not in a quest really to find or to write the brightest sDF equations but with any new shader thing that Iām writing I oftentimes have to derive a new SDF because I need it for the thing Iām doing for example so itās always motivated by the picture, or the art I am working on.
So if Iām working on a new character and sheās gonna be moving her face and talking, I needed to sculpt the shape of her mouth and her jaw. An ellipsoid wasnāt giving me enough of a sharp shape to it and a square was too sharp so I had to find new SDF which allows me to do more looking like a rhombus shape and then extrude and round it. So I did the equationsā¦ And again, now because Iāve done already so many of them I can do it in a few minutes and in five minutes it is done and then I publish it as an independent SDF and I put it in my website in the catalog of SDF and then I move on to working onthere on the shader.
The driving force is always the artistic thing. Now Iām learning to do characters or trying to learn so thatās the goal and you want to impress people with a character that they have never seen in real time as a mathematical equation in ShaderToy or in the form of a mathematical equation so thatās the the goal.
The process process of going through it: itās a big mix of maths and also those shortcuts that I take which are either learned by having done so many of the thems or sometimes by intuitions. Sometimes I am like: āIt feels like these SDF should be be able to do it with one only one square rootā or āit feels like there is too many computations here,in the end of the day this shape is just five lines, so why should I have more than five divisions for example.ā So itās a mic of math and intuitions.
Antoine
How long does it take you to make a complete scene?
Inigo
It depends a lot on the on the painting! Those who comes from already learned process, those comes very quickly. It could be from between an hour to 2 hours, then I publish them. Others who are more more technical, it could be a couple of hours. So in the case of the Yellow Jumping Guy, it was around 10 hours, split between two session on 5 hours. I normally even if I had ten hours in a row which I donāt anymore because now I have a family, while before I could do that not anymore I cannot do those beautiful session of whole day coding. Even if I had the time I wouldnāt because Iām taking a break and coming back the day after to the shader the week after helps a lot to see things that you didnāt see while you were deep into the process of making it!
But the next one I am making now is a totally different story! Because every step of it is different, I have to learn something like just for
Sometimes suddenly, you go to bed thinking āWoah! That shader that I am makingis looking good itās gonna be amazingā than the the next day is like āWoah, those lights are badā¦ā. So itās good to take a break.
But the next one I am making now is a totally different story! Because every step of it is different, I have to learn something like just for the slot in the shape I spent easily an hour just finding what were the shapes that I needed to use to to give good definition to her face. I went online to check how animator does it at DreamWork, what are the muscles around the eye etcā¦ So in the end it might be a total of 10ā15 hours I donāt know but, itās gonna be spread over a month or more, in the small chunks of time because that also gives me time to think about what Iām going with! it because
Right now in this shader, Iām just putting in place the main pieces which is the modeling itās not finished so you have to put the mouth and I want to change the proportions but Iām 70% there in terms of lighting! Iām I started with generic light rigged that with a key light and a bounce light but now Iām starting to put about a second backlight Iām starting to slowly transform the generic light rig into something that is more motivated by an environment or a setting. For character, I take time to think of the setting: āis she in an office? is she hacking into a computer that she shouldnāt be hacking into or is she making a shader at home?ā. I take my time to think about what she is doing, to give a more accurate feeling to the scene.
I like to take the time to think about all those little details, so when I sit down in front of the computer I can actually foccus on turning those reflections I had into codeā¦
Antoine
A lot of your scenes have really different vibes to them, some are more PBR driven, while other have more cartoonish look. Do you follow any rules, such as: how the light should behave in a certain type of scenes, use PBR rules etcā¦ ?
Inigo
So, itās a good question! I do a blend weāre in I use principles of PBR but not necessarily, but I donāt go full-on PBR correctness and so I would say that even if the style is cartoonish itās always good to start from some physical principles. So these days, we are already used to working in linear space doing our lighting in linear space and then doing only in the end a gamma transformation and perhaps some color gain control or something but it wasnāt true a few years ago rightā¦
Five years ago this was not knew. So when that thing kicked in and we are all learned of the benefits of doing light in a linear space and how much realistic things look like, I embraced it, right away so now I never again made a shader without a square root in the end for linear to perceptual gamma space conversion. And I never did my lighting in any other space that wasnāt linear and that helps a lot! Because again all the things things look more plausible even if you are doing something cartoonyā¦
Well there are two philosophies here I guess but Iām taking the Pixar and Disney philosophy which is when you make a cartoon which is what they do they keep shapes stylized you donāt go crazy with adding wrinkles to the face or anything. Super simple shapes just enough to convey expressions and feelings, materials are super simple as well they donāt go crazy with how much scratches things have, or if they do they make them very tiny. They keep the surface very clean, but that doesnāt mean that the lighting is not realistic. In the way it bounces around the shadows the shape the penumbraās, all that is realistic because that helps us understand depths and the relative positioning of things.
I think thatās the reason but thatās there thatās the formula, some other people go crazy with lighting as well and they do super weird things and thatās fine! But Iām taking their approach I think itās a successful combo: stylized shapes, plausible lighting.
So either youāre doing photorealism stuff or you are doing that cartoony approach where everything is cartoony except for the light. Having lighting that is physically based is a good idea so what I do is I do the basic things of physically-based stuff like linear space and so onā¦ I do more the PBR stuff on the lighting, even for cartoonish scene, but I do not go crazy on it, I donāt know by heart the micro facets models. I just do whatever it takes to make the result pretty.
And actually, oftentimes you might find me doing things where I take the n dot l term which is the basic lighting term for a diffuse lighting situation and I might just multiply it by 0.5 and add 0.5 to kind of shift thing around.
Antoine
I see this a lot in your shaders, the āmultiplied by .5 + .5ā trick!
Inigo
I mean you can choose those coefficients to decide how much the lighting wraps to the back of an object even though the light source is not hitting there you can lick some lighting in there and fill the darks which is very useful. You could do it by adding more lights to the scene and having fill lights everywhere but thatās easy and looks very looks beautiful and itās very useful for a cartoonish or painterly situations.
Actually, I saw someone doing that on Twitter the other day and he was like: āOh this is amazing, a big discovery, oh this is amazing this is what I inventedā and I was like that was just since the 80s by everyone who ever made computer graphics although now itās discouraged. He or she, I canāt remember, was doing it in Unity and we may have a complain here about these game engines, I believe I havenāt used them but it looks to me like they forced you into a physically based world or territoryā¦
Antoine
Unity is now implementing a whole new scriptable render pipeline, to allow custom flexibility! It is very powerfull and driven by Sebastien Lagarde
Inigo
Nice, that flexibility is great, altough, this is a bit of a brainstorm thing now or Iām making this idea to go but thatās great that you can crack open an engine and customize it for your needs if you need game is to be cartoony then do the cartoony shader that we were talking about like adding point five to the diffuse or whatever.
But still means you have to go out of the easy path: which is just using the thing that comes by default and thatās a pity at the same way they have a PBR default I would say, I would love that they had a cartoony default where all these tricks that we all know about: adding .5 to the diffuse, colorizing the occlusion and apply in a color ramp to the penumbra of a shadow, all those things have been used for example by Pixar before they went path tracing and global illumination! The movies I made were full of that because we didnāt have GI right so if a shadow was looking too dark you would go to that shadow and multiply it or add a bit of blue to it and then it looks great, and if it looks too sharp then you would take a smooth step and separate the dark from the bright.
In order to do those things: the lighting artists had the tools! They didnāt have to hack the renderer! They already had all these non-physically based controls ready with nice UI. I wish that these engines had as well, the same way they have all the default PBRās nicely implemented, that they had another set of non-photorealistic things with all those tools that the industry learned about since the eighties and have them available for those who want to make games that are either cartoony in look or need to run very efficiently say on an Oculus Quest, where every trick you can use to hit a giving look helps.
On Occulus, you donāt have the performance to start casting rays or doing shadow maps or doing the PBR thing. So yes, if I was there I would propose it: āHey guys, can we put a teram of five people to work on a nice set of tools for those who need something different!ā.
Antoine
Do you think the cinema industry is more carefull on following the correct way to handle PBR, and the video game industry is more hacky to hit these high FPS?
Inigo
I would say, I almost feel like it might be the other way around! Iām not in the game industry, I have never worked in a game studio, I have never played a video game and I never used a game engine on a daily basis.
But I have read most of the papers from GDC and all the presentations on the blogs by the game developers thatās how I know Sebastian Lagarde. I kind of think I know whatās going on and I feel like there is this trend where everything has to be correct and not so much there are not so many rooms for hack, at least in lighting and rendering. Iām sure they hack everything else and thatās why I was saying that it would be nice that these tools gave you ways to hack. Hacking is a good thing for the tool not a thing to avoid!
I think that films they are all about hacking! First of all you can hack because every shot is a different thing, every scene is a different set of files. Itās not like in a game where everything has to look good from every angle, you donāt know where the user is gonna be in the film you know these.
So you basically hack everything per shot basically, and a shot is three seconds of film. Every shots has a different set of lights, different tweaks every shader has its own parameters that have tweaks for that one shot, so itās a lot of hacking! Even from a lighting perspective, even real films not even CG films like you put bounce cards for the lighting to be better in the context. You put a bounce light to fill their wholes, and that bounce light shouldnāt exist in reality there is no floating white umbrellas in the street, walking around. But you hack it, and thatās fine!
So films is all about hacking about hacking and I feel game is trying to be more correct. Which is kind of the opposite of what the intuition should tell us, they cannot afford that kind of realism, films can and they donāt! I mean those who argue that now do you use PBR and path tracing so thatās realistic
But I always think, as I was saying before, that bringing plausible behavior of light which is realistic in the way it bounces in a non realistic situation, which is having bounced cards and thingsā¦
Antoine
So there is a lot of hacks in the movie industry!
Inigo
Yeah absolutely! You can decide you know, you have your PBR and path-traced stuff and that helped a lot getting a first look for your shot. So before, when there was a new shot on the film you would have an artist work for two weeks placing five hundred point lights to kind of fake global illumination. And then you could have a first look of the scene and then you could start working on the art like: āwhere do the highlights goā or āwhere do the rim lights goā.
But with PBR, physically based lighting and path tracing you hit a button and in five minutes you have something that looks great! So you saved two weeks, but itās far from being final quality!. So what you do when the show needs to be approved is you have your global illumination solution but then you tweak it and you say things like: āYou know what the bounce light is not in her face, is not bright enough letās make the bounce of object A twice as strong!ā. So you can can fake physicality!
Example: the light bounces depends on the incoming angle, the outcomig angle, the PBR, the material parameters! You have the equation and that tells you how much irradiance is getting out or whatever! But in film you tweak it! You say āokay it doesnāt look good enough, letās times two!ā. You multiply everything or that object, if PBR by two, now itās emitting more light and itās receiving, physicality is broken but it looks good soā¦
Antoine
Thatās very interesting stuff! I did not knew that. Itās very nice to know this!
Inigo
Crossing my fingers, that I donāt receive an email tomorrow from Pixar!
Antoine
Haha, if there is anything you want to remove before we publish, just let me know! The other day I was making an interview with a guy from Ubisoft, and he was going about very particular algorithms used by the studio and I was like: āWait a minute, this is not public!ā
Inigo
We are good, we are good! What I am saying is about the film industry in general, not from a particular studio! Just tweak, you need to make the picture look good, because everything is in the service of the story, right? Example if the moment is happy, but the shadow in the face of the character are dark and dramatic, those things donāt match so you have to pump it up and put a fake light or a bounce card or just take some reflectance function and make it brighter!
Antoine
Are you a movie fan?
Inigo
I like making films, I donāt watch them. I donāt watch TV neither. I think last year, 2019, I watched only one film!
Antoine
Wich one!?
Inigo
I watched the Pixar release! I have friends working there so they invited me to see the films when they get released. It was Toy Story 4. So I saw that one and then at home we donāt do Netflix or anything like that.
I saw a couple of episodes of The Marvelous Mrs. Maisel. The story is not that strong, I mean itās interesting, itās good for keeping us hookedā¦ But the lighting, the costume design and color grading was beautiful. Iām not a story person that much, either.
I like things that looks pretty!
Antoine
If you are curious, about ābeautiful thingsā you should check out the work of Wes Anderson. When you watch his movies, you are somehow more attracted by the look than the story itself. Anyways, we are a bit moving away of rendering and time is passing, so I would like to ask you about VR!
You worked in the VR industry and movie! You worked on Dear Angelica, wich is a wonderfull piece of artā¦ Where do you find the inspiration?
Inigo
This is a team work! In this one, Dear Angelica, in terms of visuals besides the technical aspects which I took care of, the visuals themself are 90 percent the work of the artist who painted it and and the director who had the idea for the story and and led her to paint all those things.
So my role in that movie was VFX supervisor.The idea was: we want to make a movie or a film which is VR meaning is all around you, you are in it not watching it from the distance and we are not going to do something cartoony as in a CG DreamWorks film like we had done before with Henry! We needed to do something that is like illustrations and drawings, and asking ourselves: āhow do we do that?ā
Thatās where I created Quill, and VR drawing and animation tool. So my role there was to provide solution like making drawings in VR a reality and what does it mean to. So we tried things, from having to do the drawings in space, to having 2.5 D drawings you know drawings with depth but that still always looked very CG, you could see that it was 2d drawings in billboards basically.
drawings you know drawings with depth but that still always looked very CG, you could see that it was 2d drawings in billboards basically.
So I brought my role there again was basically enabling the look of the film, for then the artist to go wild and bring their creativity to it. And it turned out to be beautiful, I think.
Antoine
Being a huge movie watcher, I can say that the final result is really doing the thing, I cried in my VR when watching itā¦ What is the main challenge to write VR code?
Inigo
The challenge is clearly performance! How much you can do, especially now that VR is becoming mobile you no longer have a PC necessarily! You donāt have a 2080 by your side to run your program. You have to run on a mobile phone hardware. So iās really about performances.
There is a few things you can do, the first one that should always be done is: Pick your visual style wisely! Donāt try to do photorealism sutff in VR. And to link it to what we were saying earlier, as engine should provide default not-pbr and cartoony tools and looks. Donāt try to do what the hardware cannot do, if you try, you will end up with something that does not really look good.
If you do, youāre gonna have to sacrifice too many things: the resolution of your shadow Maps, how many of them you have, your specularā¦ Everythingās gonna look ugly and weird, iās gonna land in no mans land! If itās not cartoony the viewers expect realistic but then it is not good enough to be fully realistic and you are in a weird place, weird spot, where it looks like a game from the 90s or something.
Antoine
When itās movie or short films, are all those things precomputed? Example on my Occulus GO, I cannot really move in space, except from turning my head in 360 degrees, do you pre-compute a lot of the data that are rendered or is it all real time
Inigo
No, it is in real time, because the user can be anywhere. I mean, with the GO, wich is the one you have, you donāt have positional tracking, all you can do is to look around. So in that case, you could pre-compute things. You could render the whole movie in a cubemap or in a stereo cube map, but thatās the past!
Now VR is positionally tracked, meaning you can move anywhere! Not only that you can walk, meaning you can stand up from your chair and explore every part of the film, but also meaning that if you are sitting on your chair and moving your head left to right, you need an effect of parallax. So you have to render in real-time.
So the tricks: again the first one choose your style and then I would say secondly hack the rendering techniques that work for that style! You have to render polygon spin thereās no way around it, but maybe things in the distance can be rendered with billboards or 2.5 D billboards or as point clouds or maybe, you can render the left eye and reproject it on the right eye and only fill the gaps of the reprojection. But this may only works for flat surfaces because you can easily fill the gap, so if your style is cartoony and everything is shaded flat you can do it but maybe if you had a lot of texture maybe that does not work anymore!
So again: everything start with what you want to see visually and work the techniques, the rendering algorithms and the shaders backwards to what is the artistic goal. And hack a lot!
Antoine
Do you think using analytical formulas or ray-marching can help to reduced cost, example on the shadows?
Inigo
Traditionally if you want a soft shadow you have to render a shadow map and dopercentage close filtering, and sample it many times. You totally cannot do this on a VR device, it is way to expensive. But if you replace, for example, the object with sphere, than you can compute exact shadow in a very cheap way, only with a few multiplications and one division or something.
I gave a talk at Siggraph about this, wich is my philosphy. Sometimes itās worth replacing the problem wich is casting the shadow of my character by another simpler problem, wich is casting the shadow of spheres instead of a character. Then solve that problem analytically and perfectly.
So itās like: making an approximated solution to an exact problem which is a crappy shadow map for a character versus an exact solution to an approximated problem which is perfect shadows for the spheres that approximate the character. And again, I think a lot of these solutions has to be done these days. Example, if you have a mobile device, and you are rendering at 90 FPS you have 11 miliseconds to render your scene twice.
Luckily the latest device, like the Occulus Quest is not 90 frame per seconds, itās 72, I think.
Antoine
Do you still do VR stuff?
Inigo
No, I am taking a break! I am foccusing now on ShaderToy and videos for Youtube, and live streaming.
Antoine
You have moved freelance few months, do you like it, do you miss the industry?
Inigo
Well I am not doing so much work, I am taking a small vacation, altought I took a couple of small gigs. And I donāt know, I mean, not working is awesome!
I know it wonāt last forever, especially here in San Francisco, it is really expensive, this place is good to burns some money.
For the industry: I miss it and I do not miss itā¦ I miss being in a team, like when I was making Quill, the impact of the work we were doing was huge. We were shifting the industry of animation, to start using VR tools, so that was exciting. But when you lead a team, you have to do things that may not be your cup of tea.
While what I do know, on a daily work is great, I get immersed in my shaders!
Antoine
Thanks a lot Inigo! It was such an honnor to talk with you. Are you going to Siggraph this year?
Inigo
I might go, this year is in Washington I think. So I donāt know if I will go, because now with my daugther I donāt really leave home for many days. Washington is a bit further than Vancouver or LA. But I will do Sigraph Asia, probably.
Antoine
Ok, well letās keep in touch! If you coming to Washington for Sigraph 2020, letās go have a coffee!
Inigo
Yeah sure!
Antoine
Thanks a lot for joining! Au revoir!
Inigo