Emit is the conference on event-driven, serverless architectures.

Cornelia, ever the modest one, began her talk by saying that she "probably wouldn't blow any minds here." Au contraire—her talk was one of the most talked-about at the afterparty.

She focused on the way we educate upcoming programmers, and the implications it has down the line for their ease in reasoning about event-driven systems. She happened to learn Pascal first, which gave her primitives of variables and assignments, pushed her straight into iterations and control loops. It forced her very early on to get comfortable with leaps of faith.

What about students who start off with object-oriented languages? They tend have a hard time reasoning about recursion at first; it just breaks too far outside the paradigm they were given. When our baseline is object-oriented programming, it structures our mindset such that it can be harder to naturally reason about event-driven systems.

But she has great suggestions for turning this around, and making event-driven systems more intuitive.

Her talk is well worth watching, or read the transcript below.

More videos:

The entire playlist of talks is available on our YouTube channel here: Emit Conf 2017

To stay in the loop about Emit Conf, follow us at @emitconf and/or sign up for the Serverless.com newsletter.

Transcript

Cornelia: Boy, I don't think I'm gonna blow any minds here. But it is kind of a provocative title, intentionally so. So the interesting thing is that I'm coming at the end of the day, as you said, it's getting late in the day. And so I've had the benefit of reflecting on the talks that have gone on throughout the day. And I wanted to share that reflection even before I jumped into the talk because I think it's relevant. So over the course of the day I've heard us talk about things like CRDTs and functional programming, and computer sciences stuff. And I even texted my husband and said, "Holy crap, this is so cool. We're doing real computer science here in this environment." And I think that's a reflection of us being still very early on in this new paradigm, in that the primitives that we're dealing with are still kind of these low-level primitive. And, I think, all of us computer scientists in the room know that when you're doing distributed systems, things like immutability are key, but we have to.

I think, one of the challenges though, is that we have to translate that. Because we can't expect people with PhDs to be developing our web apps in the future. And that's really what I wanna talk about today. Is kind of, sort of, start to postulate on what those things are. Now, a little bit more about me, I do work for Pivotal. I've been working as a part of the Cloud Foundry team for four and a half, five years now. And so, I've had a little bit of a benefit of going through something similar here. So I'll tell you that four or five years ago, that what we know now and are even kind of poopooing about microservices, "Oh that is so early 2000s, right?" Was something that was newer few years ago. And there were a lot of hard problems that we as an industry made consumable and made usable. And I think that's our challenge here. And so that's, I guess, the benefit of going very late in the day as I've had a chance to reflect on that.

So let me start out with a little bit of a personal story. So the first thing that I'll ask is, "What language is this?" Yes. You guys are good. This was the first language that I programmed in college. I actually was very lucky that I programmed in high school a little bit in BASIC on a TRS-80, but when I got to college as an undergraduate at Cal State University, Northridge in the greater Los Angeles area, a university whose computer science department was really focused on teaching people what they needed to know to get out into industry and start being effective right away. I learned Pascal. That was the first language. And when I put together this example, a couple of days ago, I have to tell you that I didn't remember the first thing about Pascal Syntax. So that's what I meant by, "You're good." I wouldn't have recognized that as Pascal.

But what we learned there was it was a semester system, and what we did was we learned this. And I'll go through this in a little bit more detail in just a moment. Now, after going to school and then working for a number of years my husband and I said, " Ah, screw this work thing." And we went back to school full-time. And I don't know if majority's still here, but I went to IU for graduate school. A school in southern Indiana out in the middle of nowhere, and I like to say that they really couldn't have given a damn about getting people ready for industry. They were really kind of just a theoretical computer scientist school. And I studied programming languages, and I did my course in essentially theoretical computer science, theoretical.

And so, I was talking with somebody earlier today that one of the classes I was minoring in philosophy, and one of the classes, single hardest class I ever took was a set theory class. And so it's really hard core, you know, mathematical computer science. Well, at IU they teach in their very first programming language, in their very first freshman course in computer science they teach it in scheme. And so this is the same program in Pascal and scheme.

So let's look at a couple of differences here. So if we go back to the Pascal one, what are the primitives that they were teaching me? Well, the primitives that they were teaching me were variables. They were also teaching me assignment. So right from the get-go they taught me side effect. And from my preamble, do you know where I'm going with this? Right? So the primitives that they taught me here were side effects and that worked okay in that setting. We learned about iteration and control loops, and look at that. I was delighted when I went back and looked at Pascal because even the syntax drove home the point that I wanted to make here, which is when we're programming iteratively, we start at the beginning and we go to the end.

Now, if we contrast that to a functional programming model...and so when Bobby and I were chatting earlier, when I saw the title of this talk, I'm like, "Oh, he's doing the functional thing too." I'm a functional programmer at heart. If we take a look at the primitives here, what we have are conditionals, we have a base case, so we really need the conditional to decide between things, and then we have the general case. And what's so interesting about the general case is what's in the parentheses there, is that in order to program in a functional style, there is a point where you make a leap of faith. You say, "I'm going to solve the non-base case, the general case, by assuming that the previous case was solved for me, and I'm just going to figure out what the difference is."

And as we go through this, I'm starting to see those moments where we need to have the leaps of faith. As a programmer in this environment, I sometimes have to have a leap of faith. It's up to us as a community to fulfill that leap of faith. Now, there's another interesting thing about this. And this is really where when I was thinking about this talk, you know, several weeks ago, I'm where I wanted to get...that's why I came up with this rethinking thinking. I was at Cal State, Northridge, like I said, we were on semesters, and we learned those control structures. And when we got to recursion in week 13 of a 16-week semester, holy crap, was it hard?

People just didn't get it. The people who were still left in the room, it was something that they had to wrap their head around. It was a completely different way of thinking, because it required that leap of faith, and that wasn't a programming model that they had. At IU, and I taught while I was there, we taught recursion in week three. And you know what? Everybody got it. It wasn't that hard. And it's because we started there. And it wasn't that the students were just that much smarter.

All right. So recursion was completely natural and it had to do with the primitives that we taught in the way that we taught it. All right. So then, what are the questions? And I'm going to assert today that what we're gonna be doing here in event-driven architectures is we are asking people to change the way that they're thinking. We're asking them to go from iteration to recursion, that analogy. And so, now it's our job to say what are the primitives? So is it a variable and a side effect? Or is it a base case in a general case? What are the primitives? What are the patterns around those and I'll say more about those in just a moment. And what are the platforms?

And it was really awesome to see Austin talking about this, and many of the other speakers talking about these platforms that they're building. We just had that talk from Uber. We had a platform that we're building to make this course accessible. And so all three of those things are very, very important. So are those primitives functions and events? Maybe. I mean, certainly we're here at Emit, and I think events are central. Is our function central? I don't know. I'm being a little provocative here, I know we've talked about server listen functions all along. I'm pretty set on events, but I don't know. But honestly, I have no idea. I really have no idea. So I will tell you that this is the world I've been living in. I am going to be going through that transformation. I'm on the beginning of that journey transforming from the imperative to the functional model to the recursive model. And so this is the world that I've been living in, and so I will tell you right now this is the area that I'm the most comfortable in, and when I get later on to some of the event stuff, I'm postulating. I'm imagining, if you will, we've had some of those other talks as well.

Now, this model here, is a pretty basic model. It's a simple mental model for kind of a micro-service architecture. I like to talk about things like apps, services which kind of take apps and make add some additional things to them, and I'll talk about some of those things. There's other services like data services, or Rabbit MQ, we've heard a lot about that as well. And then there's this notion of cloud native data. How do we deal with this distributed data fabric? And it's interesting that it's in the data space, that events are starting to percolate into this microservices world that I live in on a day-in and day-out basis.

So those are kind of some of the primitives that I wanna talk about. Now, the interesting thing that happened, like I said, I've been doing this for about the last four or five years, is that when we started coming up with these primitives, and we started, and we actually brought a platform to market, we brought Cloud Foundry to market, open source and commercial versions of it, I was spending a lot of time, I'm in a position where I spend a lot of time with our customers helping them get wrapped around this new thing, this thing that was very new for them. And we're asking them to go from their monoliths to their microservices which, again, we've been poopooing out a little bit today, but that has been pretty transformative for a lot of people in the industry. It's this realization that there's no free lunch with that. And somebody had a slide earlier. I don't remember anymore who it was, they had a really great slide, I think it was Nordst...was it you Rob, who had the cartoon slide, where you had, okay, where the old problems are sailing away, but the truck was coming in with the new problems.

So when we went from monoliths to microservices we had the new truck, we're gonna have the new truck again here. And so the whole point here and the reason I brought this slide up is that we're in the same position. We're gonna see this article written about events if it hasn't already been. There's no free lunch. And we have to figure out what those things are. So I'll talk about a few of those. And I'm gonna go through this relatively quickly for two...the reason I'm spending time on the microservices approach is first of all, to share with you what I've learned over this journey over the last four or five years. Maybe for us many of us have learned this in our journeys over the last four or five years to remind us of what we went through and how we came to understand what those patterns are, and what those practices are.

Also, I wanna drop parallels because it feels in some ways like the event driven model is completely new, but there are a lot of things that carry over from this world as well. And so I'll be pointing those things out as we go. All right. So, one of the other things that I wanna point out here is on the slides, the things that I have bolded in white there, is I keep coming back to, why are we architecting things the way that we are? Whether we're doing it in an event-driven space or in a micro-services space space, why are we doing things? What are the things that we're trying to achieve? So when it comes to things like HA and scaling, well, we created multiple instances. We didn't used to have that, we know that the kind of the model in the cloud is to do scale out. Well, as soon as I have multiple instances that means I have to take on the burden of load-balancing across those multiple instances.

So I had to figure out all sorts of things about dynamic routing, load balancing. And one of the things about the cloud is things are always changing. So how do I keep my load balancer up to date? All of those things. I also have to deal with externalizing my configuration even more so than before, because I now have multiple instances, and I need to configure them all the same way. So I can't have everything totally embedded. When it comes to resilience, I need to have statelessness. And, oh, and one of the other interesting things there is, whether we know that one of the fallacies of distributed computing is that the network is stable, so my services are always gonna be available when I need them. So to compensate for that, we came up with this concept of retries. Oh, but as soon as we do retries, we actually invite the next problem which is DDoS attacks. So retry storms. And there have been some great stories out in the media about, you know, even AWS has gone down because of retries. And they're kind of good at this stuff.

So the next thing, if we talk about...that was kind of starting with apps. But now if we talk about services, it's everything that we did with apps, but now, we also want to address autonomy and agility. So we want these services to be independent. We want teams to be able to develop them independently. We've heard those same themes all throughout today. And so we need to do things like version our services, so that we know that if we have to make backward breaking changes, that our consumers know which versions that they can have and we have that great talk from Stitch Fix on even how you do some testing around that and breaking changes.

The environments are always changing. That's one of the fundamentals that it used to be years ago that we measured up time in a matter of months, or years, and now the average container sticks around for four minutes. So things are always changing. So I have to do things like, I have to deal with service discovery. So if my service went poof over here, I need to know where I can find it next. Some of these things, the same problems exist but they shift into different places when we move to events.

And talking about data, speaking of data, it's an interesting one here. So again, agility and autonomy. I'll tell you that one of the things that we've done in the microservices world, and I live this day-in and day-out with our enterprise customers, is that they've created these independent microservices and they think that they're loosely coupled, but if you peel back the curtain, you pull back the curtain just a little bit, all of those independent microservices are tied to the same shared database and same shared schemer. So what looks to be an independently loosely coupled isn't at all. And so what we do there is we...what the kind of the narrative that's out in the industry now is, well, every microservice gets its own database. And then people start talking about, "Okay, well then, I've got these distributed data fabric, how do I treat it as a single hole?"

And people start talking about things like master data management. That doesn't make you like kind of shiver in your boots. And then, of course, data also is about resilience. So I spent the last year really working on what we call Pivotal Cloud Cache which is a...and I wanna emphasize the cloud, it's a cache that adapts to all of these like dynamic environments and things like that. And being able to use caches to be part of the bulkhead is an important pattern that's in microservice architectures. So and then, of course, there's the whole collective. So I've been talking about some of these independent pieces, but how do we treat that as a whole? Now, none of that has really been talking specifically about events. And so here's kind of a summary just to remind us all of these are the things that we're trying to achieve, and by the way, these are the same things that we wanna achieve with our new architecture as well, right? It's not new outcomes we're trying to achieve, it's the same outcomes we're trying different architectures to see if it works a little bit better.

So what I wanna do now is, again, I am not the expert in the space. There are many of you in this room who are far more experienced with event-driven architectures than I am. But I wanna start kind of postulating, imagining, if you will. On the left-hand side, remember just a moment ago I talked about retries. And as soon as we did retries, which was an important thing to do, I mean, even our browsers can sometimes do retries on our behalf. Retries are an important pattern in the distributed system. But then we have to deal with circuit breakers. Instead, over on the event-driven side is the primitive that we need to start teaching, is the primitive that we need to start thinking about instead of going from top to bottom, from begin to end, and say, "I need to go get something, and if I don't get anything back I'll do a retry." Am I taking kind of the leap of faith here? Is this the general case? Where I say, "Ah, I'm just going to have faith that this promise is going to be fulfilled at some point."

Now, I want us pause here for a moment because I think this is one of the most important things for us in this event-driven architecture, is that the platform is responsible for fulfilling this promise. The platform has to ensure that that event doesn't get lost. And so somebody talked about this a little bit earlier, and it's kind of permeated through the entire day is that, "Okay, we have to make sure that this substrate that's managing these events are managing them in such a way that the programming model can just work against that." If we burden the developer with not just worrying about writing the promise, if we don't give them the opportunity to have that leap of faith, then that changes the programming burden that we put on them completely.

Okay. All right. Oh, and from circuit breakers, again, this is the point that I was just making, ensuring that they don't get lost. Now, another one, like I said, I spent the last year or so really focusing on our cloud caching product, and it's an interesting thing. I work with some colleagues who have been working in this space for a long time, and they spend a lot of time talking about things like expiration. So that's one of the hardest things in caching, is to figure out when to vacate things out of the cache. That's a really challenging problem. Is it time-based? But how do we decide that? And the interesting thing is I think this is hugely powerful as over on the event space. Instead of worrying about when we stop believing what's there, we depend on, and again, we've got a leap of faith here. We depend on the fact that the substrate is actually refreshing that cache for me. So I actually like to talk about those things as materialized views, and we've been talking about that from an event sourcing perspective. So you can kind of think of it as a cache.

In the event-sourced model, we know that the unified log is the source of record, that is the source of truth, everything else is just a cache, it's a materialized view whatever word you wanna use for it. Again, we have to ensure that the events are not lost. Now, from kind of this service-oriented thing, and that's where I was saying earlier, I don't know. I'm still kind of struggling between this. Are services different from functions? Or functions just one type of service, I think that's something that we need to kind of figure out as an industry. But I should actually have multiple instances of the service there because that's why we need load balancing.

But what's interesting here, and I couldn't help when I was putting these slides together, thinking again about the difference between kind of iteration and recursion, is that over here we've got a service that stays up and running. And so we are always thinking about that services staying up and running and we tell people, we heard Cloud Foundry have been saying for a long time, make your stuff stateless. Because you can't guarantee that that's gonna be around. And, you know what? We also support sticky sessions. Drives me insane, I'm like, "No. I don't wanna support sticky sessions." But we allow, we still give people a little bit of rope to hang themselves with.

Instead, over here, we've got functions that come into being with every single request. That right there is a significant change. What I think is really interesting about this, and so from the load balancing, to functions I have another call out, is I kind of feel like this is a little bit of a stack frame. Each one of those, every time you do a recursive call, and if you're doing proper tail recursion, you're replacing the stack frame from before. And even if you're not properly tail recursive, the only scope that you're allowed to be in is that stack frame. You get a fresh stack frame with every single call. You don't get to reuse what was in that context before. In an iterative mode, you've always got the possibility that you're pulling something from some other context that wasn't inside of that loop.

All right. And then a couple of other things when we start moving into the data fabric is going from this world where we have a data bottleneck, and I mentioned this already, that this is to a large extent what we see is we've got. We've started to scale out and we know that one of the fundamental models in the cloud is to scale out via more instances, but what we haven't done is figured out how to scale out the data TRS at the back-end. This is someplace where we definitely, even in the world that I live in, where we're very heavily microservices based, where we are starting to see some movement there. What I do find interesting though, is that it's an area where it shifts from this modeling the functionality, the functions themselves, and I don't mean functions in the serverless sense.

But modeling the compute over to making the first-class entity, the actual data topology. So the schemers and those types of things, so making something like data partitioning, a first-class concern and something that you're thinking about right from the get-go, that's not something I think most developers are thinking about right at the get-go. So that's something we need to do. And, oh, and partition resolvers, really cool stuff. I'm running out of time, so I won't talk about that.

And then this is the image that we've seen from several talks today which is to say instead of having a shared database, we want each one of our services to have their own views, their own materialized views, and this is where we're going in with. And, of course, eventing is at the core of that. So there's a couple of things...oh, and then of course, event producing, but we've seen lots of that. And I am down to my last minute or so. There are many, many, many, many questions. And I can't even begin to write them all down. But what I wanna leave you with is, these are the kind of set of things that we think about from a service-oriented perspective. And these are just my early ideas of some of the primitives that I think about when I'm making this transition over into the event world. But the last thing that I wanna leave you with, is I wanna go back to scheme.

And who knows what call/cc is? The other functional programmer in the room. Call/cc is call-with current-continuation. And so I was thinking, "How can I express this?" And I did a little bit of Googling, and I found a question on Quora that said, "What is call/cc in layman's term?" So, Bobby, you must think that's kind of funny, right? Because there's nothing layman about call/cc. So I wanna read this to you. The answer is, suppose you have a function F which takes one argument which is supposed to be a continuation. Wait a minute, what's a continuation? Anyway, then call/cc is a special function that takes F as an argument, and calls F passing it as its argument the current continuation, which is the continuation in which call/cc itself was called.

Yeah. And look at the first line, hard to explain in layman's term in just a sentence. It's hard to explain this in layman's terms if you have a volume. So I just wanted to close on this. Again, we can't expect people if we want this event-driven world to go huge and have a huge market, we can't expect people to understand call/cc. So that's our challenge and that's something that I'm excited to see us in this industry doing in the next several years. And that's all I've got. Thanks so much.