Kolja Lampe takes us on a deep dive into the internals of the Elm Language Server that powers a growing number of editors' Elm support, and explains what motivates him to continue to spend more time building tooling for his favourite front end language than he does using the language itself!
Kolja Lampe takes us on a deep dive into the internals of the Elm Language Server that powers a growing number of editors' Elm support, and explains what motivates him to continue to spend more time building tooling for his favourite front end language than he does using the language itself!
Thank you to our sponsor, Culture Amp.
Special thanks to Xavier Ho (@Xavier_Ho) for editing and production of this episode!
Recording date: 4 April 2020
00:00:00 Intro
00:02:11 Kolja introduction
00:03:17 Side projects at work in Elm
00:05:26 Kolja's history with web development
00:11:13 Elm Language Server origins
00:15:26 Language servers and Language Server Protocol
00:20:56 Elm tree sitter
00:28:09 Elm Language Server internals
00:30:13 The Elm community in Germany
00:33:47 Type inference and other future features
00:40:13 Outside contributions
00:45:36 Increasing robustness by removing dependencies
00:50:41 Depending on feedback from the community
00:53:01 Mutual appreciation society
00:54:06 Sign-off and outro
[00:00:00] Kevin Yank: Hello and welcome back to the Elm Town podcast. It's Kevin coming to you with another episode. I'm joined today by Kolja Lampe. Hi Kolja.
[00:00:09] Kolja Lampe: Hi.
[00:00:10] Kevin Yank: Kolja goes by @Razzee on the Elm Slack. You may have seen him there especially in the #elm-language-server channel. that is a bit of a spoiler about what we're going to be talking about today. Kolja is the author of the Elm Language Server and the Elm Visual Studio Code Plugin that leverages it to provide a really nice and increasingly nice editor experience for that editor, and we'll be going into all the nitty gritty details there as well as hearing, you know, what brought Kolja to Elm. But before we dive into all of that, as usual, I would just like to shout out to Culture Amp for supporting this podcast. Aside from my time, I often credit them for letting me record these podcasts during working hours, but just so happens it's a Saturday evening, so it's not working hours today, but they pay for the very cool web-based recording studio that we use to make this show. Just in general, they don't mind the slightly excessive amount of time that I put towards this show sometimes. So it is great to have an employer like that, that doesn't mind giving back to the open source community that puts together the language that we love using to build our front ends there.
[00:01:26] As I've said before, Culture Amp builds a web application that companies all over the world use to hear from their employees about what they're thinking and feeling about their jobs and their workplace. That's something I really love going to work every day to contribute to. And also a shout out to Xavier Ho.
[00:01:45] Xavier, I have been overwhelming you with the number of new episodes I've been recording ever since this, COVID0-19 virus came in and, and made us all stay at home. It suddenly seems like a really good time to stock up on podcast recordings, but as I record this, there are four episodes in Xavier's inbox waiting to be edited.
[00:02:03] So, thank you so much, my friend for continuing to help me put these out. Kolja welcome to the show.
[00:02:10] Kolja Lampe: Hi, nice to be here.
[00:02:12] Kevin Yank: For the benefit of our listeners and for myself, who this is the first time I'm getting to meet you properly, why don't you introduce yourself?
[00:02:20] Kolja Lampe: Well hi, I'm Kolja and I'm mostly working on open source stuff right now, well I'm actually going to work normally - as normal as you can go to work right now. Like I'm working from home, obviously.
[00:02:33] Kevin Yank: There's not a lot that's very normal at the moment.
[00:02:36] Kolja Lampe: Yeah, agreed. I'm actually enjoying working from home very much. Programmers are in a good position to do that.
[00:02:44] Kevin Yank: Do you work professionally in Elm or is Elm a pastime for you at this point?
[00:02:49] Kolja Lampe: It's a pastime for me. I do have, like, we have something that we call 10% time at work, and I do have a project there that's using Elm. But other than that, it's not used at work.
[00:03:05] Kevin Yank: Okay. Yeah, I've seen a lot of that happen, of Elm being used as kind of a tool for building admin UIs and internal tools and things like that.
[00:03:16] Kolja Lampe: So if you're interested in what I built, it's actually something where we as a company, like everybody can log in and say, I have these skills, like I can program, and I have spent this many hours in doing it, and maybe this many hours in doing Elixir, and I really love it, or I really hate it. And so everybody in the company gets, an idea who knows what.
[00:03:45] And, I actually do some graphs based on that and stuff like this.
[00:03:50] Kevin Yank: That's cool. okay. And that's something that you've just built, internally for the use of your company.
[00:03:55] Kolja Lampe: Yeah, at the moment right now. but we've been thinking to open source it, but it needs more work to not be entangled with the company network, I guess.
[00:04:07] Kevin Yank: I have the exact same problem with a side project that I'm working on at work in Elm. One of the things that I'm doing at work is an app called the Face Wall. And, what it does is it connects to our Slack account and pulls down all of the employee profiles, and it's an app that runs on the meeting room TV screens when they're not in use and it
[00:04:28] just kind of goes through a slideshow of showing the employees and information from their Slack profiles. And that is written all in Elm, but just like yours, there is a significant amount of code in it that is very specific to our company and the specific fields we have in our Slack profiles.
[00:04:46] And so disentangling that, making it configurable and usable by another company is something I wouldn't mind doing one day, but I'm not sure if I'll get around to it.
[00:04:53] Kolja Lampe: Yeah, I totally understand that. We actually have been thinking if it would be a good thing to open source it right now.
[00:05:01] Kevin Yank: Yeah. Oh, we have a lot in common, it sounds like. I don't know what your company's main product is, if it's even a product company, but I am fascinated by these internal tools that connect people within an organization, because that's kind of my day job, is to work on a software platform that does things like that.
[00:05:20] Kolja Lampe: Yeah. I realize that; that's actually why I bring it up.
[00:05:23] Kevin Yank: Well, thank you! So how long, apart from Elm, how long have you been doing, you know, web application development of one kind or another? And, eventually I'd like to hear how you discovered Elm.
[00:05:37] Kolja Lampe: web application development only started at my current workplace, and I've been here for three or four years. I don't
[00:05:45] remember exactly. Before that I was doing a lot of .NET, like C#, mostly a desktop application for a warehouse, basically. So you put some data into masks and those get transferred to a web shop and the web shop will do stuff with it.
[00:06:03] And logistics. We'll do stuff with it and stuff like that, like get packages and, get them to customers.
[00:06:11] Kevin Yank: And you know, more and more of that sort of software is moving to the web these days. So, I suppose, did you consider it a luxury to be working on a desktop app and not have to worry about the web and JavaScript and all of the, the cruft that comes with that?
[00:06:29] Kolja Lampe: It was pretty nice in some regards, but we did do very many things in SQL Server. And that's kind of a two sided blade. It's very, very powerful, but it's also very easy to do some stuff wrong, I guess. It's, as with any organization, there are… there might be some people that are not exactly helping, and have some point of views that are very different from yours.
[00:07:01] So it's interesting, but at some point you have to move on, I guess.
[00:07:06] Kevin Yank: Yeah. You take the good with the bad in any ecosystem until at some point you get to try something new, I guess.
[00:07:13] Kolja Lampe: Yeah. At work, I'm now doing Angular for better or for worse, I guess. And yeah, that's quite a big change.
[00:07:20] But, it's not like we're only doing the front end, like I'm doing pretty much full stack. And to stack is quite, quite deep in the project. Like we got RabbitMQ, we got TimescaleDB, Postgres and MongoDB, and some machine-specific protocols. And so it's very, very big place to do all kinds of stuff.
[00:07:47] Kevin Yank: There would be quite a contrast between your, your work during the day, on angular and some of your, side project work in Elm. I often talk to people who are familiar with React and who have moved to Elm or are exploring Elm. And there the difference is, mostly syntax. A lot of the concepts are the same and there's a lot more strictness and guarantees that you get from Elm.
[00:08:09] But ultimately the programming model is quite similar, where you are, you know, controlling a state that gets rendered to a virtual DOM in like a unidirectional data flow. And it seems to me that Angular kind of has gone the other way and accepted the other side of all of those tradeoffs. And it gets good things from that and it's gets bad— bad things from that.
[00:08:33] But, it would be more of a jump back and forth between the world of Angular in the world of Elm. Is that something you feel or is that, am I overstating that?
[00:08:42] Kolja Lampe: So we actually started to use NGXS, which is state management for Angular. So I'm using some learnings from functional programming in general.
[00:08:55] Kevin Yank: Right. So I'm not familiar with that package, but it sounds like it sort of adapts the Angular environment to that sort of functional stateless approach.
[00:09:04] Kolja Lampe: You just get a state and you get to update it and it's not as nice as it's in Elm, if you ask me. It's actually kind of weird in some regards. Like you have multiple stores, and stores kind of exist when they shouldn't exist. The debugger tells you. But yeah, it works for the most part.
[00:09:29] Kevin Yank: So how did Elm end up in the mix for you? Are you just the kind of person who's like really into open source and so you end up exploring a lot of things as they come up, and Elm just crossed your radar for that reason? Or were you out looking for something like Elm.
[00:09:42] Kolja Lampe: I don't actually remember how it happened. Like, I've been testing many languages. I had the phase where I was jumping from language to language, I guess, just to learn stuff. Maybe because I was bored at my last job. But I'm pretty much the language guy in the new company.
[00:09:59] Kevin Yank: I've spoken about, that guy at Culture Amp before, his name is Marcos and he is the reason we are using Elm, but he is the engineer who when we hired him, he was like, “Oh, I've used this and this and this and this before. And I like these ones and these ones I like less. And Elm is something that I would really love to use here.
[00:10:19] “What do you think?” So do you collect languages for fun?
[00:10:23] Kolja Lampe: It's probably not that bad, I guess, but I might be denying it. Like, I did like a small thing I published on GitHub at some time, just to test out of the waters, I guess. But I actually met Evan at Google in 2015 I think.
[00:10:41] Kevin Yank: Were you working at Google at the time?
[00:10:43] Kolja Lampe: No, we were invited to Google Summer of Code Mentor Summit.
[00:10:48] That was the one year where, I saw him, and, it was kind of weird, like, I somehow knew who he was, and he obviously did not know who I was, so I tried to speak to him and it was actually fun. But yeah, that might have something to do with be getting involved.
[00:11:08] I really liked how he thinks about stuff.
[00:11:11] Kevin Yank: You are the author of the Elm language server and, a constellation of related software products around it, from the Visual Studio Code extension for Elm, the new current Elm language support package for that editor. I understand in order to write the language server, you had to write this other thing, which is, the tree sitter.
[00:11:34] Where did that effort start? Because it's not everyone who is a hobbyist in a language, who decides the thing I want to contribute is, the tooling for editors.
[00:11:47] Kolja Lampe: Yeah, I kind of wanted to always do tooling. I'm not sure if it's always, but coming from the other project I'm involved with, we did start to do some tooling, and I really enjoyed it. And so when people were looking at how to do stuff in Elm and get better integration, there were some people that actually started out, but it didn't go as fast as I wanted it to go, basically. They took the compiler and forked it so they could use the AST and, the problem about Elm's AST is that it's pretty strict. So what we want in an IDE is you need to be able to figure out stuff, even if your code is broken.
[00:12:39] Kevin Yank: You've got to half-written function. You still need to be able to provide syntax highlighting and auto-completion on, what is code that the compiler would say, this is nonsense.
[00:12:51] Kolja Lampe: Exactly. And while Evan said it would totally be possible to figure that out, I'm not sure when he would get around to that, like, and it's probably something the compiler might not even-- should not even be doing, I guess. There was a Haskell implementation, but I don't understand Haskell. So I was like waiting for the repo to update and it did not update. And I think the people that were working on it just got lost in other things, so I started to start it again doing a TypeScript implementation, I think December two years ago.
[00:13:31] Kevin Yank: The idea of a language server in the first place, I'm interested in. I had heard the term “language server” a couple of times in relation to TypeScript itself.
[00:13:42] Ruby has a language server that, when I'm working on Ruby files, it mentions that it is talking to something called the Ruby language server. Can you talk about a language server in the abstract and why that seemed like a good thing for Elm to have at the time?
[00:13:58] Kolja Lampe: Well basically, you get multiple editors for free. We can run implementation for the server and just get, multiple clients working for free. Like Vim, now, does completions; Emacs should be able to get completions; Sublime is working; VSCode is obviously working. That's the nice part about it.
[00:14:21] Kevin Yank: So you have relatively thin editor-specific clients for this persistent process that sits running and provides kind of the brain of an editor experience.
[00:14:33] Kolja Lampe: Most clients are just going to connect to the server. They have some settings that you can do and the other stuff is all stock. There are some other nice things that are not in the language server protocol yet that you could do-- which could do on top in the client, but, it's already too much for me to maintain the tree siter, the language server and the VS Code client.
[00:15:02] So I'm not going to implement any more and I'm actually trying to focus on the server. So everybody gets most features for free and not write as many client-specific code.
[00:15:16] Kevin Yank: So you mentioned the language server protocol, there. What I've seen in the code leads me to this understanding and tell me if this is right, is that, editors that provide, standard language server support mean that you can really just-- To write something like the Elm extension for VS Code,
[00:15:36] you basically say, “Hey VS Code. This is a language supported by this language server and here's how you connect to it, and you know what to do with the language server so just, you know, consume its features.” Is this, you know, growing family of editors well supported by, like, this built-in support for language servers?
[00:15:57] Kolja Lampe: Well, it's mostly true for VS Code cause it's maintained by Microsoft. And, like if you were doing changes to the language of a protocol itself, they are saying that you need an example implementation. And most of those example implementations are done in the Node connector to a language server, basically.
[00:16:22] So that's how most stuff ends up in that one.
[00:16:27] We want to attract new developers to use Elm and there's basically no tooling, it's not gonna work, I think.
[00:16:36] Kevin Yank: What was your initial ambition for this thing? Because Visual Studio Code already had an Elm language extension that as far as it went, it was working okay. And you were starting over, I imagine there was, a certain feature or a certain point you wanted to get to where you were like: That.
[00:16:57] That is the thing that was not possible in the old, language extension. And we have now succeeded in our mission for this project.
[00:17:07] Can you remember what that was?
[00:17:09] Kolja Lampe: I don't think it was like that, for me. I feel like it is like the old implementation, which was pretty good at one point, pretty much fell into disrepair when I think when 0.19 came out, like some stuff was working, but it felt not as nice to me. Like most stuff did-- like, completions were pretty much nowhere to be found as far as I can remember.
[00:17:36] But some people who were still reporting that completions are working. I think some people would ask, they're saying that Elm Oracle is working for them, which is weird to me, I guess. But the latest author of the old implementation, moved on to do open source for, like, as a day job, and he's now working on the F# implementation of the language server,
[00:18:05] which is Ionide and which is pretty, pretty good. If you have to look at F# code, I highly recommend Ionide.
[00:18:14] Kevin Yank: And so if the old maintainer of the old extension had moved on to building language servers, that was a pretty good sign that that was a reasonable direction to take for Elm as well.
[00:18:26] Kolja Lampe: Yeah. But I did not know that at the time.
[00:18:29] Kevin Yank: So this is what fascinates me is like how much of an unknown was this project for you when you started on it? When I wanted to, to contribute to it, I was kind of like, “Well, today is the day I'm going to have to find out what a language server is.” And I went to the Microsoft language server specification website and started reading, and it was much bigger than I expected.
[00:18:52] were you in a similar boat with it?
[00:18:54] Kolja Lampe: Yeah, absolutely. Like, it's a big spec. But to be honest, you only need the parts, there that you are going to implement right now. Getting the server to run is the main thing at the start, but I think that's a good starting-- like a template for it to get started,
[00:19:11] where you can get a basic server running, or generated basically, and get it running and see that they can talk to each other and then you can just start to implement stuff.
[00:19:21] Kevin Yank: Was that the reason you ended up building it in TypeScript, because there was a strong template to start from?
[00:19:29] Kolja Lampe: Not the template, but that I could be sure that the-- like we're using a package to-- like a Node package that I mentioned before, we're using to talk to the language server basically. But the thing for me really was that I could be sure that they are going to maintain it and keep it up to date. Like I have been thinking about learning Haskell, and doing it that way, but I would have needed to have the same library, I guess, and I don't think I would have been able to pull that off in Haskell quite as good.
[00:20:13] Kevin Yank: Yeah. I think back to like the elmjutsu package for Atom, having been written in Elm itself, that always seemed to me like an amazing magic trick, but if that wasn't what you were setting out to prove was possible, it might not be the easiest path to success. How long were you working at it before you had something that you felt like was a release that people could use?
[00:20:37] Kolja Lampe: That's a good question. Like, I think I needed about six months to get Tree Sitter to work. As good as what's working at that point, I guess.
[00:20:46] Kevin Yank: So let's talk about Tree Sitter. What problem does it solve? What led you to that problem and what led you to think, okay, this is a separate enough problem that I'm going to implement this as a separate package first.
[00:20:59] Kolja Lampe: Tree Sitter itself is the library from a guy that's working for GitHub. It's, I think it's even owned by GitHub right now. So it aims to be general enough to, parse different programming languages. So you write a definition file, basically, and it will figure it out, where your nodes are, create an AST and be fast about it.
[00:21:24] Be robust so that it can even parse stuff where errors are in your file. And it's doing some different tricks to achieve that under the hood. But the next thing is like, it's completely dependency free. It-- When we started it was implemented in C, now it's done completely in Rust and it's just growing and growing.
[00:21:48] Like they started to add, like coloring some time ago and just yesterday they merged a pull request where you basically get something like ctags where you can, make the language server aware of-- no, not the language server, but tree sitter aware of how references are working, so they're actually using it on GitHub.
[00:22:13] Right now for the data science teams. And if you've ever used the JavaScript or TypeScript, functions in GitHub where you jump to a definition or find references, that's all implemented on the back of Tree Sitter.
[00:22:27] Kevin Yank: Yeah. Wow. The tree it's sitting on, I think I made this mistake when I first heard about Tree Sitter of assuming that the tree it's sitting on is the directory structure of your project. And it's, it's something that is spotting changes to files and letting you know when you need to refresh your model of the contents of that file.
[00:22:46] But it's actually the, tree of the syntax. It's the abstract syntax tree inside of a given file that it is sitting and observing and notifying you of changes to. Is that right?
[00:22:58] Kolja Lampe: Well, it's not really notifying you about anything. Like it's just, you show it input, basically-- you give it a file and you-- no, you're not giving it a file, but you're giving it the content of your
[00:23:13] file and it will parse those into an AST. You need to say “this is an Elm file”, obviously, like use the Elm parser for it.
[00:23:23] It can also have nested programming languages. But in our case we say, "Hey, this is Elm, and please parse this as Elm”, and it tries to figure out what the AST is, and the next time that we want to parse it, Tree Sitter is optimized to parse stuff on typing basically, so it should be fast enough to, reparse whenever you add something to the buffer or remove something from the buffer.
[00:23:54] So, we can actually do an update of a tree. It's really, really so fast. We're parsing really every Elm file I can basically get my hands on with Tree Sitter.
[00:24:06] So I'm downloading the package index for Elm, and I also handpicked projects. And I think it's 10 thousand files and just, okay, 45 seconds or something like that. Most of the time is spent on downloading stuff from GitHub.
[00:24:24] Kevin Yank: The actual work happens in the blink of an eye.
[00:24:26] Kolja Lampe: And that's quite amazing, actually. Like, one of the problems I had was like when I went downloading every package that has ever been there for 0.19, some people like to have broken, broken files, and they're like-- Either they are not updating their tests,
[00:24:46] I'm not sure why that is, or maybe some old 0.18 examples. We fail to parse those obviously, but like we're parsing 98% of the files I'm giving it. And parsing, is this case it just means that some nodes don't work, like most of the files will still work.
[00:25:07] Kevin Yank: So how did, how did you come to Tree Sitter? Like is that a, is that a tool that is in common use by other language servers now and that's how you get high-performance parsing, or was it a case of you knew you were going to need some parsing at the heart of this language server and
[00:25:24] Tree Sitter happened to fit the bill really well?
[00:25:28] Kolja Lampe: Yeah, the latter. I knew I needed to do something with the syntax of the files to be able to understand it and to walk the trees, basically, but I have never done an AST implementation. And so I knew we would need it in some way, and it being robust, like, it needs to absolutely parse files that are not complete, and figuring all this stuff out would have taken me probably a year I guess.
[00:26:00] Kevin Yank: It's already available as a NPM package that you can pull into your language server and, you know, it installs the native version of it for your platform automatically. It must've seemed like a miracle when you found that and realized how many problems it solved for you.
[00:26:17] Kolja Lampe: We started out with including it normally, but now we're using the, WebAssembly version, so that we can basically sidestep having your PC to download a binary
[00:26:28] for each platform individually-- like a Mac needs something different than Windows will need and stuff,
[00:26:34] so with WebAssembly we can just include the WebAssembly and every platform should be happy.
[00:26:40] Kevin Yank: Yeah. Wow. I'm trying to put myself in your shoes, here, working through this of like, “Okay, I'm, I'm going to build a language server. I guess I'm going to have to start building a parser. Oh wait, there is this thing, this Tree Sitter that is a framework for parsers and I can just contribute the Elm support to that.”
[00:26:59] You were several layers deep at this point. Imagining myself in your shoes and I'm wondering if I would ever get back to the top of the stack and actually deliver Elm language support for an editor. Or if I would end up, you know, stuck at contributing to one of these inner layers of the thing.
[00:27:16] Kolja Lampe: I think I just enjoyed writing the parser because I've never done any AST parsing, and it was kind of fun, like I was inspired by the trees that the IntelliJ plugin is using. I started to, read the Kotlin files they use to parse and tried to understand that too. And so the trees were pretty similar in the beginning, as far as I can tell.
[00:27:44] But yeah, it was nice task and something I could do on the side while watching TV, I guess.
[00:27:53] Kevin Yank: I guess at a certain point you found yourself with a working Elm parser. You were then able to start work on supporting one by one the features of the language server protocol.
[00:28:04] Kolja Lampe: Yeah, it's exactly like, I was aware that there was also a Ruby implementation that was using Tree Sitter.
[00:28:11] Kevin Yank: So it was another language server that had Tree Sitter in its core.
[00:28:15] Kolja Lampe: Yeah. And I'm not even sure if that's still alive or going, but it's the only one I've seen so far. If you start your editor, we start, like getting all your Elm files and parsing them.
[00:28:29] And like, we have a forest full of trees, which are not connected, right? So we need to figure out which tree is referencing which other tree, and basically build another layer on top of that. Otherwise we won't be able to, like, go to files or find references. As I mentioned earlier, this part might die whenever we implement the new features in Tree Sitter, so, we could move it deeper into the stack to Tree Sitter,
[00:29:02] which we absolutely should do because--
[00:29:04] Kevin Yank: It's so much faster.
[00:29:05] Kolja Lampe: Good point. It's going to be faster. Not that I'm not sure if that has been a problem, but we can probably get, GitHub to include the Tree Sitter on their website for Elm.
[00:29:17] Kevin Yank: So the embedded editor for Elm in GitHub could then be reference aware, for example.
[00:29:23] Kolja Lampe: Yes, absolutely, like, just like for JavaScript. And we could have done that, but they were saying like-- The project is named Semantic, I think, from GitHub. And you could, like, do a pull request and add Elm. But, they wanted to have like so many tests and stuff, but they're trying to make it easier to do this.
[00:29:47] And I think there's effort going into it and they really want it to happen. Like, it's only going to be a good thing for them right.
[00:29:57] Kevin Yank: I'd like to, go off on a tangent for a second. Something that you mentioned a little while ago was that you were involved in this Google Summer of Code project and, and, you know, got to meet Evan in the process. I'm curious to hear, kind of, what has been your experience of the Elm community?
[00:30:17] Like what part of the world are you in, for example? Do you go to any meetups? Have you been to any conferences? How have you joined the Elm community apart from being the maintainer of this software package.
[00:30:29] Kolja Lampe: So I'm in Germany. I have been like, I've been with a friend in Berlin, some like last year, I think, and I just popped into the german Slack room, I think, and ask if there's anybody in Berlin that wants to meet up. And so I met up with @unsoundscapes.
[00:30:49] Kevin Yank: He is on my list of people I really want to get on this show.
[00:30:52] Kolja Lampe: Yeah, it's-- Awesome dude. And some other guys like, was it Erkal?
[00:30:56] Yeah. We just hung out a bunch. And we were like, normally we go to America, for Google Summer of Code Summit,
[00:31:06] but last year it was the first time in Germany and Munich, and as it might happen, like, there was an Elm meet up the first night
[00:31:17] I was in town, so I grabbed my friend from the States. He came along and we heard a talk from Christophe, I think. We don't have any meetup in my city, but I'm just going to the, like, all the functional guys here around seem to go to this F# meetup that's done by a good friend and,
[00:31:44] well, we just enjoyed to talk about functional stuff, I think.
[00:31:47] Kevin Yank: That reminds me of here in Melbourne, the Ruby meetup, for example, has very much morphed into a social group for people who like programming languages, and were led to their current programming language via Ruby at some point in the past. I would say that on a given night, only about half the people there,
[00:32:10] are actively working in Ruby to this day.
[00:32:13] Kolja Lampe: I actually think that the thing that's bound to happen to most communities, I think.
[00:32:19] Kevin Yank: Well, I suppose that's how you know it's a good community when it outlives or grows beyond the specific language that started it.
[00:32:28] Kolja Lampe: Yeah, absolutely. Another thing I was thinking before this pandemic happened was like, I've done an Elm talk at my company, like, where we do meetups sometimes, and I did a meetup to talk about Elm to people that never heard about it, basically. But so I basically have a talk lying around, which I've only done once.
[00:32:54] So I was thinking to just fly to other communities all over Germany and say, “Hey, can I talk about Elm in your meetup?” and just drive there, like, have a good time.
[00:33:07] But yeah, that's not going to happen very soon, I guess.
[00:33:13] Kevin Yank: Yeah. Well, one day all of our doors will open again and we can get on the road.
[00:33:18] Kolja Lampe: Well, I guess we should just get started doing online meetups, like we've, like I said, F# community started to do this kind of cool online meetups and hangouts, and that's been pretty cool so far.
[00:33:32] Kevin Yank: So coming back to the language server, you put out an initial release at some point, and how were you feeling about the project at that time? What were you proud of? What were you still hoping, or still very much eager to add?
[00:33:46] Kolja Lampe: Oh, there's so many things that could still be done. Like, as soon as you add something that's awesome, you still think like we can maybe do it better this way, or maybe add this. Like one thing that's always been on the wishlist is like, type inference,
[00:34:06] which I actually started to work on, but I'm not going to give any ETA because that's just going to hurt everybody.
[00:34:14] Kevin Yank: Type inference, meaning, on any given expression in the editor, you can ask the language server, “What is the type of this?”
[00:34:23] Kolja Lampe: Yes, which will be most interesting for functions, I guess, but it's going to be good to learn stuff and figure out what the language server thinks this is. And hopefully reflect what Elm thinks it is. Like, the times I skipped type annotations always came back to bite me, in my experience.
[00:34:47] Like, if you put a type annotation above it and you figure out that something you thought was something completely different is actually not
[00:34:56] the type you get at this point.
[00:34:58] Kevin Yank: Well, yeah, even just writing my type annotations for me. That would probably be my, my biggest wishlist item. And it's something that we used to be able to do by harvesting the warning messages from the Elm compiler, but Elm 0.19 doesn't have those warning messages anymore, so we have to come up with those types ourselves these days.
[00:35:20] Kolja Lampe: Tree Sitter is nice and working pretty much, as far as I'm concerned, but we can still improve it. Like, I've mostly only done testing on happy paths for now, and we can probably improve how it fails. Like if you have incomplete code, is it really taking the optimal failure state, I guess?
[00:35:45] Some nodes might get lost, even if it figures out just to ignore those that leaf that you have, maybe it's changed some other leaf to accommodate for that.
[00:35:58] Kevin Yank: So I can imagine you would want to build up at some point a library of, like, broken Elm files and test that they break in the most desirable way.
[00:36:08] Kolja Lampe: Which is, also pretty easy to set up, but I haven't taken that time. Like, building up that library and the tests is pretty easy. But getting the Tree Sitter to actually break like that, if that makes sense, is maybe not as easy for me.
[00:36:27] Kevin Yank: And is that expertise something that you, you want to develop? Is like that the way you want to spend your time on this project, given all the other things you could be doing, I suppose?
[00:36:35] Kolja Lampe: Well, not at the moment.
[00:36:37] I really think that, type inference would be the most helpful thing right now, not just for annotating your functions, but it also will help the compiler-- not the compiler, but it will help the server to figure out, the types, basically, and help you maybe in an anonymous function to know what this type is.
[00:37:05] Kevin Yank: Yeah. So yeah, you're writing an anonymous function. It takes an argument and then you want to provide completion for the fields of that argument, if it is a record, for example. That sort of stuff. So that's your, that's your white whale at the moment. That's the thing. That's the big feature you would like to head towards. You're not prepared to give an estimate on it yet…
[00:37:27] Kolja Lampe: The other thing that I'm not sure how and when we will figure it out is, at the moment, we are only indexing stuff on startup, basically. Like, all your files get indexed and you can then edit.
[00:37:43] But normally if you install, you go-- you work in your Elm files and you add a dependency and it gets installed into your, cache. Basically, like there's a folder on your hard disk somewhere that's not in your project, which is kind of where my problem starts. And there are files that are getting added.
[00:38:07] And so do I set up a watcher for that, location that's not anywhere near your project? Should I just do that, or-- I really don't want to, jump into locations where the user doesn't want me to look. Like, is that going to be spooky for people?
[00:38:28] Kevin Yank: Yeah. Wait, how did you know about that file?
[00:38:31] Kolja Lampe: Yeah. I can see some people not liking that. That's the thing, like with NPM or, yeah, with Node you always get a node_modules for in your project. And it's pretty obvious what to do with that. But in Elm, it's all centralized, basically, which has its pros obviously. But I'm struggling a bit with this decision and how to nicely do that.
[00:38:59] Like the best thing would even be to have, in installer, have a callback and just say, “Hey, we installed, somehow, and you need to rescan.”
[00:39:10] Kevin Yank: Yeah, I mean I could imagine you providing the feature in the editor extension to install a new package and because your extension initiated that install, you could then, you could then parse the files once it was complete, but that would, that would still leave the problem of
[00:39:26] what if the user install the package a different way?
[00:39:29] Kolja Lampe: It's going to be recommended everywhere to do it via the terminal. So there's no easy way of getting out of this, I think.
[00:39:39] Kevin Yank: Oh, well, that's what makes it a fun challenge, right?
[00:39:42] Kolja Lampe: Yeah. That's that part is correct, yeah.
[00:39:46] Kevin Yank: I know as we record this, you just recently did a pretty major feature release for, the Elm language server and the VS Code, extension along with it. And you mentioned in the release that this was largely a collection of contributions from someone else on the project. Can you talk about some of the collaborations, some of the contributions you've had?
[00:40:10] If you, if you want to sing my praises as a onetime contributor, you would be very welcome to…
[00:40:16] Kolja Lampe: Yeah. I actually love to have contributions, like it's a way too big place to be owning mself, basically. Like, there are so many awesome people that have been involved, and are involved, like @jmbockhorst, I think he's named. He's been doing that pretty much, so much over the last weeks, which is mostly most of those feature releases
[00:40:44] you're seeing right now. Ahere was, Andy, which is doing anything with Vim basically. So he's maintaining the vim client as-- Well, there's not really a Vim client, but he's maintaining how to configure stuff and he's helping with documentations and he's just a great guy. He actually sent me a package to my company unannounced, and I got stickers for Elm-- for my language server, basically, with the logo.
[00:41:17] And my-- our front office was like, do you know anybody by the name of… and I'm like, no. And then I saw the nickname on it and then it clicked. But, so those kinds of things are really awesome. And your contribution was also very much appreciated because it was not the easiest thing to get fixed, let me say that much.
[00:41:46] Kevin Yank: I was very happy of how willing you were for me to kind of turn the code structure of the language server inside out in a couple of places. I think coming in, I-- you know, like any other sort of cocky new contributor I thought, “Oh, this is going to be an easy bug to fix; it'll be like a five line change.”
[00:42:11] And, and I opened the issue and I went, “Do you mind if I have a crack at tackling this?” And you said, “This is something I have tried to fix before, and it turned into a much bigger job than I thought, but if you think you can fix it, go ahead.” And I was like, “Great! I'm going to impress you with just how easy a fix it was.”
[00:42:29] And then like three weeks later, I was refactoring half of the files in the language server, and I knew exactly what you had meant. But I appreciated your willingness, to go along with me, to review my changes, to tell me what I had missed when I had an initial version ready for testing.
[00:42:45] Yeah, I enjoyed that partnership a lot.
[00:42:48] Kolja Lampe: Yeah, I think it's always fun to do stuff like this, and I think as far as I can tell, it worked out pretty good.
[00:42:56] Kevin Yank: For those listening, wondering what this was, it was, because, at Culture Amp, we have a large number of Elm apps in a single Git repository that we opened in our editor. And, in many cases, the same library modules are being used by more than one of those, Elm apps.
[00:43:14] So I think at the time, the Elm language server was built on the assumption that you would point it at a single elm.json file and it would support the editor working across that context. But in our case, we wanted a single instance of the editor working across something like 17 elm.json files.
[00:43:35] And if you open any old Elm file, it should be able to know which elm.json file is responsible for it and what its dependencies are. And, you know, clearly we were pushing the boundaries and that is-- I was not at all surprised that that was not a supported scenario in a brand new language server,
[00:43:55] but yeah now that it's working, it's working surprisingly well.
[00:43:59] Kolja Lampe: That's good to hear. Like, it's interesting that we get-- like, we have multiple levels of workspaces. VS Code can have workspaces, Elm can have something like workspaces, I guess, but then again, you can nest them, I think? It's getting pretty, pretty confusing at some point.
[00:44:21] Kevin Yank: So an elm.json file and all of the Elm source files that go with it is referred to as an Elm workspace in the code base. But like you said, Visual Studio Code has the idea of a workspace and you can have more than one of them open. And do you have one language server per VS Code workspace, or do you have a single language server for all of the workspaces?
[00:44:45] And if you have multiple workspaces with multiple elm.json files inside of them, just what workspace are you talking about when you say “workspace”? It's very hard to keep straight.
[00:44:56] Kolja Lampe: On the other hand, if you've solved that, it's pretty much going to be solved for everything.
[00:45:02] Kevin Yank: Yeah. Well, fingers crossed.
[00:45:04] Kolja Lampe: Fingers crossed.
[00:45:06] Kevin Yank: From an observer's perspective, it still feels like the language server is somewhat young in that, you know, on any given week, if you're looking at the Slack channel, there's often someone coming in and going, “Hey, it's not working for my scenario and it's crashing in a weird way. Can you help me?” And usually they get the help they need and, it ends up working. But it seems like there is still like a number of edge cases out there or strange configurations, or in my case, like really weird directory structures that weren't anticipated. It feels like the project is still somewhat working its way through those and shaking those, kinks out of it.
[00:45:49] Does it feel like that to you?
[00:45:51] Kolja Lampe: It might sound worse than it actually is. Like, I'm really, really glad if people come and report stuff, because I can only test on the codebases I have, and it might really depend on your code base of it's going to crash or your setup and there's so much stuff involved that
[00:46:11] can affect it, I guess. And I can't really test everything, like. But now that you mention it that there's like, if I had all the time in the universe, I would start writing more tests, obviously. Another big thing that we could do is not depend on elm-analyse and not depend on the compiler.
[00:46:32] Basically write our own thing based on Tree Sitter and figure out, like, not use elm-analyse, but figure out the stuff that elm-analyse gives you, via Tree Sitter.
[00:46:48] Kevin Yank: Are you saying that's where any fragility seems to, occur is at that, interface between, the language server and the external tools that it is calling, elm-make, elm-analyse and elm-format, that at that moment of going, “Hey, I'm going to invoke another process and interpret the shell output that comes out of it,” that, that tends to be the place where unexpected things happen?
[00:47:17] Kolja Lampe: Yeah. Like with a Elm 0.19 we had cases where the --report=json function from Elm did not report valid JSON, and so we couldn't parse it, and just crashed. Stuff like that you can probably handle it with integrations, but it seems like somebody has found another edge case where this might be happening like yesterday.
[00:47:43] And I'm not sure how to get out of that without, when we rely also on those tools and for some cases, I'm not sure how much time is spent to, like on the Elm compiler, there's probably not going to be a problem in the end, but like, I'm not sure how much time is spent to maintain stuff like elm-analyse, and how long it's going to be in a state that we can work with it. We actually have to maintain a fork of elm-analyse right now because we couldn't upstream stuff we have at the moment.
[00:48:19] So yeah, it's probably going to happen at one point, but, I think there's actually an open merge request, but it hasn't really moved. And as far as I could tell, it's not the fastest tool we've used.
[00:48:34] Like, I've done some performance stuff some weeks ago where we were loading the Tailwind Elm File, which is just like 1,500 functions, something like that, and it took like forever. And one of the reasons it took forever, it was just that, elm-analyse did not
[00:48:55] like that file really. We had our own problems, which increased the time, but after I fixed that and disabled elm-analyse, it worked really good. So it was a combination of both. elm-format is at the moment you get a whole new file every time.
[00:49:14] Like, what's the language is service doing is it's asking, “Hey, is this file has to be formatted? Here's the content.” elm-format would parse the file and format it for us. Then we get the formatted file back and we'd actually diff the old file against the new one, and create a diff and apply just the diff to your buffer.
[00:49:39] So that has… that causes… well that leads to your editor not jumping up and down. Like if we replace the whole file for you, where is your cursor?
[00:49:53] Kevin Yank: So it's worth doing, but it's a fairly inefficient set of layers of parsers sending, communicating with other parsers through whole file snapshots.
[00:50:03] Kolja Lampe: Yeah. It would be much nicer to just ask the formatter to get us a diff.
[00:50:10] Kevin Yank: Yeah. Right.
[00:50:11] So I think I speak for the entire Elm community or at least everyone who is using an editor that, that has an integration with the language server, that I, I want to thank you for all the work you are putting into this. Is it fair to say you are spending more time working on the Elm language server, than you get to spend actually writing Elm?
[00:50:30] Kolja Lampe: Yeah, absolutely.
[00:50:34] Kevin Yank: That is something of an injustice, I have to say, that the person who is working hardest on the tooling gets to benefit from it the least.
[00:50:42] Kolja Lampe: But it might've actually highlight how much I'm depending on feedback from the community. Like, open up issues. Even if I'm not be able to help you this week or next week, but it's really, really important that we keep stuff reported and maybe there's some people from the community that want to help out. Like, I pretty much have three projects that I need to maintain alone just for this.
[00:51:08] Kevin Yank: So bug reports, welcome. Contributors, welcome.
[00:51:11] Thank you again for all of that work you're putting into it. I am a big fan of the language server. I have to say every time I log into the Elm Slack, the Elm language server channel is the first one I click on to go and see what's new and whether there's a new release and, and honestly,
[00:51:27] there have been days where I have chosen to spend some time on an Elm side project because there is a new feature in the language server that I want to play with,
[00:51:36] and I can think of no higher compliment for you than that: that yours is a tool that makes me want to use Elm more.
[00:51:43] Kolja Lampe: Yeah, that's, that really makes me have a big smile on my face right now. Like, that's a big compliment. But I really want to thank you too, for doing Elm Town.
[00:51:52] Kevin Yank: Really happy to hear that you're a happy listener. I agree that it is a really good back catalog and that's not only my doing, that's Murphy's doing before me as well. I am only hosting this show because I was such a big fan of it beforehand.
[00:52:04] Kolja Lampe: But that's a good way to work it, right? Maybe someday you will have to move on, but maybe we can find another one that's seeing it just like this.
[00:52:15] Kevin Yank: That sounds good to me.
[00:52:17] And I think that's a great place to leave it. Thank you so much for the chat Kolja. I really enjoyed it.
[00:52:22] Kolja Lampe: Thanks for inviting me.
[00:52:24] Kevin Yank: And thank you, listener for joining us once again in Elm Town. like I said, we've got a lot of episodes in the hopper, and you can look forward to more Elm Town coming to you soon.
[00:52:33] Bye for now.