Minutes of the 15th meeting of the Scala Center, Q4 2019
Minutes are archived on the Scala Center website.
Summary
The following agenda was distributed to attendees: agenda.
Darja Jovanovic is the Center’s new executive director. Sébastien Doeraene is now technical director.
VirtusLab is now a full voting member of the board, represented by Krzysztof Romanowski. IBM has left the board.
Staffing changes in engineering: Jorge Vicente Cantero left the Center in November. Offers have been made to three new engineers.
Scala Center activities for the past quarter focused on MOOCs, Scala 3 interoperability with Scala 2, SemanticDB support for Scala 3 (to support Metals), Metals and Bloop, Scala.js, SIP (Scala Improvement Process) meetings, the contributors summit at ScalaSphere, Scala Days 2020, recruiting and hiring, and community moderation.
Full details on these activities are in Sébastien’s report.
Darja presented her plans for the future of the Center, including growth in funding, board membership, and staffing, as well as work on Scala 3.
Two proposals were discussed:
- SCP-021: Zinc improvements (Lightbend)
- SCP-022: Completing SCP-011: implement JSR-45 for improved Scala debugging experience
Both proposals were voted on and accepted.
Other topics discussed included the future of Scala Native, the Scala 2 roadmap, and the future of Bloop.
Date, Time and Location
The meeting took place virtually on Monday, December 16, 2019 at 4:00pm (UTC).
Minutes were taken by Seth Tisue (secretary).
Attendees
Officers:
- Stu Hood (chairperson), Twitter
- Darja Jovanovic (executive director), EPFL
- Sébastien Doeraene (technical director), EPFL
- Martin Odersky (technical advisor), EPFL
- Seth Tisue (secretary), Lightbend
Board members present:
- Erik Bakker, Lunatech
- James Belsey, Morgan Stanley
- Adriaan Moors, Lightbend
- Rob Norris, community/Typelevel
- Krzysztof Romanowski, VirtusLab
- Bill Venners, community/Artima
Guests:
- Ólafur Geirsson, Twitter
- Eugene Yokota, Lightbend
Apologies:
- Thomas Gawlitza, SAP
- Oli Makhasoeva, 47 Degrees
- Jonathan Perry, Goldman Sachs
- Julien Tournay, Spotify
Proceedings
Stu introduced the meeting. He is serving as chairperson of the advisory board for a year, as part of the new plan to rotate the position annually.
Krzysztof Romanowski, head of developer tooling at VirtusLab, introduced himself. VirtusLab has been a contributing member of the Center since April; “contributing member” meaning they provide engineering rather than financial support. Previously, this only gave VirtusLab “affiliate” status, but the Center was recently able to amend its bylaws so that contributing members can be full voting members of the board. Hence, this was Krzysztof’s first time attending the meeting. For the Center, VirtusLab engineers have mainly been working on Metals.
Darja Jovanovic, who has been the Center’s communication manager for the past two years, announced that she has assumed the role of executive director. Former executive director Sébastien Doeraene has moved into the newly created role of “technical director”. She described the two roles as akin to CEO and CTO of a company. (Martin Odersky remains as technical advisor.) In an email to the board, Seb described the change as follows: “I will be more available to contribute as an engineer and as a tech lead, while Darja will take on the organisational, team management, and representation tasks.”
Activities
As the Center’s technical director, Sébastien Doeraene summarized Scala Center activities since the last meeting.
Most of Seb’s remarks were based on his detailed report on the Center’s recent activities. The following notes are a supplement to Sébastien’s report.
Offers have been made to three new engineers. The current plan is that:
- one engineer will work mostly on compiler-related tasks including Scala Native
- one engineer will work on Scalafix, Scala 3, and other tooling and Scala 3 issues, including Metals and Bloop (now that Olaf and Jorge have left)
- one engineer will work on web tools including Scaladex (“which we very much intend to bring back to life, to a good life”) and on educational materials, documentation, and the Scala beginner’s experience
Seb: “I hope that next time I can share the new engineers’ names.”
Director’s statement
Darja presented the board with her thoughts about the future of the Center under her directorship.
Darja sees her skills as: “Project and team management; Facilitation, negotiation, and conflict management; Organization and basically getting things done.”
“We started three years ago, we are into a fourth year. Congratulations everyone for surviving and going through the most difficult period for any new organization. Thank you everyone for your trust, your vote of confidence, for sticking with us. As of now, we can start a new era. We can do more.”
“We all know that Scala 3 is coming, day by day it is becoming a reality,” and the Scala Center is essential for making it happen, so “we need more support.”
Darja identified “deliverables” that she wants to achieve as director:
- Grow the Scala Center team to 15 people, including interns and students, “hopefully by September 2021.”
- Meet individually with all board members, to “strengthen the relationship between the Center and its board members, in order to coordinate our efforts in spreading Scala.”
- Lift the USD 50K donation limit for board members and add five new companies to the board by mid-2021.
- Create a “solid project list” to be able to distribute work beyond Scala Center
- Grow the community and deepening relationships through conferences, meetups, trainings, education events, and moderation training.
Some of the companies that Darja has in mind as potential new board members are large enough that more than 50K of support is reasonable goal, hence the desire to lift that limit. With more funding, in addition to hiring full-time engineers it is also possible to engage contractors (for example, Alexandre Archambault).
Discussion topic for the board: how should we determine which companies are expected to pay more than 50K, and how much more? “We need your advice.” Can we create objective requirements? Note that in order to change this, every existing board member would need to approve any changes to the contract terms.
Stu and Rob asked if companies that paid more would also somehow receive more. Is there any danger that the board would be dominated by a few large donors? “No”, said Darja, in short, though the Center is open to suggestions on whether there are additional benefits that would make sense to provide. Seb: “The best we can probably provide as an additional benefit is to be listed in a different category on the website”, e.g. “platinum”. Certainly additional funds means more engineers can be hired, so companies will now that the dollars will be “impactful”, but there wouldn’t be any “direct control” over how the money is spent. Martin: “The purpose of the proposal is to have an agreement that lets companies pay more, but under the same set of regulations” we already have – at least, in this round of changes.
There was discussion about whether it might also, as a separate and additional change, be possible for companies to directly fund certain tasks. Stu said this seemed like “a departure”, and that an alternative would be for a company to fund a contractor not formally associated with the Center to do the work. But there is a clear benefit to having the Center involved, or even to ultimately “own” a change, because of the Center’s expertise and connections. The question was left open, but Darja promised to follow up.
Financial report
This was presented by Seb.
IBM has exited, leaving eight members.
At the moment, the Center has only one single full-time engineer, Jamie. (Of course, that doesn’t include the engineering work done by e.g. Seb and Julien.) But soon, there will be four full-timers again.
There is no news this time around about funds from MOOCs. That news comes only twice a year.
Community feedback
Bill: “I don’t have anything this quarter. People are happy.”
Rob asked – on behalf of Eric Richardson and others in the community – about the status of Scala Native, now that Denys Shabalin has left EPFL. What are the Center’s plans there?
Seb: “One of the new people we are hiring has a strong compilers background and Scala Native will be his main project. So we do have plans to support Scala Native. At the very least, to bring it up to date with Scala 2.13. At the moment it is stuck on Scala 2.11, so we are two major versions behind.”
Seb said that he personally plans to work on Scala Native as well, since he now has fewer administrative responsibilities, and because “Scala.js is basically done – there is very little to keep doing.”
Proposals
Both proposals were presented and discussed before either proposal was voted on.
SCP-021: Zinc improvements
Proposed by Adriaan Moors on behalf of Lightbend. Eugene Yokota, also of Lightbend, was the main author of the proposal and joined the meeting as a guest to present the proposal to the board.
“We propose enhancements to the core of the compilation toolchain, Zinc. Zinc provides Scala compilation services, including incremental compilation, to build tools and IDEs.” See the proposal text for details.
See also the September 2019 minutes, when a draft of the proposal received a round of initial discussion.
Eugene: “Zinc’s incremental compilation is how, directly or indirectly, the Scala language is experienced, whether it’s through Metals, or build servers, or the many build tools”, not only sbt (where Zinc originated). But Zinc has “lagged”, compared to the evolution of the Scala language and compiler. Language changes that come out of the SIP process have implications for Zinc’s ability to do correct incremental compilation, but Zinc hasn’t always kept up. “At Lightbend, we’re refocusing energy into improving incremental compilation.”
The proposal is “two-pronged”: to improve Zinc’s performance and correctness for Scala 2, but also to ensure that those gains are preserved as we move to Scala 3. Eugene said his “ask” is for the Center to participate in the design process so that Zinc can continue to satisfy all of these goals, and to participate in the implementation work, too.
In the past, incremental compilation has been off to the side and always taken into account. “When people send a proposal to change the language, how it will affect incremental compilation should be considered.” Incremental compilation should be a “first class citizen”.
As for correctness, when Zinc has failed to keep pace with language changes, it often results in “under-compilation”, which means recompiling too few files when something has changed. These bugs need to be identified and addressed; some are low-hanging fruit, others may be harder.
Zinc and the compiler bridge also need improvements in the area of interoperability between Scala 2 and Scala 3, in both directions.
There’s also an “emerging trend” towards “larger-scale compilation”: at Twitter, at Morgan Stanley, and elsewhere. Zinc may need changes to serve as a basis for those efforts. But even if those efforts take a different direction instead, Stu said Twitter will still use and value Zinc “for many years”.
Rob voiced support for Zinc getting continued attention: “ensuring continuity with tooling” is “super important”.
Seth asked what the Scala Center’s role in this work would be, versus Lightbend’s role. Eugene: “It depends”, that will partly come out of discussions; but at least for the time being, we can expect Lightbend to remain more focused on Scala 2 and the Center more on Scala 3. So the Center will be “bridging the 2 effort into the 3 world”.
Stu asked Seb for his input. Seb said that the current version of the proposal has now addressed some of the concerns he had with the first version. “Overall I think the proposal is a good thing. If accepted, we would probably task Jamie with it, since he’s already working in the boundary between 2 and 3. In the design, I would involve myself as well. I think we have the right people to do this.”
Voting: The proposal is accepted by unanimous vote (by the six present voting members).
SCP-022: Completing SCP-011: implement JSR-45 for improved Scala debugging experience
Proposed by Stu Hood on behalf of Twitter. Ólafur Geirsson, formerly a Scala Center engineer for three years, now at Twitter, wrote the proposal and attended the meeting to present it.
“This is a proposal to prioritize the completion of SCP-011. To reduce the scope of SCP-011, this proposal suggests to focus only contributing JSR-45 support to the Scala 2 compiler.” See the proposal text, and the text of SCP-011, for details.
Olaf: “No one really ended up working on this project” after it was initially proposed, but “I believe that right now is a good time to focus on it again”, because “we’re now adding debugging support to Metals” and because “the Scala 3 inliner is getting a lot more powerful”, and that increases the need for JSR-45 support so you can set breakpoints inside of inline functions.
He said that the debugging support that has been added to Metals so far is “pretty rough” and there will be a “long tail” of work needed to bring it to a comparable level of polish as, for example, the IntelliJ debugger.
He also raised the subject of code coverage. The Scala 2 solution is scoverage, but that’s a “fairly heavy” compiler plugin that adds “significant compile-time overhead and runtime overhead” and won’t work on Scala 3. Olaf thinks a new solution could be bytecode-based instead and consume the same information that a debugger uses.
James expressed interest in the code coverage angle, but said that Morgan Stanley hasn’t found bytecode-based code coverage to be a satisfactory approach. Olaf allowed that “there is no guaranteed success” in the approach he’s suggesting, but he thinks it’s worth seeing if the bytecode-based approach can be done better and be more successful than before. (There was some further discussion of the technical basis for Jacoco, code coverage in IntelliJ, Kotlin’s coverage tools, and so on.)
Stu wondered if too much focus on coverage rather than debugging might be “a stretch” for this proposal’s scope.
Stu asked Seb for his input. Seb said he didn’t know enough about code coverage to comment on that aspect, but he is “definitely convinced” by the debugging support angle, especially when you consider Scala 3’s inliner. But “I’m a bit concerned” by the effort level (six engineer months). “The implementation of JSR-45 itself should be much less than that”, shouldn’t it? So we need to decide whether the proposal should involve pursuing coverage, or not. It sounds fine to Olaf to do a month or two of work to “kickstart” it, and then revisit whether further effort should be added. That initial effort “may encourage other people” to help pursue it further.
Voting: The proposal is accepted by unanimous vote (by the six present voting members).
Other business
Scala 2 roadmap
Adriaan broke the news to the board that the Lightbend team has decided that Scala 2.14 isn’t happening. This same news was announced to the public a few days later in this blog post.
Adriaan: “We’re not saying it’s certain that we’ll never do a 2.14,” but if it eventually happens, the main reason to do it would be to make additions and changes to the standard library.
Stu’s reaction: “I’m actually happy not to have a 2.14. At Twitter we were considering two completely blocking Scala upgrades before we could move to Scala 3 (2.13 first, then 2.14), so I’m excited we don’t have do that.”
And James: “This seems like the best approach. Only doing 2.13, and waiting to see how things are in 3, before iterating on building things we need to bridge the gap. I think it’s a good move.”
Adriaan: re: backporting Scala 3 features to 2, “We thought the cost/benefit wasn’t good because it wouldn’t smooth migration that much.” Stu asked: “Can we keep options open w/r/t to 3-to-2 backports to improving compatibility?”
Adriaan’s primary anwer was “We’d rather invest in Scalafix and Scalafmt”, so you could have one canonical source tree which is the 3 one, and you automatically rewrite back to 2. But on the other hand, yes, it remains possible to “backport syntax stuff and offer it under a flag” in 2.13.x, doing that in some cases isn’t entirely ruled out in advance.
Rob asked about the other direction: forward-porting tools, from 2 to 3. Seth, referring to the Lightbend team: “Yes, this roadmap change will free up some of our time for that kind of thing.” And Seb also said “yes, the Scala Center is committed to doing this, using Scalafix, and note that the Dotty compiler also does some of this itself already.”
Martin: forward-migration tools may not be as pressing a concern as you might imagine, since in general Scala 3 supports most Scala 2 syntax and semantics, including Scala 2 style implicits. Regardless, offering forward-rewriting is still valuable for users who don’t want or need to cross-compile between 2 and 3, or who are eager to start writing fully 3-style code sooner rather than later.
Martin: “I want to add one thing which I think would be really helpful. The idea is from you, Adriaan: to offer a rewrite that adds explicit type information to a Scala 2 program, to make it much more likely that it will compile the same on Scala 3. Then some of those types could be dropped afterwards, perhaps by binary search.” Olaf: “We have this at Twitter, using Scalafix. It doesn’t remove the types afterwards, but it does the adding. It was not trivial but it is working pretty well.”
Stu: It took them a long time, but Python 3 migration has some positive lessons for us. They had “phased” migration tools. First, rewrites that let you stay in 2 but get your code ready. You commit that, then you do a second phase where you make the jump.
Bloop
Adriaan: let’s discuss Bloop’s future. Bloop forked zinc; can those changes be brought back in? And what’s the future for Metals on Scala 3?
Seb: for Metals, most of our involvement is through VirtusLab, which is going great, we have no concerns. For Bloop, in the immediate future, Jorge has offered to keep maintaining it, so far, but if that doesn’t work out for some reason, we would definitely reassign someone to keep evolving and maintaining Bloop, at least as long as it’s a core component of how Metals works.
Seb, re: reconciling the Zinc forks, “We should do that at some point”, yes, and it could fall under SCP-021. Adriaan: “As we invest in Zinc, we want to only do it once.”
Conclusion
As usual, the next meeting will be in approximately three months, likely in late March, almost certainly held virtually.