Minutes of the 11th meeting of the Scala Center, Q4 2018

Minutes are archived on the Scala Center website.

Summary

The following agenda was distributed to attendees: agenda.

Spotify has joined the advisory board. Julien Tournay is their representative.

After this meeting, Lars Hupel is stepping down as community representative. A replacement has yet to be nominated.

Scala Center activities for the past quarter focused on Coursier, Almond (formerly Jupyter-Scala), Bloop and BSP, Zinc, build pipelining, 2.13 collections, Dotty, Metals, Scalameta, Scalafix, Scala.js, SIP meetings, MOOCs, Scala Days 2019, and ScalaIO.

Full details on these activities are in Sébastien’s report.

Other topics discussed at the meeting included the Center’s sponsorship terms.

Two new proposals were made:

  • SCP-019: Focusing on backwards compatibility for Scala 2.14 and Scala 3.0 (Twitter)
    • voted on, not accepted
  • SCP-020: SBT transitive dependency conflicts management improvements (Spotify)
    • voted on and accepted

SCP-20 was exceptionally accepted as a late submission from Spotify, as they were only confirmed as board members between the submission deadline and the meeting.

Date, Time and Location

The meeting took place virtually, via Google Meet, on Wednesday, December 5, 2018 at 3:30pm (GMT).

Minutes were taken by Seth Tisue (secretary).

Attendees

Board members present:

  • James Belsey, Morgan Stanley
  • Eugene Burmako and Stu Hood, Twitter
  • Lars Hupel, community/Typelevel
  • Olga Makhasoeva, 47 Degrees
  • Adriaan Moors, Lightbend
  • Jonathan Perry, Goldman Sachs
  • Frederick Reiss, IBM
  • Julien Tournay, Spotify
  • Mark Waks (a.k.a. Justin du Coeur), community/Artima, on behalf of Bill Venners

Apologies:

  • Thomas Gawlitza, SAP

Officers:

  • Sébastien Doeraene (director), EPFL
  • Jon Pretty (chairperson), Propensive
  • Martin Odersky (technical advisor), EPFL
  • Seth Tisue (secretary), Lightbend

Proceedings

As chairperson, Jon Pretty conducted the meeting.

He welcomed Spotify as the newest advisory board member, and Julien Tournay as their representative. Julien works for Spotify in Sweden.

Jon also welcomed Olga Makhasoeva as 47 Degrees’ new representative on the board.

Filling in for Bill Venners at this meeting was Mark Waks, Bill’s colleague at Artima.

Activities

As the Center’s Executive Director, Sébastien Doeraene summarized Scala Center activities since the last meeting.

Most of Sébastien’s remarks were based on his detailed report on the Center’s recent activities.

The following notes are a supplement to Sébastien’s report:

Coursier isn’t itself new, but it is a new project for the Scala Center. Alexandre Archambault, who is now a full-time engineer at the Scala Center, is Coursier’s original open-source developer. Seb highlighted Alex’s plan to add publishing support to Coursier and eventually integrate Coursier into sbt more directly as a faster replacement for Ivy. “It’s a big enabler” of productivity, “basic speeds of day-to-day tools” that Scala developers use. A new documentation site for Coursier will be released “soon”.

Martin Duhemm has left the Scala Center. The Dotty IDE work described in Seb’s report was Martin’s last project.

Seb’s report covers two different IDE projects, namely Dotty IDE and Metals. Unlike Dotty IDE, Metals aims to support Scala 2. The two projects’ features and UI “will be reconciled in the coming months,” says Seb. Metals “doesn’t have a lot of features” yet but “the features it has are really polished” and “it’s really simple to install and to import projects.” The goal is to add features gradually, “to do one thing at a time but do it really well.”

Jon asked what quality level that Metals has attained so far. Seb said that the handling of compiler diagnostics is already “perfect”. The go-to-definition feature has good test coverage, but users may still uncover unknown bugs. Importing builds, via Bloop, may fail on especially complex builds. Julien offered that Metals “worked perfectly” so far on a 20-module project he’s been trying it with, though it “takes a while” to initialize.

Jon asked about Scala Days sponsorships. Seb said that interested sponsors should get in touch with Darja at the Scala Center.

Eugene asked what’s next for build pipelining. Seb said that Jorge will continue to work on it, next steps in that work include support for incremental compilation and mixed Scala/Java codebases. “Once it’s usable for incremental compilation we’ll make a bigger fuss about it and advertise it much more widely,” said Seb.

Eugene also asked about the status of Dotty IDE. Seb returned to the question of aligning the work on Dotty IDE with the work on Metals. In many ways they are similar (for example, they are both LSP based and both integrate with VS Code and other editors). But right now the user must install “one or the other”, installing both tends to “not quite work.” These projects need to become “one concerted effort” with a single unified download, which would then operate in either Scala 2 or Dotty mode depending on what kind of project you have.

Stu asked about the status of SCP-018 (“Converging the Intermediate Representation of Scala 2.14 and Scala 3.0”), “was anyone assigned to work on that yet?” Seb said the Center is looking for “a new person we could assign this project to.” (If a new person can’t start soon, someone else might be assigned in the meantime.)

Financial report

“We got the final numbers for the MOOC income for Q1 through Q3 of 2018,” said Seb. This totaled just under 300K CHF, of which the Center’s share is 215K CHF. (The CHF/USD exchange rate is currently 1-to-1.) This is similar to past proceeds, so that part of the budget is “stable”.

With Spotify joining the board, there are now eight member companies. There may be enough budget to open an additional position, besides the one vacated by Martin Duhemm, but it’s not certain yet.

Jon asked if there was any possibility of using extra money to increase salaries, to help ensure retention of engineers. Seb said that EPFL’s rules around compensation are so strict that there’s really no leeway.

Jon also asked if the Scala Days 2019 (in Lausanne) has any financial risks or financial opportunities for the Center. Seb said that “the finances of Scala Days are a completely separate budget”. Because there is only one Scala Days in 2019, a larger hall was obtained, on the assumption that attendance will be higher than two separate conferences on either side of the Atlantic. The break-even point is 900 attendees, which is not very many more than attended Scala Days Berlin in 2018.

Proposals

Both proposals were presented and discussed before either proposal was voted on.

SCP-019: Focusing on backwards compatibility for Scala 2.14 and Scala 3.0

Proposed by Stu Hood on behalf of Twitter.

The theme of the proposal is the relative priority, during the transition of Scala 3, of backward compatibility (Scala 3’s ability to consume libraries built using Scala 2) versus forward compatibility (Scala 2’s ability to consume libraries built using Scala 3).

See the proposal text for details.

There was some discussion on the pull request for the proposal

between Martin and Stu. This resulted in some revisions to the proposal shortly before the meeting. These changes are included in the proposal text linked above, and the updated version is also what the board finally voted on.

In presenting the proposal, Stu noted that an alternative to forward compatibility already exists, namely cross-compilation. “Libraries can do what they already do,” namely cross-compile and cross-publish, in this case against both Scala 2 and 3, as long as they only use the common subset of the language (which encompasses nearly everything in Scala 2).

“This is not an anti-TASTy proposal”, Stu said. “This is a proposal to use whatever means necessary to allow backwards compatibility.” If Scala 3’s support for Scala 2’s pickle format is good enough to get good backward compatibility, then so be it. But if Scala 2 emitting TASTy turns out to be a better way to achieve backward compat, then so be it also. (Whereas Scala 2.14’s proposed ability to consume TASTy is less relevant to backwards compat.)

Martin commented on the proposal. He described Dotty’s already existing Scala 2 pickle support as “a solved problem.” “Every Dotty program ever written uses Scala 2 pickle support” and it’s never been a problem, so Martin hopes that the backward compatibility problem is actually already solved, but does that still need to be validated further? To more fully validate it, we need to test it with “a lot of code”. There are two ways to get that much code, one is to wait for the community to catch up and write a lot of Scala 3 code, but that takes a long time. Or you port a lot of existing Scala 2 code, but that takes effort, especially for porting macros, but also (for example) if the code hits differences in type inference. The Scala Center can help Scala 2 projects port their code when the time comes, but in the meantime there isn’t anything to do to improve binary compatibility, so Martin sees no need to do deprioritize the forward-compatibility parts of SCP-018.

Jon asked Adriaan what the impact on Lightbend’s 2.14 efforts would be. Adriaan addressed that and also commented more generally on the new proposal. “The key point is that forward compatibility is also important” for Scala 3 “because in order to identify potential challenges with Scala 3, we need to be able to link both ways. Applications might not upgrade to Scala 3 right away, but libraries may be eager to upgrade.”

Adriaan also said that SCP-018 was intentionally “pretty vague” about the exact nature of the best path forward. “It’s the task of the Scala Center” to figure out exactly what to do, depending on what engineers are available, and any other factors that arise as the work progresses. Seb agreed that this new proposal (SCP-019) “is directing, more than usual, what the Center should do and how we should do it.”

Eugene asked what barriers exist, or are expected to exist, for cross-compiling libraries against Scala 2 and Scala 3. Understanding that better could help identify the best approach to compatibility. Stu added that the Dotty community build is currently “leaf nodes”, that is, libraries with few or no dependencies. We won’t know more about compatibility until it’s expanded to cover inter-library dependencies better. And Jon observed that we need to test applications that depend on the libraries, too.

Martin says the blocker, currently, for expanding the scope of the Dotty community build is macros. “Transitive dependencies using macros are absolutely everywhere as you go up the stack. That’s why it’s so difficult. Bottom dependencies need to be adapted to use new-style macros” before we can test downstream projects. So yes “there are porting problems”, but “they are not related to binary compatibility. Macros are the problem.” But if the Scala Center can identify key macro-based libraries and help port them, that will be a “big win”.

Mark added that ScalaTest is currently being ported to Dotty, so that will be a “big, chewy test” of a widely-used library that is macro-based. He also mentioned that we need to keep in mind that the world of Scala development at companies is very different than what we see in the open-source world. For example, relatively few companies are writing their own macros, even though open-source macro-based libraries are widely used everywhere. So we’ll need to keep that in mind when gathering data on compatibility.

Martin agrees with Stu that producing a “compatibility reference” is needed, so that the community has the information it needs to cross-compile and/or cross-consume between Scala 2 and 3, but he sees producing this as primarily LAMP’s and Lightbend’s job, rather than primarily the Scala Center’s job.

Seb explained some the motivation for valuing forward compatibility, as he sees it, though “not taking sides”. He explained, “If we only support 3 depends on 2, but not the other way around, then library developers are stuck on 2 for the next 5 years, since if their codebase must cross-compile, they cannot benefit from Scala 3’s features. That means that in practice, the visible Scala ecosystem would remain on Scala 2 for a long time, and that could send a worrying message to the public.” Martin seconded this. He said most of the users he talks to who are worried about Scala 3 are worried about slow adoption, but then the forward-compatibility part of SCP-018 strongly reassures them.

Some complex discussion about macros and the overall compatibility followed, difficult to summarize. The key point is that neither forward compatibility nor backwards compatibility provide macros that work across both Scala 2 and Scala 3.

It’s clear that library authors and users will need guidance to understand what will or won’t work in what scenarios, and when it matters how macros are involved.

Jon noted that even if SCP-019 isn’t accepted, that doesn’t close the door on other future proposals aimed at refining SCP-018, as the overall Scala 3 situation develops further.

Voting: three in favor, four opposed (with Seb casting the tie-breaking vote, and promising to “take the proposal into account” regardless), three abstentions, one member not present. The proposal is not accepted.

SCP-020: SBT transitive dependency conflicts management improvements

Proposed by Julien Tournay on behalf of Spotify.

See the proposal text for details.

In presenting the proposal, Julien said that library version conflicts have been a recurring problem in deploying code at Spotify. Sometimes “we only realize there is a conflict when we run our data pipelines,” sometimes not until after the job has been running for hours.

Lars asked to what extent the proposed work on sbt should overlap or interact with the work the Center is now doing on Coursier.

Julien said that Coursier isn’t the default in sbt, so for now, “most builds” would have the same issue even if it were fixed in Coursier. But if Coursier fixes it and also becomes the de facto standard for Scala builds, then yes, “done.”

Seb observed that sbt does already print eviction warnings. Lars asked how a build tool can do better than that, given the lack of an accepted standard way for libraries to signal binary compatibility. Not all libraries use semantic versioning, or do it the same way. “The reason I sound pessimistic,” said Lars, “is that I’m concerned about false positives,” a tool reporting conflicts where no incompatibility exists. But the proposals seems to give “leeway” to the Scala Center to do whatever’s feasible and helpful.

Seb suggested perhaps adding a “2b”, between levels 2 and 3 in the proposal, which would involve “a more elaborate API for conflict resolution”, since ConflictManager, even if it worked perfectly as designed, arguably isn’t a flexible enough mechanism, for the same reasons that Lars brought up: “you want to resolve conflicts in a different way for different libraries.”

Mark spoke up in support of the proposal, because the problem is real, and the proposal gives the Scala Center flexibility to choose how to address it. “At least level 1 would totally be helpful.” Perhaps plugins such as sbt-dependency-graph could become part of sbt.

Jonathan and James both expressed some doubt about the usefulness, in their shops at least, of upping the overall level of “intricacy” around dependencies, especially in polyglot environments, by introducing yet more conflict resolution rules and mechanisms.

Voting: seven in favor, one opposed, one abstention, one member not present. The proposal is accepted.

Community feedback

Lars Hupel announced that he is stepping down as community representative; Jon thanked him for his more than two years of service. A replacement has yet to be nominated. (Last time around, the spot was offered to Typelevel to fill; Jon anticipated that happening again, but it’s up to the board.)

Mark said that “he’s glad to see a focus on tooling, since that is still the thing that comes up most often” in his conversations with community members.

Other business

At the previous meeting, there was some discussion about perhaps changing the Center’s sponsorship terms. Jon asked Seb if this has been considered further yet, and also asked if it would be good to create a concept of a “developer sponsor” who would pay a smaller fee but make up for it by contributing development work directly. But the board agreed there wasn’t really time to discuss it. James asked if the Center could make specific suggestions, to be voted on at the next meeting. Seb agreed to raise the topic on the board’s mailing list, for discussion in advance of the next meeting.

Conclusion

Jon said that as usual, the next meeting will be in approximately three months, perhaps early March. Then the meeting after that will be held in Lausanne, in conjunction with Scala Days.