Minutes of the 20th meeting of the Scala Center, Q1 2021

Minutes are archived on the Scala Center website.

Summary

No agenda was distributed in advance, but the usual meeting structure was followed.

Note that less than two months passed since the previous meeting. This was done to reset the quarterly schedule, which had slipped.

SwissBorg has joined the Center as an Affiliate member. Databricks, known in the Scala community for their stewardship of Apache Spark, has expressed interest in joining and sent Kris Mok to this meeting as a guest.

Maureen Elsberry has replaced Rebecca Mark as the 47 Degrees representative. Starting next meeting, she will share the slot with Diego Alonso.

At the Center, Vincent Derouand has joined the staff as communication specialist at 40% time. Valérie Pedroni is expected to begin soon as a communication intern.

Center activities for the past quarter focused on MOOCs, Google Summer of Code, new Scala 3 video series, new Scala 3 migration tool, Scala 3 migration guide, Scala 3 compiler improvements, Scala debug adapter, sbt, Scalameta and Scalafmt and Metals, the TASTy reader and TASTy manipulation library, Scaladex, Scala.js, Scala Native, blog posts, a conference and meetup organizers summit, and creating a new communications team.

Details on all this are in Darja’s direction report and Seb’s technical report.

There were no new proposals this quarter.

Other business discussed included the Center’s efforts to respond more effectively to harassment within the community, Scala 3 migration in the community, Databricks’ technical concerns around Scala (both Scala 2 and the eventual migration to 3), and the impending shutdown of Bintray.

Date, Time and Location

The meeting took place virtually on Tuesday, March 30, 2021 at 3:00pm (UTC).

Minutes were taken by Seth Tisue (secretary).

Attendees

Officers:

  • Adriaan Moors (chairperson)
    • also board member, representing Lightbend
  • Darja Jovanovic (executive director), EPFL
  • Sébastien Doeraene (technical director), EPFL
  • Martin Odersky (technical advisor), EPFL
  • Seth Tisue (secretary), Lightbend

Board members:

  • Chris Kipp, Lunatech
  • Maureen Elsberry, 47 Degrees
  • Graham Griffiths and Alex Hurst, Goldman Sachs
  • Rob Norris, community/Typelevel
  • Krzysztof Romanowski, VirtusLab
  • Daniela Sfregola, Morgan Stanley
  • Julien Tournay, Spotify
  • Bill Venners, community/Artima
  • Eugene Yokota, Twitter

Apologies:

  • none

Guests:

  • Kris Mok, Databricks

Activities reports

Darja and Seb summarized Scala Center activities since the last meeting.

Their remarks were largely based on their reports.

The following notes do not repeat the content of the reports, but only supplement them.

Technical report

Seb went first with the technical report. He identified the main highlights to be the Scala 3.0.0-RC1 and RC2 releases, Google Summer of Code, the first release of scala3-migrate, and the first release of the Scala Debug Adapter.

The Scala Debug Adapter was “extracted from what was used for sbt, so it can be used from mill and other build tools”.

The scala3-migrate tool is “semi-automatic”, an “assistant”.

Rob asked if there would be a blog post announcing scala3-migrate: “that would be helpful”. Seb explained that so far they have purposely only announced it on the contributors forum, in order to get a first round of feedback from people who are already in-the-know about Scala 3; a blog post to advertise it more broadly will come later.

Seb talked about the Center’s plans for Scala for the rest of 2021. Most of this material is summarized in Darja’s report. One thing Seb mentioned that doesn’t appear in either report is a planned effort to partner with universities to teach Scala as a first programming language. Some professors are already doing that, and the Center wants to get in touch with additional professors who want to try it.

Making Scala “easy to use” is a phrase from Darja’s report; Seb expanded on this by identifying the following tools as especially needing the Center’s attention this year: debugging tools, Metals (a 1.0.0 release is planned), sbt (including pipelining and build caching, and in general planning for sbt’s future), scala3-migrate, and Scalafix.

Another phrase is “easy to leverage”; Seb expanded on this by saying work is planned on Scaladex, library versioning policies, a TASTy-based successor to MiMa, and planning the future of the Scala standard library.

For the Scala language and compiler itself, the Center plans to work on stability and quality, bringing warnings and linting in Scala 3 up to speed with what already exists in Scala 2, an initialization checker, language evolution through the SIP process, Scala.js, and Scala Native. Scala Native’s main needs are Windows support, multi-threading, and Scala 3 support (“by the end of the year”).

Krzysztof mentioned that planning for the Scala 3 community build includes interfacing it with Scaladex, to locate candidate libraries.

Seth asked about Scalafmt, is that a Center project, or is it independent? Seb: “technically an independent project, but we invest quite a lot in it, we contribute a lot” (especially through VirtusLab). And Scala 3 support is “very close” but not quite done yet. “Everything is supported today except braceless syntax”, and braceless syntax support should come “within one month”.

Rob asked about Scaladex: “Something that the ends up being really important is that there are distinct, not really silos, but distinct ecosystems that have emerged in Scala” – would it worthwhile for Scaladex to recognize that somehow? It affects library choice, since some libraries are designed to work well together. But having Scaladex understand something about this might require human curation, which might be tricky? Seb: Yes, “this is what we want to do, basically,” as part of the overall effort to help people get started with Scala, since in practice that involves libraries and not just the language.

Direction report

Darja said the material in her report about the Scala vision and Scala Center vision going forward came out a workshop between herself, Seb, Martin, Julien Richard-Foy, and the Center’s original director Heather Miller.

The following section on diversity and inclusion addresses the Center’s process for responding to reports of misbehavior and harassment within the community, including at conferences. Darja: “If we don’t act in this area, the other actions that we identified in general around diversity and inclusivity actions would not really ring true.” The new communications team coming on board will expand the Center’s ability to address this and to help community organizers and open source maintainers address it.

Darja also covered the Center’s plans for SCP-025 (Use of inclusive language).

Multiple board members including Eugene, Rob, and Krzysztof expressed support and appreciation for the Center stepping up on these issues, especially since in-person conferences will be starting again before long, post-pandemic.

Chris asked for more detail about how harassment incidents will be handled, including incidents that have actually already happened. “What is a real-world scenario” of how this could be addressed, “What is the expected outcome?” Darja: “There was no process” for past incidents; that’s what needs to change, both to deal with past incidents which the Center has only partly addressed so far, and to establish “best practices” to be ready for the future. Some further discussion ensued about how conference organizers have dealt with this in the past, whether inside or outside of Scala.

Financial report

This section was skipped, since relatively little time passed since the last meeting and there wasn’t significant news.

Community feedback

Bill, representing library maintainers especially, asked how long deprecated features will remain in Scala 3. For example, implicits: Scala 3.0 still supports the old implicit keyword and semantics, alongside the new given and using keywords and semantics. How long will that transition period last? How long will it remain possible to cross-compile projects for both 2 and 3? “That’s important, to have a long deprecation cycle.” And does scala3-migrate help?

Seb said that scala3-migrate that although the tool “is very good at what it does”, it doesn’t aim to (for example) rewrite implicit class to extension, or at least that kind of thing isn’t currently a goal. “The goal of scala3-migrate is to get your codebase to compile and run and test on Scala 3”, but once it’s there, the tool’s job is done.

Martin said old features will stay around “as long as it takes,” and removals will be a multistage process. For example, an old feature such as symbol literals might be retained for a long cycle but require a language import, so become opt-in, rather than simply be present by default. (Seth noted that this approach has worked well for some Scala 2 features such as postfix syntax.) For actual removal, timing will depend on “where the community goes”; features will be removed when they can “safely” be removed. And in the meantime, if corresponding language imports need to be made available in Scala 2 to allow cross-building, the Scala 2 team can do that.

Rob said that from his point of view and within Typelevel generally and the community more widely, the migration to Scala 3 is going well, “we’re working hard to be ready and we’re looking forward to it.”

Other business

Databricks

Kris Mok from Databricks joined the meeting as a guest to bring up technical concerns around Scala both on the Apache Spark project and within Databricks more generally.

Topics he touched on included:

  • Deprecation of symbol literals in Scala 2 and plans for their eventual removal in some future Scala version. Could scala3-migrate help? Notebooks are especially a concern.
  • Traits for declaring configurations are hitting the JVM’s 64K limit on method size, particularly in constructors. “When that happens we have to start telling people you can’t put these in the same trait.” Could the compiler handle this transparently?
  • Spark notebooks are REPL-based, using the Scala 2 REPL. Multiple concurrent users are supported by running multiple REPL instances in the same JVM. Databricks has patched the REPL to make this work. Could changes in this area be upstreamed, could this be supported more officially? The customizations tend to break in new Scala versions (which isn’t Scala’s fault, but is nonetheless an issue).
  • Spark’s integration with the REPL also raises issues around serialization. Spark’s “closure cleaner” seeks to avoid serializing unnecessary outer references. When trouble arises, it’s hard for users to reason about. Could Scala provide better tools for understanding what is being serialized and why?
  • Scala reflection can leak memory unless custom code is used to plug them, especially when multiple REPLs are in the same JVM.

Regardless, Kris also praised the language and says Databricks’ experience with onboarding former C++ and Python engineers (for example) has been very positive.

Martin: “We were not aware” that users were hitting the 64K limit in that way. He thinks Scala 3 may be less susceptible to the REPL-related issues.

Darja suggested following up on these technical concerns further later; the Center’s “doors are open” for followup meetings and also for formal proposals. Kris noted that Databricks engineers would hopefully be available to participate as contributors in addressing some of these issues.

Bintray

Eugene updated the board with news about the impending shutdown of Bintray and JCenter.

The “task force” the Center assembled to cope with this has met several times already.

Eugene said JFrog has recently agreed to provide sbt, as an important open-source project, with a level of free service that would have been quite costly if we needed to actually pay for it. Not all the details are clear yet, but it seems promising that this will be sufficient for hosting all of the sbt artifacts and community plugins, probably with no disruption to users.

But it remains unlikely that sbt will continue to provide publishing for new plugins or new versions of existing plugins; plugin authors are encouraged to look elsewhere. Sonatype is a popular choice.

Rob asked who JFrog is granting this service to, what organization will hold this license? Eugene: the Scala Center (not Eugene personally!). And both Twitter and VirtusLab are willing to provide a certain amount of engineering or maintenance effort on this.

Eugene: “I’m not fully relaxed until the jars are actually transferred”, but he expects it to work out.

Conclusion

Darja encouraged all of the board members, including companies who are just joining, to submit proposals, and also to check in with their teams and community contacts to find out what concerns could be raised.

The next meeting will likely take place around June 30.