Minutes of the 2nd meeting of the Scala Center, Q3, 2016

Minutes are archived on the Scala Center website.

Executive Summary

The following agenda was distributed to attendees: agenda.

New board members from SAP, Tapad, and Twitter were present.

Scala Center activities for the past quarter included hiring four engineers and launching three successful MOOCs. Scaladex and the new SIP process were launched as well. Efforts are underway on the Spark MOOC, the SLIP process, code rewriting for Dotty migration, Scala.js integration for front-end developers, conceptualizing a “Scala platform”, and starting a series of Scala ecosystem hackathons.

The following proposals were voted on and adopted:

  • SCP-001: Native Execution of Scala/Spark via LLVM
  • SCP-006: Compile Time Check of Serializability

Other topics discussed included the Scala license, presentation compiler support for Scala macros and for Dotty, ScalaBridge, and expanding the advisory board to include more community seats.

Date, Time and Location

The meeting took place virtually, via Google Hangouts, at 5:00pm (Lausanne time) on Tuesday, August 9, 2016.

Minutes were taken by Seth Tisue (secretary).

Attendees

Attendees present

The following new board members were present:

  • Rami Akkad, SAP
  • Stu Hood, Twitter
  • Dag Liodden, Tapad

Other board members and officers present:

  • David Grove, IBM
  • Alexy Khabrov, Nitro
  • Heather Miller, EPFL
  • Adriaan Moors, Lightbend
  • Martin Odersky, EPFL
  • Tim Perrett, Verizon
  • Jon Pretty, EPFL
  • Raúl Raja, 47 Degrees
  • Bill Venners, Artima

Also present:

  • Nakul Jindal, IBM
  • Jakob Odersky, IBM
  • Denys Shabalin, EPFL
  • Seth Tisue, Lightbend

Apologies received

  • Jonathan Perry, Goldman Sachs

Proceedings

Opening Remarks

As chairperson, Jon Pretty conducted the meeting, made the opening remarks, and asked all participants to introduce themselves, including the following new board members:

  • Stu Hood has been at Twitter for five and a half years. He is primarily interested on the tooling experience around Scala. He and others at Twitter are working on gathering input from within the company to bring to the board.

  • Dag Liodden is cofounder and CTO of Tapad. Tapad has long been active in the Scala community, both in New York City and more generally.

  • Rami Akkad has been at SAP for 3.5 years, in Berlin. SAP is now investing into Scala and the Scala community. They are using Scala to build a platform on which end users will build applications.

Also present for the first time:

  • Jakob Odersky from IBM’s Spark Technology Center attended to provide input on a Spark related proposal, about serialization.

  • Denys Shabalin from EPFL attended to provide input about the Scala Native project.

Reports

Scala Center Activities

As Executive Director, Heather Miller summarized Scala Center activities since the last meeting.

There are four new employees:

  • Julien Richard-Foy, previously at Zengularity, known for his articles and talks about Scala
  • Ólafur Geirsson, recent MS graduate of EPFL, author of the formatting tool scalafmt
  • Guillaume Massé, already previously an independent contractor at the Center working on Scaladex
  • Jorge Vicente Cantero, recent BS graduate from EPFL, contributor to research projects there

At the last meeting, a proposal was accepted for the Scala Center to coordinate the SIP (Scala Improvement) and SLIP (Scala Library Improvement) processes. Jorge has successfully restarted and reformed the SIP process. The first SIP meeting was in July, with another meeting scheduled in August; for details, see this blog post. Jorge is now working on a redesigned SLIP process, inspired in part by the Haskell Process. The SLIP committee will be separate from the SIP committee. SLIP goals include: providing a “batteries-included” distribution aimed at newcomers and providing infrastructure and scaffolding to commonly-used community libraries to help them thrive.

(Raúl asked if we could see the proposed SLIP process and the plans for the platform. Heather said not quite yet, but soon; the Center wants to present strong initial versions, even though they will surely evolve further in response to feedback. They’re trying hard to strike a balance between providing structure but not being too controlling.)

A proposal was also accepted to clarify the migration path from Scala to Dotty. Olafur has begun working on an automatic code-rewriting tool, Scalafix, that can make changes to existing Scala code, such as rewriting uses of Scala’s procedure syntax which has been dropped from Dotty. A first Scalafix release is imminent.

Another initiative is ensuring the continuity of the Scala.js project. This will be Julien’s other big project besides working on the MOOCs. He plans to focus on better-integrating Scala.js into the workflow of front-end developers.

On the MOOC front, the newly launched courses have enrolled 123,758 students since May 23. The completion rates are reported by Coursera in an odd and difficult-to-summarize form, but are high. The Spark course will launch within the next month. Julien is working on the capstone project for the Progfun classes; that should launch by the end of September.

Scaladex launched version 1.0 on the same day as the meeting. There were hundreds of users daily even during development, before the launch. The following short video (1.5 minutes) shows off Scaladex features: https://youtu.be/TBoJivIJsbU . Additional features are planned.

The center plans to hold a series of Scala Ecosystem Hackathons. A future blog post will describe this in more detail, but the goal is to have library authors guide new contributors, in person, through the process of pair programming together to create their first pull request for that library. Preliminary trials will be run at Scala World and Lambda World and another trial is planned for Scala Exchange. (Alexy from Scala by the Bay expressed interest in participating as well.) The goal is to make these events “portable and repeatable” so they can be run in cities everywhere.

Scala Center Financial Statement

EPFL provided the Scala Center with seed funds for the first year, providing for two employees and a part-time administrative assistant. This funding is kept separate from money from companies and primarily funds the MOOCs.

There are now nine advisory board members. 225K CHF of funds has been received from board members; 225K CHF more is in process. As of August 1st, these funds are now being used to fund developers (see the list of new hires above). Two advisory board members very roughly equals the ability to hire one engineer.

The hope is that funds from the MOOCs will replace the first-year EPFL seed money. Coursera takes a cut of course fees. So far about 900K CHF has come in. From that Coursera takes half and EPFL takes a sixth. That leaves a third for the Center, but production costs must be repaid from that amount. The exact eventual amounts aren’t known yet; it’s working its way through the EPFL system.

Rami asked if advisory board funds go entirely to engineer salaries, or also to costs like equipment and travel. Heather says approximately 5K CHF is allocated per engineer for equipment and travel, such as conference attendance. Most infrastructure is already provided directly by EPFL.

Scala Center Proposals

Proposal texts are in the proposals directory.

Proposal SCP-001: Native Execution of Scala/Spark via LLVM

proposed by IBM’s Spark Technology Center

Presented by Dave from IBM. Would it be practical to use Scala Native to run Spark jobs, compiled to LLVM? The main goal would be improved performance. How much of the language and standard library will work? How is Denys’s work progressing?

Denys said the Scala Native project is now exactly a year old and was open-source three months ago. Since the open-sourcing, a community has sprung up and over 100 pull requests from 19 contributors have been merged. The project is close to a 0.1 release; see the 0.1 milestone on GitHub for details.

“Essentially all” of the Scala language is supported, just like Scala.js. There are still a few semantic differences from JVM, but those are considered bugs and are being addressed. The main reason to not release 0.1 yet is incomplete support for core JDK libraries: stuff in java.lang, java.util, java.io, and java.nio that is essential. In particular, we don’t want to force people to use native APIs to do I/O. “There is still lots of work to be done” on this and assistance is welcome: from the Scala Center, from companies, and from the community. Work on the JDK libraries is “parallelizable” across contributors.

The library reimplementation work is done “clean room” style, using Oracle documentation but without ever consulting Oracle code (since their license is GPL but Scala Native is BSD 3-clause). There was some discussion among the board about the legal issues around clean room implementations. Dag suggested looking at the contributor questionnaire from the Apache Harmony project and said “The legal implications… need to be iron clad early. If this will form the basis of native Spark, for instance, enterprises will need proper legal indemnification.” This was seconded by Tim.

Jon asked if instead of implementing Oracle’s API, if a new API could be invented, and then implemented on all back ends. Denys explained his reasons for not choosing this path; mainly, that it would require convincing all Scala library authors to switch to the new API.

Stu asked if the API implementation would be reusable by the Scala.js project. Denys said JavaScript is such a dissimilar target that there isn’t much opportunity to share code. “Such sharing is a harder task than it initially seems”, but perhaps some sharing can happen “long term”.

Denys is also working on implementation Java reflection, though this has been “controversial”. Only a subset of common operations will be supported. It will be necessary for users to statically declare ahead of time what they will be reflecting on at runtime.

Jon asked if Scala reflection might be supported as well. Denys thought this would probably be too ambitious. Java reflection is simpler, is “frozen” in its design, and is much more commonly used.

Heather addressed the issue of whether the Scala Center has enough resources for this. It seems possible, and Jorge is interested, after the SLIP process is running. Dave will try to find out whether the Spark Technology Center can provide direct assistance.

Outcome: Adopted, by a 7-2 vote (1 board member not present)

Proposal SCP-006: Compile Time Check of Serializability

Jakob: “In our work with Spark, we run into this issue all the time.” Serialization errors happen at runtime; we want to catch them at compile time. He showed slides with motivating example. The most common scenario is a class that can’t be serialized because it refers to an identifier from a non-serializable enclosing scope. The usual workaround is to locally alias the identifier. Another scenario is accidentally capturing a value that can change later, when you meant to capture the current value.

The Spores proposal (SIP-21) includes similar examples and addresses the same problem. Could SIP-21 be revived?

Heather, as one of the creators of spores, addressed the status of SIP-21. It has not progressed recently. In the context of Java serialization (as opposed to Scala Pickling, which Spark doesn’t use), the main limitation of spores is around transitivity – if an enclosing scope in turn captures something else. Proper transitivity checking needs a compiler plugin; macros aren’t enough. “I would go with a compiler plugin,” she said. Another possible solution path would be to use Scalameta’s macro annotations, but it’s unclear if that’s the right path. There was some further discussion of technical details between Raúl, Stu, Heather, Jakob, et al.

Writing an initial version of a compiler plugin would take “a couple months”; does the Center have the resources? Jorge might be interested, but he might be allocated to Scala Native. Maybe Guillaume? Heather will ask him.

Outcome: Adopted, by a 8-1 vote (1 board member not present)

Other business

Proposals and process

Jon observes that all proposals so far have been adopted. Does that mean we aren’t getting enough proposals?

Jon asks that future proposals devote more attention to estimating time and budget requirements.

Stu suggests allowing a longer time for review and comment on proposals, before the board has to vote. Heather agreed.

Rami says SAP intends to submit proposals soon; Stu said the same of Twitter.

Bill received a meta-proposal from Sam Halliday from the ENSIME project, incorporating several topics around licensing and tooling. That’s the source of the next few agenda items (on the Scala license, the presentation compiler, and sbt). These topics could eventually become concrete proposals.

The Scala license

Bill summarized Sam’s licensing concern as follows. Scala uses the BSD 3-clause license, which doesn’t mention patents. The Scala CLA does address patents. “This is concerning because there is no guarantee to recipients of the Scala compiler and standard libraries that they have the right to use the software without being liable to pay a patent royalty,” wrote Sam. Apache License covers patents in the license itself. Could Scala be changed to a Apache 2 license?

“I don’t think we can relicense,” said Adriaan.

Heather said she forwarded Sam’s questions to the EPFL technology transfer office and in their opinion, having the separate CLA doesn’t cause any major legal loopholes or problems.

Seth wanted a clearer writeup of what the problem is and what the proposed fix us. Relicensing would be hard, so we need a clear benefit.

Tim: “Licensing is definitely a big deal, relicensing is a big deal, since it affects all users and contributors.” And “The purpose of the CLA is to protect the receiver of the contribution” so “it makes sense” that the CLA would have stronger wording than the licensing.

Martin: “I can quickly say what the risk of this would be. Generally from EPFL it’s very hard to do an Apache license, and EPFL is one of the copyright holders for Scala. EPFL has a patent office and holds patents. Having to have EPFL check every Scala release for issues with EPFL’s patents could create long delays for every Scala release.”

Consensus: no further action on this for now. Bill will communicate this to Sam.

Presentation compiler for Dotty

Sam wrote, “It is concerning that Dotty does not yet have a presentation compiler. Without a presentation compiler, it could mean the end of the road for both ENSIME and Scala IDE.”

Martin told us that Microsoft and IntelliJ have a new standard for how an IDE should interact with a presentation compiler. “It would be interesting” to see if we could implement that standard”, but “right now nobody’s working on that because we simply don’t have the people and resources”, but “I would definitely like to be able to do it”. He also said that Dotty is better designed than the old compiler as a host for a presentation compiler. It has been designed “from the ground up” for incremental changes, for a quickly changing codebase.

Raúl: “I think that this is very important because a lot of the community depends on ENSIME. Other IDEs like IntelliJ may not be affected.”

There was general agreement that yes, we are interested in receiving a formal proposal on this.

Macros in the presentation compiler

Sam wrote, “The current presentation compiler is sadly lacking in the area of macro support.”

Martin: “macros are still experimental” and “nobody’s very happy” with the current API, so it would really be better to put the energy into Scalameta and have everyone support that. Scalameta is also “the most important” thing we need for “smooth migration from Scala 2 to Dotty”.

So, it sounds like there could be interest in a proposal centered on Scalameta.

Support for sbt development

Sam wrote, “It is no secret that the Scala compiler is slow… An efficient incremental compiler… could speed up the workflow for developers… sbt is a critical part of most build pipelines and it includes an incremental compiler… It would be beneficial to all Scala developers if the Scala Center could provide the resource to help sbt and IDEs integrate with each other via sbt-server.”

Adriaan reiterated Lightbend’s support of sbt development: on a 1.0 release. The ambitions for sbt server have been “scaled down” to something “more achievable”.

There wasn’t clear interest or disinterest from the board, but no one spoke up strongly in favor.

ScalaBridge

Bill briefly described the new ScalaBridge effort. Its goal is “Building an inclusive Scala community with introductory programming workshops for women”, modeled on RailsBridge.

Heather said she’s in communication with Kelley Robinson about it, about the curriculum design and about rallying the community around it. For the most part, they are doing fine on their own without a lot of Scala Center involvement, but Heather is “investing time” and wants to “bring visibility” to it.

Alexy: “SF Scala supports it fully and so will Scala by the Bay and surely other community vehicles.”

Adding more community delegates to the board

At the last meeting, adding an additional delegate was suggested. Jon now suggests we should consider having “as much as a quarter” of the board be community-based. “As a board, we need to have legitimacy in the community. we don’t want to be seen as purely acting in the interest of large commercial entities.” Adriaan: “Agreed with Jon we should balance OSS & commercial.”

There was a discussion of whether the board should invite additional community delegates directly as individuals, and/or allocate seats to organizations such as Typelevel to fill themselves. Alexy, especially, spoke up in favor of a “bottom-up” process for selecting the community delegates, instead of doing it “ad hoc”. Perhaps candidates could compete, to strengthen legitimacy and help get new faces involved. Rami, Dag, and others agreed “we need a process”. Jon pointed out it would be difficult to define the electorate.

The discussion was inconclusive, but there was consensus to invite at least one additional individual directly very soon, but to continue revisiting this issue in the future.

Closing remarks

See you in Q4 – perhaps in coordination with Scala by the Bay, in November?