Scala Center team:

  • Jorge Vicente Cantero, 100%
  • Ólafur Geirsson, 100%
  • Martin Duhem, 100%
  • Guillaume Massé, 100%
  • Julien Richard-Foy, 90%
  • Darja Jovanovic, 80% (intern)
  • Heather Miller, 50%

Initiatives worked on this cycle:

Scalameta

Ólafur Páll Geirsson @olafurpg

Released Scalameta v2.1.2, see changelog. Highlights:

  • #1143 Fix incompatibility with the semanticdb-scalac compiler plugin and scalamacros/paradise macro annotation.
  • #1144 Upgrade semanticdb-scalac to 2.12.4. Due to large internal refactorings in scalac typer, this upgrade turned out to be unusually complicated.
  • #1145 Fix bug in handling of by-name and repeated parameters.
  • #1148 Add Denotation.members. This opens the possibility to reproduce lexical scoping at a position enabling rules like 1) ExplicitResultTypes to safely insert unqualified named with accompanying imports or 2) an OrganizeImports rule to safely.
  • #1154 Fix bug where Foo in new Foo {} resolved to an anonymous class instead of the Foo symbol.
  • #1158 Amongst many things, significantly reduces memory pressure while compiling sources with the semanticdb-scalac compiler plugin, which is required by Scalafix.

Scalafix

Ólafur Páll Geirsson @olafurpg and Guillaume Massé @MasseGuillaume

Guillaume joins Ólafur on the Scalafix project. He will work on a wide range of topics, including adding new rules and polish the general user experience, which is quite rough at the moment.

Released Scalafix v0.5.6, see changelog.

Highlights:

  • #412 Make it easy to enable sbt-scalafix for custom configurations like IntegrationTest. (@olafurpg)
  • #415 Suppress false linter errors with comments. This PR makes it practical to use Scalafix for linting, since previously there was no way to silence Scalafix for exceptional cases where the linter reports a bogus diagnostic. (@olafurpg, @MasseGuillaume)
  • #418 Added detailed documentation comparing scalafix with alternative tools like Scala Refactoring, WartRemover, ScalaStyle and IntelliJ Scala plugin. (@olafurpg
  • #419 Simplify installation of sbt-scalafix. Based on feedback from users, this PR adds a scalafixEnable command that can automatically setup all the correct scala compiler settings from an sbt shell session, regardless of existing settings in build.sbt. Previously, it was necessary to manually update build.sbt with fairly tricky corner cases. (@olafurpg)
  • #420 Reviewed PR adding documentation for how to use scalafix for Maven builds. (@olafurpg)
  • #431 Implement a Syntaxtic rule to disable Scala keyword, xml, tabs, carriage return and semicolons (@MasseGuillaume)
  • #454 Refactor the sbt build to improve release cycle.
  • #465 Add –diff option to only run scalafix on a set of changes pending code review. This should makes it easy to incrementally incorporate scalafix rules into an existing large codebase.

Macros

Ólafur Páll Geirsson @olafurpg

  • Wrote “What’s happening with macros?” blog post.
  • A lot of failed experiments trying to automatically expanded trees from scala-reflect macros that that mix untyped and typed trees.
  • Studying @odersky’s ideas about a new design for macros that unifies LMS and macros, see discussion
  • Orienting Ryan Culpepper, the author of Racket macros, on the design challenges in this new macro rework. Ryan has expressed interest in contributing to this project.
  • Leaning about Squid, a metaprogramming framework based on type-safe and hygienic quasiquotes, developed at the DATA lab at EPFL. Squid seems to address a lot of issues in scala-reflect and may be a great complement to the new macros. Squid also builds on top of scala-reflect, which means it could maybe provide a smooth migration story for the existing ecosystem of macros.

Compiler Profiling

(SCP-010)

Jorge Vicente Cantero @jvican

scalac-profiling has been significantly improved to be battle-tested in big, well-known open source libraries of the Scala community. This work started as part of my writing of a technical guide to let people in the Scala community use the plugin to profile their projects. This technical guide will be published towards the beginning of December.

Improvements to the project:

  • Add flamegraph generation for implicit searches (!).
  • Add dot graph generation for implicit searches.
  • Add more measurements of the compiler to get more data for users.
  • Profile the cost of macros and implicit search in all the integration builds.
  • Add shapeless, scalatest, better-files and scalac as new integration builds of the scalac-profiling build (complementing our previous integration builds circe and monocle). Every merge in the project is tested against these projects in the CI to ensure there’s bad interaction.

For a sneak preview, head to this page that explains the current status of the visualizations of implicit search. If you’re too busy, check the example Flamegraph.

The plan for the technical guide is the following:

  • Explain the overall idea of scalac-profiling and why it allows people to fix their compile times.
  • Write the guide by showing how to profile compile times for several use cases. Every use case will be for some of the integration builds that we have: shapeless, scalatest, and circe.

Zinc

Jorge Vicente Cantero @jvican

(Read the blog article)

My work on Zinc has focused on several areas:

  • Improving classpath hashing for incremental compilations. The classpath hashing was added before the Scala Center got involved in the project and created a severe regression in incremental compiles for users of Zinc 1.0 – incremental compiles were much slower because it had to hash all the classpath entries. This regression was measured to be in the range of 15 to 40 seconds in medium to big projects.
  • Fix #127: Use unexpanded names.
  • Reviewed around 8 PRs on our queue; some have been merged, some are still under work.
  • Fix #436: Remove annoying log4j scripted exception.
  • Work on a better build and test workflow for Zinc. The scripted integration was slow, and that put off contributions and made changes to Zinc slow to happen. The rework of the build is split into several PRs (#438, #428, #429 and #440) that add:
    • A Zinc scripted mechanism that is reproducible (isolated from users’ cache) and fast (parallel execution of tests). Note that Zinc uses an independent scripted than sbt.
    • On-merge releases (so that users of Zinc can depend on versions of pull requests that have been merged right away instead of being blocked on maintainers to release them).
    • A publishLocal implementation that is cached and avoids publishing when the same version is already in the cache.
    • Better UI for both the build and the test execution (colors, highlighting, log collection in an independent tmp folder, etc).
    • Documentation for all the steps of the build and removal of dead code.

Scala Platform

Jorge Vicente Cantero @jvican

  • Installed the new Drone CI version (0.8) in platform.scala-lang.org for the benefit of the Scala Platform maintainers and the Scala Center engineers.
  • Coordinated two pull requests for Scala JSON and Better files that use the new Scala Platform infrastructure.
  • Made several significant improvements to the Scala Platform plugin, which now fully integrates with the sbt-release-early and has a detailed test infrastructure.
  • Had the second SPP meeting with all the Committee members. These are the outcomes of the meeting:
    • Introduction of the roadmap for the Scala Platform.
    • Initial explanation of how the Scala standard library split will happen.
    • better-files got approved to be added to the platform by unanimity.
    • first steps discussing whether the Scala Platform modules should support cross-platform.

Community

Darja Jovanovic @darjutak

Started at the Scala Center as an intern on October 24th, 2017.

Role: Communication manager.

  • Make sure the SIP/SPP meetings are organized on a monthly bases, minutes published in a timely manner and communication distributed throughout the Scala channels. Goal: Both with a goal of developing trust, gaining momentum, deciding about different proposals and moving forward.
  • Gather and publish Scala Center monthly updates to community and AB alike.

Results so far:

Big increase in the number of people who tune into SIP meetings:

2017-q4/youtube_view_overview.png

Big uptick in views on Scala Contributors posts announcing what the SC has been up to.

  • Sept-Oct updates, 792 views:

2017-q4/project_update_792_views.png

  • Oct-Nov updates, 1.5k views:

2017-q4/project_update_1.5k_views.png

And lots of positivity about these updates too!

2017-q4/contributors_comments_project_updates_1.5k.png

Stickers

Community engagement exercise with incredible outcome!

  • Decided to give away Scala and Scala Center stickers.
  • After a single tweet about it, we got 891 responses with an amazing comments full of love and appreciation for Scala! Here are those responses
  • Sending out stickers to users weekly; 200 letters in each batch to every corner of the world!

Scala Improvement Process

Jorge Vicente Cantero @jvican

  • The SIP meeting scheduled to happen on the 29th December was cancelled because we didn’t reach the quorum. Darja is working on the reschedule.
  • Create first prototype for opaque types. The pull request will be made before our next SIP meeting.
  • Coordination with the Committee on several topics.
  • Organisation of a voting in the next SIP meeting to vote for the admission of a Typelevel representative: Miles Sabin.
  • Synchronization with Miles on several proposals he’s preparing.

Scala Days PC Organization

Darja Jovanovic @darjutak

Scala Center’s goal: assist in pulling in as much of the community to the Scala Days events as possible. We’d like to see more community talks this year.

  • Worked with Lightbend and Heise to release the joint call for presentations for Scala Days 2018.
  • New this year: CFP application for managing submissions and PC discussions. Available at: Scala Days 2018 CFP
  • Organized PC, set up meetings, managing deadlines.
  • 2018 Scala Days PC members:
    • Martin Odersky
    • Lars Hupel
    • Adriaan Moors
    • Manuel Bernhardt
    • Heather Miller
    • Markus Eisele
    • Josh Suereth
  • Already ~40 talk proposals submitted!

SBT 1.x Plugin Migration

As part of SCP-013, we migrated two of the most starred open source sbt plugins that had not been ported to sbt 1.0 yet:

Guillaume Massé @MasseGuillaume and Martin Duhem @Duhemm

Based on the sbt 1.x migration wiki. We migrated non-commercial plugins that have an impact on the community:

  • scala-native is a new target for the Scala programming language. It uses llvm to link an run Scala code to any platform. It has a seamless integration with sbt. You can run and test just like you do it on the JVM. PR #1039 (@MasseGuillaume)

  • Dotty is a research platform for new language concepts and compiler technologies for Scala. It’s tightly coupled with sbt and Zinc and the migration to 1.X is still in progress. There is more detail on the Dotty section (@Duhemm)

  • sbt-dependency-graph is a tool to understand how your external dependencies are wired together. It’s essential to debug anomalies on the classpath. PR #140 PR #139 (@MasseGuillaume)

  • g8 in an sbt plugin that powers the sbt new command. It allows the creation of template to quickly create new projects. PR #350 (@MasseGuillaume)

  • sbt-aws-lambda is an sbt plugin that helps developer in deploying code to AWS lambda. PR #39 (@Duhemm)

  • sbt-spark-submit is an sbt plugin that makes it easy to submit jobs to spark, using spark-submit under the hood. PR #9. (@Duhemm)

Thanks to the authors of these plugins for the quick reviews!

As far as we can tell, most of the open source, non-commercially owned sbt plugin now support sbt 1.0. Please let us know if we missed something important!

Dotty

Martin Duhem @Duhemm

Specializing functions

The work I started last month (from an initial PR by Felix Mulder) to specialize functions in the dotc compiler is working. Our tests show that the code that is produced by dotty allocates fewer boxes to hold primitive values. However, it turns out that it has a negative impact on the performance of the compiler and increases compilation time.

I am working on reducing the overhead of this optimization so that compilation times are not affected.

The work is happening on this pull request.

Embedding documentation in TASTY

I started working on embedding documentation in TASTY, so that it is always available with the classfiles of a compiled project. The goal is to be able to easily query the documentation from the REPL, or an IDE:

scala> object Foo { /** hello! */ def bar = 123 }
scala> object Random {
     |   /** Returns a random number
     |    *  @return A number chosen by fair dice roll,
     |    *          guaranteed to be random.
     |    */
     |   def getRandomNumber(): Int = 4
     | }
defined object Random

scala> :? Random.getRandomNumber
/** Returns a random number
 *  @return A number chosen by fair dice roll,
 *          guaranteed to be random.
 */
def getRandomNumber(): Int

At the moment, my work in progress can store the documentation strings in a new section of the TASTY information and the REPL starts to support queries but needs more work.

Improving the Dotty Community Build

The Dotty Community Build is a corpus of open source Scala projects that we use to detect regression in Dotty. At the time of writing, eight projects are part of the Dotty Community Build.

Every night, we use the latest nightly release of Dotty to compile those eight projects, and can therefore quickly detect any regression. However, we didn’t have a simple way to test the Dotty community build against a pull request, a tag or a specific commit.

I took some time to improve on that and added a new script that allows us, given a reference inside the Dotty repository, to locally build, publish and test Dotty against the community build. This way, we can merge with confidence pull requests that may cause a regression that would be hard to catch with smaller tests.

Here is the pull request.

We do not have integration with our bot at the moment, be I hope to add that soon.

Porting Dotty to sbt 1

The dotty sbt plugin can currently only be used with sbt 0.13. However, now that sbt 1.0 is out, it makes sense to upgrade to the latest versions of sbt. Future releases of Dotty will drop support for sbt 0.13.

I started upgrading our build, plugin and compiler interface to sbt 1.0 and the latest release of Zinc. Upgrading to the latest version of sbt lets us benefit from the recent improvements that were made in sbt, and most notably it allows us to enjoy faster incremental builds, thanks to the new class based name hashing featured in Zinc 1.0.

Because of the many changes that happened in Zinc, especially regarding how a Scala compiler must communicate API and dependency analyses with the incremental compiler, upgrading our compiler interface was a large task. We also discovered several bugs in Zinc in that process.

After I got our build to work and all our tests to pass, I needed to upgrade all the projects that appear in our community build to use sbt 1.0. The changes that were made to upgrade the build definitions to sbt 1.0 have been proposed for inclusion the upstream projects, and I am waiting to hear back from them.

The work is happening on this pull request.

Fixing two bugs in generic java signatures

My first task with Dotty was to add support for generic java signatures. See my description in our previous Scala Center update to know what this work is all about.

We discovered a bug with the generic signatures, where additional parameter lists may be missing.

I fixed this issue in a subsequent pull request.

Another bug was discovered, regarding singleton types and their signatures. I worked on a fix for this issue. The patch was merged and released in Dotty 0.5.0-RC1.

MOOCs

Julien Richard-Foy @julienrf

Added XQueue support to our grading infrastructure. I implemented an XQueue client that periodically polls the queue for assignments to grade, and launch the corresponding grader each time there is a new assignment to grade.

Scala.js Bundler

(SCP-005)

Julien Richard-Foy @julienrf

  • Made it possible to release scalajs-bundler by pushing a Git tag #184, so that any contributor that has write access to the scalajs-bundler GitHub repository can cut a release (they don’t need to deal with Sonatype access rights)
  • Released scalajs-bundler 0.9.0

Collections

(SCP-007)

Julien Richard-Foy @julienrf

  • Fixed undefined references in API documentation #274
  • Ported scalacheck tests of the current collections to the strawman #276
  • Reviewed @marcelocenerine’s work on lazyZip. The goal of this operation is to provide a more discoverable alternative to (xs, ys).zipped. The new syntax is xs lazyZip ys. This operation lazily zips collections together (the zipped collection is not evaluated unless it is iterated on) and provides transformation operations (such as map and filter) that don’t create intermediate tuples. #223
  • Tried an alternative approach to implement MultiMap: create its branch in the hierarchy. #269
  • Fixed performance regressions #270, #271.
  • Added an updateWith/updatedWith operation to Map collection types (in collections-contrib) #268.
  • Merged foldSomeLeft and lazyFoldRight operations #295.
  • Added MultiSet and SortedMultiSet #299. This addition is only available in the collections-contrib artifact. These new collections form a new branch (they are not comparable with Set).
  • Reviewed and merged minAfter and maxBefore operations on sorted collections #302.
  • Overrode a bunch of methods in strict collections, for performance #305.

Collaboration: Introducing Bloop, a wrapper around Zinc

Martin Duhem @Duhemm and Jorge Vicente Cantero @jvican

Bloop is a command-line tool for fast edit/compile/test workflows. Its primary goal is to give you the fastest possible experience specifically in the edit/compile/test loop. Bloop is not meant to replace your current build system. The name stands for “build loop”.

For more details about Bloop, see our recently released blog post: Meet Bloop, a fast tool to compile and test your project

Analysis of Scalac performance

We created bloop because we observed that Scala compile times slow down when within sbt when compared to isolated benchmarks. This observation was at the beginning just an intuition, but when we sat down to measure a prototype we did see a significant difference.

Project name sbt (version) bloop speedup
sbt/sbt 21s (1.0.4) 15s 1.17
guardian/frontend 37s (0.13.16) 28s 1.32
apache/spark 159s (0.13.16) 107s 1.4
scala/scala 65s (0.13.16) 50s 1.14

These numbers have been obtained on an isolated machine, measuring at the fifth hot clean-compile iteration (of all the projects and subprojects of the build without counting the tests) and with 2 gigabytes of heap for both sbt and bloop (the shell, not the nailgun integration). When benchmarking sbt, we have made sure that dependency resolution and formatting are not measured.

Note that in practice, the provided speedup is higher because sbt’s inherent model runs all the dependent tasks on compile every time people do compile, ~compile, test or ~test. This cost is “constant”, but it’s not negligible for big codebases and although we don’t add it in our analysis, we’re confident that bloop outperforms sbt’s Scala compilation by wider margins that the ones here reported.

Note also that we have not taken into account the fact that Zinc 1.0 (the incremental compiler that Bloop uses is much more precise and takes more time than the Zinc 0.13.16). 2-3% can probably be gained in favor of Bloop in those previous benchmarks.

This project is publicly available at https://github.com/scalacenter/bloop

Presentations and Workshops

Scala.io

Julien Richard-Foy @julienrf

Lambda World

Martin Duhem @Duhemm and Jorge Vicente Cantero @jvican Organised the Spree at Lambda World in which around 20 people attended the event. More than 15 pull request was merged in well-known open source projects, and we had special help from Daniela Sfregola @DanielaSfregola!