Converting 600k lines to TypeScript in 72 hours

Background

Users have been using Lucidchart for a long time to make their diagrams, since 2010 and IE6. And for nearly that long, Lucid has used Google's Closure Compiler to manage its increasingly sophisticated code base. Closure Compiler is a typechecker and minifier that uses JSDoc annotations in comments to understand type information. As Lucid grew from 5 to 15 to 50 engineers, the tool proved enormously helpful for quality and productivity. Then in 2016, Lucid began experimenting with TypeScript. TypeScript offers more flexible and comprehensive type checking, cleaner syntax, and better IDE support than we could get with Closure-annotated Javascript.

// Closure JavaScript

/** @const {!goog.events.EventId<lucid.model.DocumentEvents.PagesContentChanged>} */
lucid.model.DocumentEvents.PAGES_CONTENT_CHANGED =
    new goog.events.EventId(goog.events.getUniqueId('PAGES_CONTENT_CHANGED'));

/**
 * @constructor
 * @extends {goog.events.Event}
 * @final
 * @param {!Object<boolean>} pages
 * @struct
 */
lucid.model.DocumentEvents.PagesContentChanged = function(pages) {
    lucid.model.DocumentEvents.PagesContentChanged.base(
        this, 'constructor', lucid.model.DocumentEvents.PAGES_CONTENT_CHANGED
    );
    /** @const {!Object<boolean>} */
    this.pages = pages;
};
goog.inherits(lucid.model.DocumentEvents.PagesContentChanged, goog.events.Event);

is equivalent to

// TypeScript

export const PAGES_CONTENT_CHANGED: EventId<PagesContentChanged> =
    new EventId(events.getUniqueId('PAGES_CONTENT_CHANGED'));

export class PagesContentChanged extends Event {
    constructor(public readonly pages: {[key: string]: boolean}) {
        super(PAGES_CONTENT_CHANGED);
    }
}

TypeScript was well received among our engineers, and by summer 2017, Lucid had 100k lines of TS and 600k lines of Closure-typed JS. To bridge various parts of the codebase, we used clutz (converts Closure-annotated JS to TypeScript declarations) and tsickle (compiles TypeScript to Closure-annotated JS). While the benefits of TypeScript were clear, it looked like it would be years before we could entirely replace our JavaScript with TypeScript.

Idea

Every summer, Lucid has a two-day hackathon during which Lucid employees work on an interesting project of their choice. The hackathon presented a unique opportunity to migrate to TypeScript while no production-ready code was being authored. Compared to an active migration, a stop-the-world migration could be much faster and less overall effort.
Google has an open-source tool called gents (generate TS) which converts Closure-annotated JavaScript to TypeScript. The tool has many caveats and limitations — bare-bones, idiomatic code translates well, but more unusual code and sophisticated language features (e.g. generics) do not. Initially, gents crashed with runtime errors on Lucid’s codebase. A few patches got gents to about 80% of what we needed. Ad-hoc pre- and post-processing scripts added an additional 10%. Even then, the remaining 10% was still enormous in absolute terms and would possibly be too large to complete in a couple days. We asked our CTO to join in the migration. He supported the idea but preferred working on a project more within the realm of possibility.

Six of us engineers decided to try anyway.

Plans

A very challenging aspect of the migration would be the lack of feedback. Computers are picky things; 70% of code working looks a lot like 0% working. While it would be difficult to get incremental runtime feedback, we decided to at least get incremental compile-time feedback. We would create a dependency graph of the 2000+ files and start working from the leaves, moving each file to a new source root, fixing the type errors, and committing it. At any given time, we would have a set of successfully compiling TS files. To parallelize this effort across the six engineers, we constructed a 2,840-line spreadsheet with each files, its status, and its dependencies. Once a file's dependencies were marked completed, the file would be color coded as ready for work. One of us would self-assign it, move it, fix it up after automatic translation, and mark it as done. Then that process would repeat on the remaining files.

If you would like to try using this example, make a copy of the Google Sheet.

Hackathon

To process all 2,840 files, we needed to complete one file per minute for 48 hours straight. The six of us worked around the clock, sleeping only a few hours during the two-day hackathon. Some files had cyclical dependencies (allowed, as long as some of the dependencies are type-only rather than value dependencies), and we frequently had to tackle large groups of files all at once. Velocity varied. Twenty hours in, prospects looked bleak. This was disappointing, as partial success for this hackathon project was the same as no success—there would be too many conflicts if this project had to be extended into a normal working day. There were several recurring challenges: 1. Base constructor requirements are different between Closure JS and TS. Complex, overgrown inheritance trees with tricky initialization semantics were challenging to re-architect. 2. TypeScript is more aggressive than Closure Compiler at identifying errors. For example, tsc detected the assignment-vs-comparison mistake that Closure Compiler did not:

var usernameOnTeam = (alreadyOnTeamFilter['usernameOnTeam'] || []).map(onTeam =>
    goog.array.find(me.loadedUsers, user => user.username = onTeam.username);
);

Naturally, in a legacy system, it’s hard to say what is the bug and what is the unexpected feature. Therefore, we generally left the code as is and added a workaround for the typechecker. No one is proud of this, but we had a VS Code macro to insert `as any` for highlighted expressions. 3. Closure JavaScript predates the ES2016 module system by many years. Traditionally, it uses `goog.provide` and `goog.require` to identify dependencies between files, and all functions are added to a global namespace. Idiomatic TypeScript typically uses modules and imports, which was a substantial paradigm shift. For example, Closure JS permits odd constructs like multi-file classes and implicit circular dependencies, but these do not work with TS imports. 4. The autogenerated imports from #3 meant we often had shadowing conflicts with existing local identifiers. And thanks to #2, we often missed this and mistakenly forced tsc to accept broken code. Despite these challenges, after thirty-six hours, it looked more likely that we might finish, though it was impossible to be sure. Perhaps it would all compile, but we’d encounter a hopeless wall of errors at runtime. At forty-four hours, we met with engineering leadership to discuss the project’s potential. The stakes were significant. On the one hand, a wholesale rewrite of the entire codebase could harm the company's success. On the other hand, if it wasn't completed now, it would take years to reach this point incrementally. At forty-six hours, we partially loaded the main Lucidchart document list and editor. They were definitely broken, but we had proof of life within the two-day hackathon. That was enough. The hackathon ended on a Friday, and with the bulk of the work completed, we worked through the weekend on rounding out the build process and getting the primary parts of the product working well enough that it would not impede the 60 engineers returning to work on Monday. Monday at 9 a.m., we pushed. 600k lines of typed JS became 500k lines of TS.

Release

Of course, the story was far from over. We still needed to ship it. Lucid releases every two weeks and has not missed a release for over four years. The next two weeks were spent getting secondary parts of the product functioning, unit tests passing, tree-shaking and minification working (still using the Closure Compiler), and CI running. QA made multiple passes, finding dozens of bugs. When release day came, every team had a member on call in case of problems. Though we had worked hard to ensure everything would run smoothly, we anticipated something within the half a million lines of a seven-year-old code base to go wrong. We waited and waited. And the issues never came. The combination of unit tests, manual testing, and a new, very robust type system bought us a TypeScript migration with zero customer-facing issues.