Your app is slow. It does not spark joy. This post will show you how to use memory allocation profiling tools to discover performance hotspots, even when they’re coming from inside a library. We will use this technique with a real-world application to identify a piece of optimizable code in Active Record that ultimately leads to a patch with a substantial impact on page speed.

Some years ago, vim expert Tim Pope tweeted a clever suggestion for using bin/:

PATH=".git/safe/../../bin:$PATH" and do mkdir .git/safe in the root of repositories that you trust.

When we originally read that, we liked the suggestion so much that we incorporated it in our workflow and into some of our tools. Read on to know more.

Why would I want to do this?

If you’re a Rails developer you’ve definitely seen a bin/ directory in your project root. It contains programs like bin/rails and bin/rake and bin/rspec. You may use these commands instead of bundle exec <command> to ensure you’re getting the correct versions of gem dependencies. bin/rails and the other binstubs in the bin/ directory eliminate the need for bundle exec. But, it’s nicer to be able to just type rails instead of bin/rails or bundle exec rails. There are a number of workarounds described in the But I Don’t Want to bundle exec blog post.

Even if you’re not a Rails dev, other projects might have a handy bin/ directory with helper programs for developing the app; for example, the gitsh project has a bin/gitsh helper program that only works while developing the project.

Where do I start?

Well, you could add the relative bin directory to your $PATH:

% PATH="bin:$PATH"

The downside is that this means that anywhere you cd, ./bin will be in your path. This may be fine most of the time, but devs who navigate around other people’s codebases might not want to find a bin/ls lurking somewhere under node_modules, waiting for you to cd in while investigating and debugging some third-party library.

The trick described in Tim’s tweet is to add bin to your $PATH through a directory you control, like so:

% PATH=".git/safe/../../bin:$PATH"

Whenever we vet and trust a project’s directory and their developers, we can add that project’s bin directory to our $PATH:

% mkdir -p .git/safe

In practice

# Initial configuration
% PATH=".git/safe/../../bin:$PATH"

# without having trusted a project, we fall
# back to the default `rspec` in our ruby 
# version manager (asdf in this case)
% cd work/project_with_bin_dir/
% which rspec
/Users/home/.asdf/shims/rspec

% mkdir -p .git/safe # we decide we trust this repo
% which rspec
.git/safe/../../bin/rspec # great! now we can just run `rspec`
% rspec # no need to `bundle exec rspec`

Built into thoughtbot’s dotfiles

If you’re using thoughtbot’s dotfiles, you’ll find that you already have a handy way of trusting your project. You can just run:

% git trust-bin

This will create the .git/safe directory for you. You can find some more information about this script in the pull-request that created it.

Built into thoughtbot’s Rails template, Suspenders

If you use thoughtbot’s Suspenders as a template to create a new Rails project, you get a bin/setup script that can be used to bootstrap the project onto new machines.

This script checks to see if you have .git/safe in your path. If you do, you’ll get a helpful message reminding you to create the .git/safe directory to trust the repo.

-> When you trust this repo, remember to run: mkdir -p .git/safe

Other uses for .git/safe

While we’re talking about easily using the bin/ directory here, this technique is also useful for files that are not in bin. In Chris Thorn’s dotfiles, you’ll find an example of trusting the git repo to maintain a vimrc.local and reading that in from a vimrc.

if filereadable(".git/safe/../../vimrc.local")
  source .git/safe/../../vimrc.local
endif

In my previous post I argue that JavaScript is by default a low trust environment for programming, that ideas that build trust like optional typing, functional transformations, and immutability have severe trade-offs in JavaScript, and that choosing one of these ideas often means rejecting another. Despite that, JavaScript is a better environment than most popular languages for building trust, and the broader argument applies to most popular languages, typed or not.

In PureScript you get types, immutability, and functional programming “for free”– the trade-offs aren’t as steep. This is largely true of a few other languages as well, but let’s see what our discussion of fear and trust from the previous post looks like in PureScript from the same two perspectives: understanding the shape of data and changing data.

Fear and the shape of data

PureScript starts with a high level of trust in the data by default. What is the shape of our data? Whatever the type says it is.

Here’s what the JavaScript example from the previous post might look like in PureScript. Like the JavaScript version, we load user data from the network and later render it. Our loadUser function takes in a user ID and returns an asynchronous effect– think of Aff like a Promise– containing either the validated user or errors. We then take this response in loadAndRenderUser and either render the user’s name, if we received a user, or log the errors if not.

The differences with JavaScript can be subtle. Everything is required to be typed within the PureScript code, so whenever you see the User type, you know it has exactly what the type says: an ID, optionally an email address, and a name. Everything is also immutable unless the type tells you otherwise, so you know this data won’t change underneath you. There are no null values. There are no any types. The value is what it says on the label. The compiler will tell you if you’re using it wrong. You can simply trust the data has the shape you expect, and will stay that way, and stop worrying about it.

What about data you get off the wire? Validation in this example happens with the call to readJSON. This takes in something with an unknown shape – a String – and attempts to turn it into a known shape – the User. It returns either the parsed user data or errors that occurred in validating the data. This is the default way of handling data that comes from outside, and this means that after this point you can trust that the data has the shape of a User. You’re not trusting the external service to provide correct data, because you validate the data as it comes in. You’re also not trusting developers to write correct types – if they write incorrect types, the validation will fail, and you will notice that.

Here’s the boilerplate for validating the data:

In most cases you don’t actually need to write any type validation logic. The compiler and library write it for you. If you have special needs, you can also write out this logic in a straightforward way manually.

The point is, as a developer you can simply trust that the data is shaped how the type says it is. What does this data look like? Does this field exist? Will changing this break the code? Read the type. Change it and let the compiler or a validation error tell you if it breaks something.

Fear and changing data

What about when the data changes? By default in PureScript all data is immutable, so instead of writing code like the mutable example from JavaScript, you might write this:

This looks a lot like the JavaScript example in the previous post using functional conventions. You take in data. You return new data. But the similarity is deceptive. This tells you much more than the JavaScript code because of the way PureScript restricts the language.

This looks just like the JavaScript code with functional conventions, but:

• It can’t mutate the data.
• It can’t modify your file system.
• It can’t launch rockets.

They’re not in the type. The type of the function is

This says you take in a SourceDoc and a String and return a Document. It can’t return anything else. It can’t return null. It can’t do other side effects like changing the data or launching rockets. This is a stupidly simple way of thinking about functions and types. A function takes in something and returns something and does nothing else.

Suppose you actually do want to mutate the data. You could write it like this:

The type of our formatDocument function changed:

Now the function takes in a mutable reference to a document, and a string, and returns something – an effect. The effect has a type Eff _ Unit, which means that when this effect is run, it will do some side effects and return an empty value (Unit).

This is similar to the JavaScript mutation example, but with key differences. We can change the data we receive (the SourceDoc), but only that data – we still can’t change any other data in our program. We also can only change the data in ways that respect the developer’s trust. For instance, we can’t give the data a new field that is not in the type, because that would mean the type is wrong.

Suppose you actually want to launch a rocket from this function. You might write it like this:

Now our function type is

Because launching a rocket is a side effect, the function needs to return an effect. The launching of rockets is explicit in the type – you can’t launch rockets without writing functions that return effects. Users of those effects have to recognize that the function returns an effect and handle it accordingly. This still doesn’t tell you what kind of effect will be performed when this is run – it could be doing many types of side effects. But it tells you clearly that the function does other things than just return a document. And the effect hasn’t actually happened yet. You might decide later that you don’t want to do the effect, and never execute it.

You could also cheat and write your document formatting mutably in JavaScript:

Then you’re closer to the JavaScript default, where you rely heavily on developers to maintain the shapes of your data. You’re not completely abandoning type safety – the PureScript compiler will still prevent you from changing the document within your PureScript code or using it in ways that are not supported by the type. But you’re weakening your level of trust in the code.

You probably don’t need to do this. You probably don’t want to. But you could. The code starts with the assumption of types you can trust, pure functions, and immutability, and you can selectively weaken those assumptions as needed. Or you can make them stronger, by making your types more restrictive. You have the tools to do both.

In JavaScript, by comparison, the base assumption is that you can trust very little. You build and rebuild trust into your system every day with tools like good conventions, optional typing, functional programming, and immutable data. But the bar starts low, and in JavaScript those ideas can only take you so far.

Types supporting functional programming

There’s no need to choose between types, immutability, and functional transformations. In PureScript the ideas all support each other and are pervasive in the code and ecosystem. Here are two examples.

Types and plugging functions together

Many JavaScript developers adopt tools like Ramda to do data transformations in a more functional way, with immutable defaults and useful ideas like currying and function composition. To parse and transform a set of documents received over the wire you might write something like this:

Or the same thing using the proposed `pipe` operator:

But as these pipelines get larger and the number and complexity of transformations grows it becomes harder and harder to trust that the transformations are working correctly. In theory optional typing could help with this, and you could just write the above and have TypeScript or Flow make sure the piping lines up. In simple cases the type inference for this will probably work fine and you can do that. Other times it seems to work, but it quietly wiped out your types in the middle of the pipe. Or the type inference doesn’t work at all, and you end up writing something like this, with the code logic drowned out in type annotations:

This is still a relatively simple example, but in this and especially larger examples you might also want to clean this up in a functional style. You might rewrite it like this:

Even this in simple cases type checks in TypeScript or Flow. Throw in generics right now in TypeScript and even the simplest cases fail. Flow seems to fare better, but in my experience you still run into lots of little edge cases. The TypeScript and Flow teams have been improving their handling of these cases all the time, but at some point you run into more fundamental issues around e.g. how it is possible to do type inference, while maintaining JavaScript compatibility, and avoiding making the type system brittle or complex. Ultimately when you write a lot of code like this, using function composition or lenses or other functional constructs, you have given up on the optional type system. You either don’t use types at all there, or you add verbose type coercions everywhere to make it work.

In PureScript, you just write this:

Or you rewrite it like the second JavaScript example:

In both cases, the PureScript code looks a lot like the untyped JavaScript code, but everything is strongly typed. You can add type annotations if you like, but you don’t have to. The compiler tells you if your functions don’t line up. Most Ramda functions, even those that are difficult or impossible to type out in TypeScript or Flow, are either built in to PureScript or are trivial to write using more general tools.

More broadly, in PureScript the types help you make the piping line up for any types of transformations – data transformations, but also asynchronous network calls, or server middleware, or error handling, or config validations. When you change your code, the type system tells you whether what you wrote even makes sense with everything else that you wrote. Compared to TypeScript or Flow, PureScript is both more expressive and has better type inference. Together these mean you can write code like in a dynamic language, but keep the types. You can also use the types more easily in ways that are difficult or impossible in TypeScript or Flow, like when using function composition or lenses.

Types and immutability

PureScript uses standard JavaScript data structures under the hood, but with immutability enforced through types and the ecosystem. Libraries don’t mutate data, or if they do, it shows up in the type. There is no conflict between types and immutability – on the contrary, the types are necessary to guarantee immutability.

In JavaScript you might get immutability by adopting immutable persistent data structures, where structural sharing is used to reduce copying of data. In PureScript persistent data structures are just a performance optimization. You already have immutable data structures, which map to normal data structures in JavaScript. If you find that structural sharing would help your performance issues, you can adopt persistent data structures. Or you can write that code in JavaScript in a fast mutable style, while exposing an immutable interface to the rest of your code.

Trust and the ecosystem

Ultimately it is possible in PureScript to write the same kind of code as in JavaScript. You could even write it in JavaScript. The higher base level of trust in PureScript comes not just from the language and compiler, but also in part from strong defaults and conventions in the ecosystem.

Types are required and pervasive, and the defaults nudge you toward validating at the edges of your system. Unlike with TypeScript and Flow libraries, types live with the code that uses them, and when the types are checked, that code is checked, too. Of course, at some point many libraries wrap JavaScript libraries, and you are trusting the library developer to handle that accurately. But there are strong conventions and defaults around writing sound types, and the compiler and ecosystem help to support that. You can trust the types within your system, or if you can’t, the issue is likely in your JavaScript code or at the boundary between PureScript and JavaScript.

There’s a similar dynamic with immutability. It’s not that you couldn’t write mutable code in PureScript, or write it in JavaScript and call it from PureScript. But it’s usually much easier to do it immutably. Writing mutable code has worse ergonomics, and in some cases you would be fighting the compiler and the ecosystem. There’s a strong default of manipulating data in immutable ways, backed by the compiler. This means you can trust the data won’t change underneath you, or if it does, you see it in the types or look to the mutable JavaScript code.

Learning to code without fear

PureScript has many interesting and practical ideas, from pattern matching and ADTs to the utility of type classes to property-based testing and type level programming. But from a JavaScript perspective the biggest gains come from the simple ideas. Developers have worked hard to bring ideas like types, immutability, and functional transformations to JavaScript. They end up being a patchwork of useful tools that kind of work, if you apply them deliberately and avoid foot-guns and don’t use them too much together.

In PureScript, there’s no need to choose between these ideas. The ideas all support each other and are pervasive in the code and ecosystem.

What do pervasive strong types, immutability, and functional programming give you? A high base level of trust in the code that you write. A feeling of relative security. The confidence to refactor code freely as needed.

Programming without the fear.

The post Fear, trust and PureScript: Building on trust with types and functional programming appeared first on Reaktor.

As developers, we want to reduce fear of our code failing and increase trust in our code working well. Many developers working with JavaScript borrow useful ideas from functional programming and strongly-typed programming languages to reduce fear by transferring trust from developers to tools and the code. Ideas like optional types, functional transformations, and immutability can all help to write better JavaScript. When pulling these ideas together in JavaScript, however, they come with severe trade-offs, work together poorly, and ultimately fail in the goal of effectively transferring trust from developers to code and tools.

To illustrate this idea, let’s look at how data is handled in JavaScript from two perspectives: understanding the shape of data and changing data.

Fear and the shape of data

In a dynamic language like JavaScript, it can be hard to know what the shape of your data is. The default approach is to rely on convention. You trust other developers and other systems to give you correct data in agreed upon formats and to follow certain norms within the code base.

I like to call this the “pretend it’s what you want” approach. In high-trust environments, it can work well enough.

But then the fear creeps in. The code grows in complexity. You work with code from developers who follow different conventions. You receive data that you cannot control from upstream in erratic formats. You start seeing null pointer errors. Trust in the code breaks down, and questions about the data start to provoke anxiety rather than confidence.

• What values does this data actually contain?
• Can I delete these values without breaking things?
• Can I pass in this data to this function?

You can see the fear in the code base. It looks like this:

This is defensive programming. It happens when you can no longer trust your own code to provide the data you expect at the appropriate times. Your beautiful code becomes cluttered with defensive checks, you lose readability, and the code becomes more brittle and harder to change. Fear grows, and it is harder and harder to trust that your code actually works.

Optional types: Pretend really hard

One way to stave off the fear is to introduce optional types via TypeScript or Flow. You receive a user and then proclaim joyously that it is of the User type, and henceforth shall be treated only as a User.

This is like pretending really hard. You’ve shifted your trust around. You still trust other systems to give you data in the correct shape. But within your code base, you trust the type that you’ve given to that data and that the compiler will complain if you use that data incorrectly. Instead of trusting developers to know the shape of data and use it appropriately, you’re trusting developers to write and maintain correct types, and you’re trusting the compiler to not lie about those types. More on that later.

Adding types to our example doesn’t solve the underlying problem. It improves trust within the code base by helping to ensure that data is used consistently, but it says nothing about data received from the outside world.

Validation: Trust but validate

In a low trust environment, you may need to introduce data validation at various points.

You could do this by hand, but the validation would be ad hoc, laborious, and error-prone. Or you could write JSON schema definitions and validate with ajv or the like to verify that the data matches your schema. This is less ad hoc and allows other uses like generating documentation, but is likely no less verbose or error-prone because you have to manually write out schemas like this:

Optional types + validation

Or you could introduce both types and validation. Types to stave off fear internally, and validation to be able to trust data from external sources.

To avoid writing essentially the same type definitions for both validation and optional types you can use the TypeScript or Flow compilers directly as libraries, or use another library like runtypes (TS), runtime-types (Flow), or typescript-json-schema (TS). After going through a few hoops you start feeling more trust in your data. But there are deeper issues here, which I will get to later.

Fear and changing data

What about when the data changes? By default in JavaScript data can change willy-nilly. For example, this function receives a document, and then changes the document to format a field properly and to include a new field.

But in this style, the flow is hard to follow, and fear starts to creep in. What if our data is used elsewhere? What if it was already changed elsewhere? What values do I have in my data at this point? How can I trust that the data I have at this point is the data I want at this point and will stay that way? This is a trivial example, but the problem becomes much worse with a large code base or a highly concurrent system.

You turn to optional types, but those types won’t save you. In TypeScript and Flow, both of these functions have the same type:

One of these does what you want; the other burns the city down. As far as these type systems are concerned, these functions do nothing.

Convention: Pretend immutability

So you write better JavaScript. You agree with your team, explicitly or implicitly, to write in an immutable style.

You favor const over var and duplicating values over mutation. You use let to indicate value references that change. You rediscover the ternary operator as a functional alternative to if statements, at least for short lines. You use functions to return new values instead of changing values. You use map, filter, reduce, and other functional constructs to create new data structures without changing the underlying data.

Immutability by convention is convenient, produces idiomatic JavaScript, and works well within the JavaScript ecosystem. But it relies heavily on both trust and discipline from developers. You trust developers to follow conventions like avoiding mutation or indicating clearly where mutation happens. You might want something stronger.

Libraries: Pretend really hard

You can shift the trust partly from other developers to tools by adopting libraries for data transformation and immutable data structures. You might start using a library like Ramda pervasively as a functional utility belt, or adopt lenses à la partial.lenses, monocle-ts, or the like.

One fundamental idea in these types of libraries is that the underlying data is treated as though it were immutable. It’s not – even Ramda only does shallow clones – but if the convention of immutable data is strong enough, then everyone can pretend it is. You may take a slight performance hit from copying data, but you gain some level of trust in the code. This works best if the use of the library and this convention is pervasive.

To enforce actual immutability and avoid the performance hit for changing data, you might also introduce immutable data structures via something like Immutable.js, seamless-immutable or Mori.

This makes the data itself actually immutable, in that only immutable ways to touch the data are exposed. But it only applies to data that is expressed within these data structures. As most of JavaScript relies on classic JavaScript data structures, you end up converting back and forth between the two a lot and you lose that trust whenever you have to use standard data structures.

Both of these approaches have limitations, but most importantly they clash hard with optional types.

Trusting JavaScript

The previous examples pulled out several tools for writing more effective JavaScript: optional types, functional transformations, and immutable data. But in JavaScript these tools come with some severe limitations, and they are hard to use together.

Optional types give a false sense of security

Optional types for JavaScript are optional by design, which means not everything is typed and you can’t trust that everything has a valid type. Flow is unsound and TypeScript is deliberately unsound, which means that in various cases the types are wrong and the compiler doesn’t care.

And optional types in JavaScript lie for other reasons. Some things in JavaScript are just hard or impossible to type out in TypeScript or Flow.

• Higher order functions
  • Ramda’s call, compose, chain, lift, and lenses
  • partial.lenses’ functions
• Dynamic functions
  • Ramda’s invert, dissoc, mergeWith
  • Immutable.js’ pretty much anything
• Monkey-patched anything
  • AWS SDK client promises
  • Bluebird promisified APIs
• Highly dynamic data structures
  • ElasticSearch schemas
  • winston custom loggers

To type these out in TypeScript or Flow, you sacrifice on one or more principles:

1. Sacrifice type safety, the whole reason you use types: Type them out with any types, which allow any values and essentially disable the type checker for all values in the “path” of any.
2. Sacrifice usefulness: Make the functions less general in order to provide more specific, accurate types.
3. Sacrifice other developers’ time: Make the user of the function provide the correct types, as in

Then you add libraries into the mix, with their own type definitions with mixed levels of accuracy. This transfers some trust not to the developers of libraries, but to the developers of type definitions for libraries. Many of these libraries will contain any annotations, and calling those functions will quietly render your trust in types invalid. In Flow, type-checking can also be quietly disabled when a file is missing a @flow annotation.

You can work around this trust issue by adopting type annotations pervasively, disallowing both implicit and explicit any types, setting the linter to complain when files are not type-checked, and otherwise tightening up configurations.

But it’s like plugging holes in a leaky ship. The problem isn’t just that you can’t trust the types in your system, but that you think you can. You rely on the types to tell you when a change breaks something, but because they were quietly disabled by an any type, or by use of a library, or by a soundness issue, it doesn’t. Types in JavaScript are different from types in most other languages people use: They can’t be trusted in the same way.

Ultimately the strength of your types depends on the knowledge and belief of the team in applying them. If the team has a high level of belief and knowledge of types, they can encode a high level of trust into the system. But this is dependent on the team’s attention and discipline to maintain this level of trust, and fear can creep in and destroy that trust in many subtle ways.

Functional programming. Types. JavaScript. Pick two

Optional types and basic functional programming like maps and filters and reduces and so forth work alright together in JavaScript. It’s when you try to go further that you run into problems. Two examples:

Immutable.js is a persistent, immutable data structure library for JavaScript. It provides common data structures for JavaScript that do not rely on modifying the underlying data in-place. It has built-in type definitions for both TypeScript and Flow – go look at them. There are countless any annotations, which disable type-checking for those values. Then there are other types which pass the burden on to the user to provide the correct types. Essentially every time you use the library, you are either opting out of optional types or going to extra lengths to make the types work. This discourages functional programming.

Ramda is a functional utility library for JavaScript. One set of type definitions can be found here, along with this comment (emphasis added):

“Note: many of the functions in Ramda are still hard to properly type in Ramda, with issues mainly centered around partial application, currying, and composition, especially so in the presence of generics. And yes, those are probably why you’d be using Ramda in the first place, making these issues particularly problematic to type Ramda for TypeScript. A few links to issues at TS can be found below.”

Despite the impressive work of people like Giulio Canti, every time you choose even slightly more advanced functional programming concepts, like immutable data structures, function composition, or currying, you are essentially opting out of the type checker or going to extra lengths to make the types work. This discourages functional programming.

Why we can’t have nice things in JavaScript

Immutability works best when it is pervasive. But the JavaScript language and ecosystem is designed around mutable data, you can’t enforce immutability from a library, and optional types in JavaScript are not expressive enough to handle immutability as a library.

Types work best when they are pervasive. But types in JavaScript are optional by design and limit their expressiveness and utility by taking steep trade-offs to maintain compatibility with JavaScript.

Types, immutability, and functional programming can all support each other, just like they do in many languages. Types can be used to enforce immutability, even when the underlying data structures are mutable or the types don’t exist at runtime. Types can help developers connect the piping correctly when using functional composition or transforming data using lenses. Functional transformations can be easier to understand and maintain when you see the types. Functional transformations can be more efficient when you know the underlying data is immutable.

Learning to code with fear

So how do you learn to code with the fear? You write better JavaScript. You start with the base assumption that you can trust little in your code, and learn countless tricks to write more functional JavaScript and avoid the wartier parts of the language. You introduce type validation where necessary. You use immutable data, but only where you have a specific need or you enforce it by convention only. You adopt optional types where it makes sense, but abandon types where functional data handling or immutable data provide greater benefits. You use functional composition or lenses while knowingly opting out of type checking guarantees.

Or you change the game and just use PureScript. Or ReasonML, or Elm, or even ClojureScript. These exist today. Production software runs on them. They work with the JavaScript ecosystem, where necessary. And they provide a higher base level of trust in the code that you write and an environment where immutability, functional programming, and types (where applicable) work well and work together.

Adopting one of these languages is not going to solve all of your problems. It will introduce its own problems. But it might give you a higher level of basic trust in your code, and better tools to increase or decrease that trust as needed. In my next post, I’ll discuss how these ideas play together in PureScript.

But in JavaScript, the fear is always with you.

The post Fear, trust and JavaScript: When types and functional programming fail appeared first on Reaktor.