Clear your codebase with fundamental info concept

Reduce out all the things that’s not shocking.

– Derek Sivers

Shock

The next equation measures
“surprise”:

It’s straightforward to estimate the shock of textual content information:

const rx = /[s,][()]+/g;
const counts = {};
for (const file of Deno.args)
  for (const x of (await Deno.readTextFile(file)).cut up(rx))
    counts[x] = 1 + counts[x] || 1;
const whole = Object.values(counts).cut back((a, b) => a + b, 0);
let entropy = 0;
for (const c of Object.values(counts)) {
  const p = c / whole;
  entropy -= p * Math.log2(p);
}
console.log(entropy);

Be aware: this script calculates shock on the degree of phrases. You are able to do
related estimates on the character degree, but it surely’s not as helpful for enhancing
readability.

Operating the script on a number of random markdown information:

$ ./entropy src/sufficient.md
6.371408749206546

$ ./entropy src/two-toucans-canoe.md
6.452173798618295

$ ./entropy src/sufficient.md src/two-toucans-canoe.md
7.1796868604897295

$ ./entropy src/music.md
8.172144309119338

$ ./entropy src/pardoned.md
8.180170381663272

/enough, /two-toucans-canoe, /music,
/pardoned

In case you have a look at the examples, you’ll see that entropy usually will increase with
file measurement, excluding information with loads of repetition, e.g.
my music ratings.

Compression

To compress textual content on the word-level, substitute repeated phrases with shorter
phrases, e.g. exchange “for the aim of” with “for”. Be aware that changing
“gargantuan” with “huge” might enhance shock on the character-level, however not
essentially on the word-level.

This can be a consequence of
Shannon’s source coding theorem.
The theory additionally offers calculable upper-bounds for textual content compression.

In concept, you would use one thing like
Huffman coding to shrink the
measurement of your code.
Minifiers use
associated methods to provide unreadable-yet-valid gibberish.

You can too
estimate upper-bounds for compressed programs.
Implement one thing like gzip in a goal language and extract this system
with its decompressor to a brand new file. The variety of bits within the
self-extracting archive
approximates the dimensions of the smallest doable program.

In observe, compressing packages for people means changing bigger snippets
with smaller features, variables, and so on.

Right here’s an instance from a real-world Elm file with 2.2k LOC:

$ ./entropy BigProject.elm
8.702704507769093

# `Css.hex "#ddd"` -> `ddd`
$ sed -i 's/Css.hex "#ddd"/"ddd"/g' BigProject.elm

# much less repetition creates extra shock
$ ./entropy BigProject.elm
8.706398741147163

How would this quantity change over the lifetime of a undertaking? I’d like a script to
evaluate the full entropy of all information in a repo at every launch and throw it in
a chart. Please electronic mail me at [email protected] for those who
have any implementation options.

To search out compression candidates, search for frequent sequences of phrases, i.e.
code collocates.

Though it’s doable to automate long-sequence identification, I want to
checklist frequent phrases after which compress in conceptual chunks. This script
calculates frequencies on the word-level, but in addition weights phrases primarily based on their
character size:

const rx = /[s,][()]+/g;
const len = {};
for (const file of Deno.args)
  for (const x of (await Deno.readTextFile(file)).cut up(rx))
    len[x] = x.size + len[x] || x.size;
console.log(
  Object.entries(len)
    .filter(a => a[1] >= 10 && a[0].size > 2)
    .type((a, b) => b[1] - a[1])
    .map(a => `${a[1]} ${a[0]}`)
    .be part of("n")
);

Operating the script on my Elm undertaking:

$ ./freq-lengths BigProject.elm
1467 Attrs.css
1295 Css.rem
732 Css.fontSize
616 Html.div
594 Html.textual content
570 Css.backgroundColor
455 Css.hex
391 Possibly.withDefault
360 import
354 Css.px
336 Css.fontWeight
322 Css.lineHeight
304 Css.borderRadius
297 Css.shade
297 Html.span
273 Css.marginTop
266 Nothing
252 String
231 Css.int
225 Css.stable

Based mostly on these outcomes, my first intuition is to drag out among the CSS stuff
into helper features, particularly for font-related styling. C’est la vie.

Readability

Code golf is entertaining, however my boss
can be pissed to discover a program like
“t,ȧṫÞċḅ»Ḳ“¡¥Ɓc’ṃs4K€Y in our
repo.

Whereas
readability formulas exist for text,
“readable code” stays controversial.

In my expertise, the important thing to sustaining readability is growing a wholesome
respect for locality:

1. coarsely construction codebases round CPU timelines and dataflow
2. don’t pollute your namespace – use blocks to limit
   variables/features to the smallest doable scope
3. group associated ideas collectively

The toughest a part of this course of is deciding what “associated ideas” imply. For
instance, must you group all CSS in a single file, or must you group types subsequent
to their HTML buildings? Do you have to group all of your database queries collectively,
or inline them the place they’re used?

I don’t have all of the solutions, but it surely’s abundantly clear that we now have a
thirty million line problem and
that
complex ideas fit on t-shirts.
Once I see
an entire compositor and graphics engine in 435 LOC,
I count on many surprises from the future of coding.