Contributing Scrutiny to nixpkgs · Jon Seager

2024-02-24 02:39:55

Introduction #

I’m fairly overwhelmed by the quantity of people that learn, and engaged with my
last post on packaging
Scrutiny for NixOS. I hadn’t supposed on posting to this weblog greater than possibly as soon as per 30 days, however I made a decision to proceed documenting the journey of touchdown Scrutiny upstream in
nixpkgs.

One of many issues that basically struck me about NixOS early on was how easy the contribution course of is. A single Github repository incorporates all of the packages, all of the modules, and all of the assessments.

I’ve discovered the group extremely useful and respectful in all of my contributions – typically with 2-3 folks rapidly offering evaluations to assist form the contribution. I’ve heard complaints that evaluations might be fairly pedantic, and whereas I conform to some extent, the evaluations almost all the time include concrete recommendations that are an effective way to study. However the truth that you might be contributing to an working system package deal that would land on many hundreds of machines – it’s vital to watch out!

When you’ve been deliberating about whether or not or to not submit your package deal, module or in any other case, I’d encourage you to go for it. I’ve all the time discovered it rewarding and it’s taught me quite a bit about Nix and NixOS.

My
first contribution was including a package deal, module and take a look at for
multipass – as you’ll be able to see I had lots to study, however folks had been beneficiant with their time and it resulted in a contribution that I exploit every day to get my work achieved on NixOS.

For these dropping sleep over the poorly disk I confirmed in my final submit, you’ll be relieved to know that it’s now been changed! By the way this was the primary time I’ve changed a disk in a ZFS array and I used to be tremendous impressed with how simple the method was!

healthy disks

Bettering My Work #

Over the weekend following my first submit about Scrutiny, a few folks recognized some small points and made pull requests to resolve them – thanks to these folks! You’ll be able to see their work
here,
here and
here. Somebody additionally
pointed out to me that string-interpolating YAML wasn’t essentially the most very best resolution to rendering the configuration file, and the I might have used
builtins.toJSON, profiting from the truth that JSON can be legitimate YAML.

I additionally needed to make it possible for there was a manner for folks to make use of the module with an present InfluxDB deployment. My unique implementation assumed that it might simply add companies.influxdb2.allow = true to the system configuration and use the default credentials to attach. I added
some options to the module for configuring a distinct InfluxDB occasion if required. These had been later modified barely to make use of lib.varieties.nullOr lib.varieties.str following a useful
review comment.

Subsequent up was refactoring the helper perform I wrote for producing the configuration file. As talked about above, Nix comes with a helpful
builtins.toJSON perform which takes a local Nix expression (an
attribute set on this case) and renders it to JSON. This obviates the necessity to string-interpolate YAML, which is error inclined at the most effective of occasions. The renewed perform seemed like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  mkScrutinyCfg = cfg: pkgs.writeTextFile {
    title = "scrutiny.yaml";
    textual content = builtins.toJSON {
      model = 1;
      net = {
        hear = {
          inherit (cfg) host port;
          basepath = if cfg.basepath != "" then "/" + cfg.basepath else "";
        };

        database = {
          location = "/var/lib/scrutiny/scrutiny.db";
        };

        src = {
          frontend.path = "${pkgs.scrutiny}/share/scrutiny";
        };

        influxdb = {
          inherit (cfg.influxdb) scheme host port token org bucket;
          tls.insecure_skip_verify = cfg.influxdb.tlsSkipVerify;
        };
      };
      log = {
        stage = cfg.logLevel;
      };
    };
  };
}

I additionally observed that I had hard-coded the trail to the binaries within the rendered systemd items. You would possibly recall that every Nix package deal incorporates meta block, which provides details about the package deal such because the license, maintainer, and so on. For every of the packages,
I added a mainProgram attribute which permits the usage of one thing like lib.getExe pkgs.scrutiny fairly than ${pkgs.scrutiny}/bin/scrutiny.

1
2
3
4
5
6
7
{
  # ...
  meta = {
    mainProgram = "scrutiny";
    # ...
  };
}

Lastly, I needed to simplify the packaging of the Go applications barely. Within the earlier submit I used buildPhase to override the default construct behaviour of the buildGoModule perform. I later found the subPackages attribute, which enabled me to attain the identical impact with out overriding the method manually. You’ll be able to see the commit with all of the adjustments
on Github, however under is a (barely modified) preview of the simplified collector derivation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
{ pkgs, lib, ... }:
pkgs.buildGoModule rec {
    # (supply, title, model, vendor hash omitted)

    subPackages = "collector/cmd/collector-metrics/collector-metrics.go";

    buildInputs = with pkgs; [ makeWrapper ];

    CGO_ENABLED = 0;
    ldflags = [ "-extldflags=-static" ];
    tags = [ "netgo" "static" ];

    installPhase = ''
      mkdir -p $out/bin
      cp $GOPATH/bin/collector-metrics $out/bin/scrutiny-collector-metrics
      wrapProgram $out/bin/scrutiny-collector-metrics 
        --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
    '';

    # (meta block omitted)
}

Contributing To nixpkgs #

With these adjustments made, it was time to open a pull request. The NixOS group present some
contributing guidelines that are value a learn earlier than you begin.

There was a structural transition within the pkgs listing recently, so this was my first contribution below that new construction. There was a
detailed talk about that transition at Nixcon in the event you’d like extra info.

I wanted to do the next:

  • Create the scrutiny and scrutiny-collector packages
  • Add the scrutiny module
  • Add the assessments
  • Replace the discharge notes for the following launch of NixOS

Which translated into the next stack of commits on
PR #289934:

commits on nixpkgs pr

There have been a couple of different minor adjustments, comparable to how enter packages (like smartmontools and makeWrapper) are handed to the packages. Within the nixpkgs repository, packages are handed on to the derivations, fairly than pkgs being handed as a top-level argument and packages being referred to by pkgs.smartmontools and so on.

I additionally discovered one thing new about module configuration. Within the first iteration I had set the default worth of companies.scrutiny.collector.allow to the worth of companies.scrutiny.allow, that means that the collector could be enabled mechanically when Scrutiny was enabled if no different configuration was specified. It transpires that this isn’t permitted in nixpkgs, and resulted in
this CI failure. I up to date the module with a refined API change, that means that customers of the module will now must do the next to get the identical behaviour.

1
2
3
4
5
6
{
  companies = {
    scrutiny.allow = true;
    scrutiny.collector.allow = true;
  };
}

Overview Suggestions #

From the primary overview I discovered concerning the
lib.types.nullOr meta-type which permits the specification of optionally null configuration choices in a NixOS module.

What adopted was
a suggestion to embrace a distinct configuration format in keeping with
RFC42: a proposal for structured module configuration which is extra rigorously sort checked, and extra versatile as new choices are added to the underlying workload.

Many NixOS modules resolve this downside immediately with choices like extraConfig or configLines, which append strings to the top of any options-generated configuration file. What RFC42 proposes is a hybrid strategy whereby sure choices are well-defined (and thus characteristic
in the documentation), however extra choices might be injected as wanted. The has the good side-effect that modules don’t must assist dozens of choices which might grow to be a burden to keep up and take a look at over time, however customers can nonetheless make use of latest options and choices which are launched to the underlying workload.

You’ll be able to see the total ensuing choices definition
on Github, and small annotated excerpt under:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  companies.scrutiny = {
    allow = lib.mkEnableOption "Allows the scrutiny net utility.";

    # ...

    settings = lib.mkOption {
      description = lib.mdDoc ''
        Scrutiny settings to be rendered into the configuration file.

        See https://github.com/AnalogJ/scrutiny/blob/grasp/instance.scrutiny.yaml.
      '';

      default = { };

      # That is the important thing to RFC42 - defining `settings` as sort `submodule`.
      sort = lib.varieties.submodule {
        # This ensures that attrsets, lists, and so on. are rendered as JSON
        freeformType = (pkgs.codecs.yaml { }).sort;

        # The next are examples of "well-defined" choices with
        # formal varieties and metadata.
        choices.net.hear.port = lib.mkOption {
          sort = lib.varieties.port;
          default = 8080;
          description = lib.mdDoc "Port for net utility to hear on.";
        };

        choices.net.hear.host = lib.mkOption {
          sort = lib.varieties.str;
          default = "0.0.0.0";
          description = lib.mdDoc "Interface deal with for net utility to bind to.";
        };

        # ...
      };
  };
}

This permits the next person configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
  companies.scrutiny = {
    allow = true;
    settings = {
      # These choices are well-defined within the module with varieties,
      # examples, descriptions and are rendered within the docs.
      net.hear = {
        port = 8080;
        host = "100.64.12.34";
      };

      # These aren't, however can be rendered nonetheless as JSON and allow
      # folks to utilize choices supported by Scrutiny, however
      # indirectly by the module.
      notify = {
        urls = [
          "discord://token@channel"
        ];
      };
    };
  };
}

I used to be impressed by the simplicity of the strategy, and subsequently adopted the identical for the collector, which now has the flexibility to take configuration by the
services.scrutiny.collector.settings attribute.

One facet of Scrutiny’s design that helped is the flexibility to take configuration both by from file, or by surroundings variables, or each! I selected to provide some mounted choices
as environment variables to the systemd companies, and permit the remainder of the configuration to be set utilizing the generated file.

See Also

@JulienMalka reviewed subsequent, suggesting adjustments to make sure that varied preInstall and postInstall hooks had been being referred to as the place I used to be overriding installPhase, although what adopted from
@katexochen’s overview was a simplification that decreased this want barely.

@katexochen prompt a few good adjustments, amongst which was the flexibility to drop explicitly mentioning the netgo tag within the Go packages, since CGO_ENABLED is disabled already, this means utilizing the Go inner model of many libraries. In addition they
suggested shifting the set up of the frontend items to postInstall, fairly than overriding the usual installPhase.

And eventually
@SuperSandro2000 reviewed with some naming adjustments to simplify the package deal definitions and cut back repetition. I additionally observed that I used to be unnecessarily declaring the frontend definition utilizing the
rec keyword. As soon as these had been mounted the PR was merged!

Contribution Suggestions #

I’m nonetheless comparatively new to this – you’ll be able to see that my
list of contributions will not be in depth, however I’ve been by the method of including, updating and refining packages and modules a couple of occasions.

Take this recommendation for what it’s, however I hope you discover it useful:

  1. Learn the
    contribution guidelines
    : That is in all probability the most effective begin. Attempt to observe the steering as greatest you’ll be able to, and check out another
    recently closed Pull Requests to see in the event you can decide up on any patterns.
  2. Be humble: Everyone seems to be new after they begin. You would possibly discover some evaluations just a little terse. Relaxation assured that that is solely as a result of the maintainers are extremely busy. Assume good intentions, stay well mannered and respectful. Nixpkgs is without doubt one of the
    busiest Github repositories I’ve seen, and it’s largely maintained by volunteers who typically have restricted time to present for an enormous variety of contributions.
  3. Be affected person: Don’t count on a overview instantly. When you’re searching for a preliminary overview, be happy to ping me
    on Mastodon and I’ll attempt to supply any steering I can.
  4. Use the Discourse: Due to the amount of PRs, the group run ongoing “PRs Prepared for Overview” threads
    on Discourse which you should utilize to draw reviewers in case your PR has been stale some time. It’s additionally an awesome place to
    ask for help.

As soon as merged, commits trickle by the varied branches of the repository. You’ll be able to observe the progress of your PR in that course of as soon as it’s merged utilizing the
PR Tracker. You’ll be able to see an instance of this by
tracking the Scrutiny PR:

nixpkgs pr tracker

Abstract & Thanks #

My Scrutiny packages, module and assessments landed in nixpkgs roughly 4 days after I submitted the pull request. As ever, I acquired a bunch of useful evaluations from the group, and Scrutiny is now simpler than ever to devour on NixOS.

You’ll be able to see the completed elements right here:

When you had been beforehand utilizing my flake to devour Scrutiny, you’ll now see a deprecation warning. Emigrate over, import the module from unstable in your flake and be certain that you explicitly allow the collector:

1
2
3
4
5
6
{
  companies = {
    scrutiny.allow = true;
    scrutiny.collector.allow = true;
  };
}

Earlier than I wrap up, I’d wish to thank all these individuals who took the time to overview the PR, and particularly
@JulienMalka and
@RaitoBezarius who’ve helped me by numerous contributions over the previous months and been extremely aware of my questions.

Give it a go and let me understand how you get on!

Source Link

What's Your Reaction?
Excited
0
Happy
0
In Love
0
Not Sure
0
Silly
0
View Comments (0)

Leave a Reply

Your email address will not be published.

2022 Blinking Robots.
WordPress by Doejo

Scroll To Top