Show HN: Htvend, a tool to capture internet dependencies

htvend is a tool to help you capture any internet dependencies needed in order to perform a task.

It builds a manifest of internet assets needed, which you can check-in with your project.

The idea being that this serves as an upstream package lock file for any asset type, and that you can re-use this to rebuild your application if the upstream assets are removed, or if you are without internet connectivity.

To build just htvend, you need Go installed and then:

Here's a simple example:

Output:

Creates assets.json in your directory, with contents:

If you now disconnect your internet (for example turn wifi off on your laptop), you can run:

and the same contents are output but without making upstream connections to the internet.

But if instead you run:

Then a 404 will be returned, and a log message printed:

To package up the assets (e.g. to transfer to a different environment), run:

and this creates:

When invoked as htvend build or htvend offline it creates a local HTTP and HTTPS proxy server on a dynamic port, with a self-signed certificate.

It then runs the specified child process with appropriate environment variables specified. For example:

Shows:

When the proxy server receives a URL that is found in assets.json, then that content is served, along with any relevant headers in that file.

If it isn't found, then if invoked as htvend build, it will be fetched from upstream, or if invoked as htvend offline, a 404 not found response will be served.

This is useful for a number of reasons, including:

1. Some environments (such as air-gapped networks) don't have internet access. Here you can supply a directory of blobs instead.
2. Assets on the internet often change, and not always on a schedule that supports your team.
3. Assets on the internet can become unavailable due to commercial, geopolitical or other reasons (e.g. Dockerhub rate limits), or a maintainer simply deleting their repository.

Perhaps most importantly, this lets you accept changes on your schedule. If you have to make a small change to a script that lives inside of an image to address a production issue, this makes it easy to make that change without inavertently bringing in additional changes due to other upstream changes that are pulled in via an otherwise uncontrolled image build process.

This is the main tool built by this repo.

Runs the passed subprocess to create/update assets.json in your current directory.

After setting up a proxy server with a self-signed certificate, it will set the relevant environment variables and execute a sub-command. If none specified, an interactive shell will be made.

See make assets.json for an example that creates a manifest for the dependcies of this project.

This runs the specified sub-porcess with a proxy which only serves the contents referenced in assets.json. Anything else will return a 404 not found error.

make offline does this for this repository.

If you have unshare installed, then a good way to really verify that you are offline can be as follows:

The unshare -r -n runs the sub-command in a new namespace with no networks. The ip link set lo up creates a loopback interface in that empty namespace so that htvend can create a server that it's sub-command can then hit.

By default all blobs are saved to and retrieved from ${XDG_DATA_HOME}/htvend/cache/blobs (XDG_DATA_HOME defaults to ~/.local/share).

A cache assets.json is also saved at ${XDG_DATA_HOME}/htvend/cache/assets.json, and this is useful during rebuilds of assets.json to avoid needing to connect to upstream servers more than neccessary.

This copies all cached blobs referred to by assets.json to a directory of your choosing. This is useful when packaging your assets to send to another environment (which may not have internet access).

make blobs runs this for the assets.json file in this repo and creates the blobs directory.

Iterates through all referenced and confirm they exist locally and with the correct SHA256.

If --fetch is set, it tries to fetch anything missing.

If --repair is set, then the local manifest is updated if the content has changed since.

make blobs runs this for the assets.json file in this repo, it also runs htvend export.

Removes any dangling blobs (ie not referred to by global assets.json cache) from global cache blobs directory.

Pass --all to remove entire global cache.

Yes. Packaging software into OCI Images is a very useful way to distribute software.

Further using a Dockerfile to populate assets.json is an excellent way to ensure that a build is done from scratch (that is, it pulls through all needed assets) and thus is a great way of producing a canonical assets.json file.

See OCI Image Building for details on how to use this.

Yes it is. We use the assets.json in this repo as an example only - not all languages are as good as Go.

Shipping built images around might work well for your use-case.

This tool recognises that many projects end up being a combination of public upstream images / packages / assets and private application source code.

The intent is to help make it easier to make changes to the private application part without pulling in any other changes from the internet.

Yes, they likely can. However they can be tricky to setup and may require specialist configuration for each package type (e.g. Maven vs Docker vs apt vs Python) and modification of each Dockerfile to use.

This project tests the hypothesis that we can do this at a simple HTTP layer.

Yes. Please contact [email protected] for information and pricing for enterprise support by our Australia-based local team.