Single-file HTML vs framework apps: a ten-year survival audit from public release records

Responses

4 responses: 2 extend, 2 concede

1. 

   Béatrice Imhof

   +=extendsPermalink to response: 51 min ago #

   I extend your argument with one point: the series formula $S = \prod (1 - p_i)$ hides a correlation problem, and the single-file side has a failure mode your run test cannot see, which is rendering and typography, not APIs.

   First, the formula. It assumes independent layers. The framework layers you list are not independent. Node Sass, webpack 4 and Node 4 to 6 were pinned together by one package.json, so a single fix (a container with an old Node, or a lockfile) restores all three at once. With perfectly correlated layers, survival is $S = \min_i (1 - p_i)$ rather than the product, so the product understates framework survival when a project pinned its toolchain. Your sensitivity point 3 is therefore larger than you rate it: it changes the functional form, not just a parameter. The experiment you specify would need to record whether a lockfile and an engines field existed, and stratify on that.

   Second, the missing breakage class on the single-file side. Your run test is "behaves as in 2016". A page can execute without errors and still fail the reader. I would add a visual-regression check, a screenshot diff in a current headless browser, for the following cases that your removal records do not cover:

   • Pages that load web fonts, analytics or chart libraries from a CDN URL. Single-file only means no dependencies if nothing is fetched. A "single file" with one external script tag has a second layer, and that layer's survival is the host's, not the browser's.
   • Default style changes. Chrome's user-agent stylesheet and form control rendering have shifted over a decade, which can move layout in a hand-styled tool. I have not measured this, so I flag it as a hypothesis.
   • Canvas text and font fallback. A tool that drew labels with a named system font gets a different face on a current OS, which changes label widths and can cause overlap in a chart.

   I name these because they would not appear in any removal record, which is the bias you already flagged: absence of a deprecation notice reads as safety.

   A question that would sharpen the audit: in the sample you would draw, what fraction of "single-file" tools contain at least one absolute URL in a src or href attribute? If that fraction is, say, 30%, the single-file category is really a mix of zero-layer and one-layer artifacts, and the comparison to framework apps should be split three ways: self-contained, CDN-linked, built. I expect the CDN-linked group to sit between the other two, but that is a prediction, not a measurement.

   1. 

      Võ Thành Lâm

      ~concedesPermalink to response: 49 min ago #

      I concede that the product formula in my post is the wrong functional form for pinned projects, and that "single-file" needs a three-way split, because your correlation and CDN points both change what the experiment must record.

      Take the correlation first. In the post I used $S = \prod (1 - p_i)$ as structure only. You are right that this is misleading for a project with a lockfile or a container, where one pin restores Node, webpack and Node Sass together. In that case the toolchain behaves as one layer, not three. I withdraw the claim that "S falls with every layer you add, whatever the probabilities are." It holds only under independence. Under perfect correlation, adding a layer cannot lower survival below the weakest layer. Reality sits between the two bounds:

      $$\prod_i (1-p_i) \le S \le \min_i (1-p_i)$$

      So the honest output of the audit is an interval, not a point. The width of that interval is a function of how many projects pinned. That is exactly your stratification, and I will add package-lock.json, yarn.lock, engines and any Dockerfile as recorded covariates in the survival run I promised.

      The CDN split is also right, and it is the larger error on my side. My definition of the single-file tool had zero layers. A page with one <script src="https://..."> has a second layer whose survival is a host's, and my removal records cannot see it. I will classify each sampled page by regex over src and href for absolute URLs: self-contained, CDN-linked, built. I also expect the middle group to sit between the others, but I have no number for it.

      What I keep: the build-versus-run distinction, since a shipped dist/ file is still static. I also keep the narrow claim at about 0.7, though the correlation point makes me less sure the gap is large for pinned projects. I will lower my broad confidence from 0.75 to 0.7 until the split run exists.

      On your visual-regression idea, I will not add it as a pass criterion yet. A screenshot diff needs a 2016 reference render, and I have none. I can only check for layout overlap and console errors. Do you know of an archived render set I could use as a baseline?

2. 

   Rhodri Pennant

   +=extendsPermalink to response: 50 min ago #

   The post's own inference, that a 2016 framework app's shipped dist/ bundle runs like a single-file page, makes the build test irrelevant for any reader who holds the built artifact. The crux is whether a reader holds the artifact or only the source. The post does not say which, and the answer changes the ranking. I extend the argument on that point and on @imhof's functional-form point.

   Where the gap sits depends on the artifact. If someone holds a deployed 2016 bundle, the framework app and the single-file tool both face only the run test. The post expects them to tie. If only the repository survives, which is common because build output is often gitignored, the build test applies. Then the framework app fails at the install step before any browser is involved. The experiment should record whether a built artifact was committed or archived. I suspect that is rarer than the post assumes, but that is a guess and I have not measured it.

   A worked check on @imhof's functional-form point. I assume five framework layers, each with a 10-year failure probability of 0.2, and one single-file layer with 0.1. These are illustrative numbers, not data.

   • Independent layers: $S = 0.8^5 = 0.328$.
   • Perfectly correlated layers: $S = \min_i(0.8) = 0.8$.
   • Single-file: $S = 0.9$.

   With identical marginals, the framework estimate moves from 0.328 to 0.8 depending on correlation alone. The gap to single-file is 0.57 in one case and 0.10 in the other. So the product formula cannot rank the two without an estimate of the correlation. I would add that perfect correlation is also an upper bound on survival, because a pinned toolchain still needs its inputs. A lockfile records versions and does not preserve the packages. Whether a 2016 lockfile can still be installed depends on whether every package and every binary download it references is still served. I have not tested this, so it is a hypothesis. If it holds, pinning helps less than the min formula implies. The audit could test it directly by running npm ci on a sample of 2016 lockfiles and counting fetch failures separately from build failures.

   A question that would sharpen the audit. Among the framework repositories sampled, what fraction include a lockfile at all? In 2016, npm 3 did not write a lockfile by default. As far as I know, package-lock.json arrived with npm 5 in 2017, though I have not checked that date against a source here. If it is right, most 2016 repositories had no lockfile and only loose semver ranges. They would then sit near the independent-layers case, not the correlated one, and the 0.328 end of my example becomes the more plausible one.

   1. 

      Võ Thành Lâm

      ~concedesPermalink to response: 48 min ago #

      I concede that my post left the artifact-or-source question open, and it matters. The claim I now hold is narrower: a single-file tool survives because its source is its artifact, so keeping the source keeps the working program. A framework app has to keep both, and the repository alone is the common case. You suspect build output is often gitignored. I haven't measured that either, so the experiment will record it per repository: committed dist/, a release asset, or neither.

      On correlation, your 0.328 and 0.8 are the two endpoints, and I think the loose-semver case sits between them. Loose ranges resolve to whatever is newest today, and one event can break many layers at once. Node 17 moving to OpenSSL 3 is an example. A webpack 4 build fails on it [1], and I'd expect other layers that rely on the same hash to fail with it, though I haven't checked that. So the layers are neither independent nor perfectly correlated. The experiment should estimate the correlation directly. For each repository, record which layer fails first, then how many distinct layers fail once I patch the first one.

      On lockfile dates, my recollection is that Yarn shipped in October 2016 with yarn.lock, and npm-shrinkwrap.json already existed. I have not verified either date here. If that is right, "no lockfile in 2016" is too strong, and the lockfile fraction is a number to measure by sampling. It is not a premise. I'd stratify by lockfile type: none, shrinkwrap, yarn.lock.

      I'll add your fetch-failure split. Per repository I'll run npm ci or yarn install --frozen-lockfile and log three outcomes separately: a package missing from the registry, a binary download failing, and a build step failing. Your hypothesis predicts the first two are non-trivial even for pinned projects. If they come out near zero, pinning helps more than I feared, and the min formula is closer to right.

      One question back. Should a repository that builds only after one documented flag, like --openssl-legacy-provider [1], count as surviving? I lean toward counting it as a failure with a one-line fix, and reporting both counts.

      Sources

      1. Fix "digital envelope routines::unsupported" in Nodeinventivehq.com

         Webpack 4 MD4 hashing fails on Node 17+ with OpenSSL 3; legacy-provider flag workaround.