Skip to content

Skip unused distributions in entropy-weighting bisection - #14

Open
kevinmoran1988 wants to merge 1 commit into
TravisWheelerLab:mainfrom
kevinmoran1988:stable
Open

Skip unused distributions in entropy-weighting bisection#14
kevinmoran1988 wants to merge 1 commit into
TravisWheelerLab:mainfrom
kevinmoran1988:stable

Conversation

@kevinmoran1988

Copy link
Copy Markdown

eweight_target_f() and eweight_target_exp_f() call p7_ParameterEstimation(), which reparameterizes match/insert/delete transitions and both emission distributions.

Only hmm->mat is read, by p7_MeanMatchRelativeEntropy(); the other four are discarded on every root-finder iteration (~10-15 per model).

Add p7_ParameterEstimation_MatchOnly() and call it from both callbacks. Cuts esl_mixdchlet_MPParameters() calls in the bisection from ~5M to ~M per iteration.

bathbuild over 5232 dipteraODB12 alignments, serial: 859.6s -> 778.3s (1.10x). Output byte-identical on that corpus
and across the build/weighting/prior option matrix.

eweight_target_f() and eweight_target_exp_f() call
p7_ParameterEstimation(), which reparameterizes match/insert/delete
transitions and both emission distributions. Only hmm->mat is read, by
p7_MeanMatchRelativeEntropy(); the other four are discarded on every
root-finder iteration (~10-15 per model).
Add p7_ParameterEstimation_MatchOnly() and call it from both callbacks.
Cuts esl_mixdchlet_MPParameters() calls in the bisection from ~5M to ~M
per iteration.
bathbuild over 5232 dipteraODB12 alignments, serial: 859.6s -> 778.3s
(1.10x). Output byte-identical on that corpus
and across the build/weighting/prior option matrix.
@traviswheeler

Copy link
Copy Markdown
Member

Thanks for the PRs. I've reviewed both, and the changes seem reasonable.

One request before merging: this PR's branch (stable) holds two commits, 7593192 (the entropy-weighting change described here) and 9742af0, a copy of the bathalign/--nofs commit that is also PR #15. As filed, merging this PR would pull in both changes. Could you rebase stable down to just 7593192 so the two PRs are independent?

Additionally, a comment: this commit contains essentially no context (who you are, why you're making the change, whether an LLM was involved), and comes from an account that isn't connected in an obvious way to a human and that has very little activity. It's nice that you're making useful changes, but I think recipients of those changes will be a lot happier if they know who you are and what role you played in creating the code.

@kevinmoran1988

kevinmoran1988 commented Sep 5, 2026

Copy link
Copy Markdown
Author

Apologies, I should have led with this rather than dropping patches from a cold account.

I'm Kevin Moran. I'm developing a phylogenomics pipeline that recovers orthologous gene sets for downstream phylogenetics from either high-throughput sequencing reads or genome assemblies. The account is quiet because most of my work lives in that one repo. It's private because it's unpublished work. I can provide access if desired.

Why I'm here. The pipeline depends on both BATH and HMMER. BATH does translated search and replaced a per-frame hmmbuild + hmmsearch pipeline. HMMER is in the tree solely for hmmalign, which does all of the pipeline's profile-to-sequence alignment. BATH HMMs aren't readable by hmmalign, so every gene is built twice, once for search and once for alignment. Since BATH already ships the whole tracealign engine, that second install exists only to reach a driver BATH dropped.

What I can and can't cache. References are sampled per taxon, so search HMMs can't be cached across taxa to begin with, and the alignment HMMs are a second, separate rebuild on top of that. Folding alignment into BATH would let me build once per gene and use the same model for both across a first-pass run. Two places would still have to build on the spot:

  • Consolidation. When the processed taxa are merged for phylogenetic matrix creation, the alignment HMM is new by construction.
  • Iterative runs. Iteration 2+ folds the previous run's HQ output back into the reference set. For high-copy multi-gene families I split the references into child HMMs to boost sensitivity, then subsample those to build a primary HMM for alignment, so hits from the child-HMMs land on a single coordinate system. The reference set has changed, so nothing from iteration 1 is reusable. The child-HMMs are incomplete samples, so they can't serve as the alignment model either.

Models built at those two points are alignment-only and never consulted for frameshift statistics. Thus why I added the --nofs flag.

Where the time goes. The build/align split inverts depending on the dataset shape.

dataset genes hmmbuild hmmalign total in HMMER
Zygentoma ORGRv3 8 0.65 s (7.7%) 7.80 s (92.3%) 8.5 s
Psychodidae dipteraODB12 2,061 352.31 s (80.5%) 85.41 s (19.5%) 437.7 s

Disclosure on how these were written

  • 7593192 in BATH (skip unused distributions in the entropy-weighting bisection) and 3dfa20a in Easel (cache alpha-invariant lgamma terms in the mixture Dirichlet priors): I wrote the code. Both modify existing behavior, in someone else's repo, so I'd prefer to be hands on . An LLM (Claude Code Opus 5.0) set up the profiling harness and the validation harness; I spot-checked outputs myself and then ran the resulting binary in my own pipeline before proposing anything.
  • 3a7deac in BATH (bathalign + --nofs): this one adds a driver over the existing tracealign engine and touches no existing code path, so I judged it low-risk and had an LLM (Claude Code Opus 5.0) do the port and write the validation harness. I reviewed the result and verified the claims (amino output byte-identical to HMMER 3.4 hmmalign across --mapali, --outformat, --trim, Stockholm and afa; --nofs models byte-identical apart from the omitted FS3/FS5 STATS and FRAMESHIFT PROB lines).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants