Analyze literal relevance across visible content The HTML TF-IDF Analyzer examines the visible text stored for a page and explains how LuceneUtil’s lexical vocabulary contributes to its HTML TF-IDF factor. Enter a search phrase and inspect the complete content heatmap. Yellow identifies normalized query-word occurrences; weighted orange identifies catalog Similar Words, including bounded singular and plural variations. Gray dotted stop words remain visible because readers need them, but the scoring calculation excludes them from occurrences and field length. This tool does not infer arbitrary paraphrases, entities, or topical meaning. It explicitly loads Similar Words and scales each one by its stored relationship to the search phrase; a term must still occur in the content to contribute. An unhighlighted passage may therefore remain highly useful and semantically relevant. TF-IDF asks how often the configured vocabulary appears, how distinctive those words are in the preview collection, and how concentrated the evidence is within the scored content. Keeping that boundary clear prevents lexical optimization from replacing genuine subject coverage.
Trace every occurrence into term frequency Each meaningful query word receives its own evidence row. The occurrence count comes from the entire visible-content field, not only the first highlighted passage. Term frequency uses a square-root transformation, so moving from zero occurrences to one matters more than moving from nine to ten. Repetition therefore grows the score slowly and should never be treated as a linear keyword target. Hover highlights to connect the visible words with the term row that explains their contribution. Review where occurrences come from. Navigation, repeated cards, form labels, related-content modules, and footer text can all appear in stored visible content. Ten matches in template chrome do not necessarily mean the article answers the query. Scroll through the field and distinguish central explanations from incidental interface text. When useful occurrences are absent from the main content, add a real answer or clarification rather than inserting isolated words into unrelated sections.
Use rarity as collection-specific evidence Inverse document frequency gives more value to a query word that appears on fewer pages in the comparison collection. The term cards show how many displayed documents contain the word and the resulting IDF value. In a showdown, both target and comparison content provide the two-page preview corpus. In target-only mode, the analyzer uses one page so an unauthenticated visitor can still test a draft without inventing a hidden outperformer. A complete Ranking Sensor contains more documents, which means its production IDF values can differ. Use the visualizer to understand the formula, compare visible evidence, and predict direction. Then recalculate the sensor to obtain the full-corpus score. Rarity is not universal keyword importance. A technical term may distinguish pages in one result set and occur everywhere in another. Avoid exporting a preferred occurrence count or IDF threshold across unrelated queries.
Follow the complete HTML TF-IDF calculation The first sub-score is search-concept coverage. It reports how many distinct configured, non-stop-word concepts occur in visible content, which helps locate a genuine vocabulary or information gap. LuceneUtil deliberately fixes clause coordination at 1.0, so the coverage count does not multiply the entire HTML score down. Missing vocabulary loses its own term contribution without receiving a duplicate global penalty. The second row totals weighted term evidence. For each search word or Similar Word, the formula uses the square root of occurrences, inverse document frequency squared, and the term’s relationship weight. Document frequency records how many pages contain the word; IDF turns that corpus frequency into a rarity value. A Similar Word with a 0.7 relationship can contribute 70% of the otherwise equivalent original-word evidence. The cards show every input and the resulting additive contribution, so a user can distinguish repetition, rarity, and relationship instead of seeing only a highlight. The third row is query normalization. The query-vector magnitude is visible, but Market Brew fixes the applied query-normalization multiplier at 1.0. The fourth row is visible-content length normalization. HTML uses 1 divided by the square root of the greater of its scored-word count or 1,000. This 1,000-word floor prevents a tiny body from receiving an outsized concentration reward. Beyond that floor, additional scored words gradually change the norm. Stop words remain excluded from the scored count. The fifth row multiplies the weighted term total by coordination, query normalization, and content-length normalization to produce raw TF-IDF additive points. Those points are the honest result of a standalone edit preview. A task-linked view adds the sixth row, the calibrated HTML field boost used by LuceneUtil, and the seventh row, final task-score normalization. Final normalization divides the page’s raw Lucene HTML contribution by the strongest raw HTML contribution in that task’s modeled result set and multiplies by 100. Thus 100/100 identifies the corpus leader rather than declaring that the page has ideal content. None of these stages establishes a universal article length. A comprehensive guide can need thousands of words, while a focused service page may answer its purpose concisely. If large unhelpful sections dilute a page, removing them can change length normalization and improve focus. If essential information is absent, adding a useful explanation may strengthen term evidence and reader value even when total length increases. The calculation shows the tradeoff instead of reducing it to keyword density.
Distinguish TF-IDF from BM25 TF-IDF and BM25 both examine literal words and collection rarity, but their frequency and length formulas are different. This analyzer uses Classic Similarity-style square-root term frequency, inverse document frequency, a fixed coordination value, a fixed query-normalization value, and field-length normalization with an HTML floor. BM25 uses its own term-saturation curve and a tunable document-length formula. The BM25 Content Analyzer is therefore not a duplicate, and the same edit can move the two previews by different amounts. Semantic similarity is different again. Embedding tools can recognize related meaning even when neither the search phrase nor any configured Similar Word appears. Use HTML TF-IDF when a task names that ranking factor, BM25 when its factor identifies literal saturation or length issues, and semantic tools when the page may express the idea through alternate language. Comparing the three views can reveal whether the problem is missing configured terminology, diluted lexical focus, or broader intent alignment.
Compare useful passages, not keyword totals Authenticated showdown mode places the target content beside an indexed Ranking Sensor URL. Start with the term rows, then locate the highlighted words in both complete previews. The comparison may use an important term inside a definition, example, requirement, table label, or decision step that the target lacks. That context is more actionable than copying its raw frequency. Identify the missing information that naturally produced the lexical difference. Do not assume every outperformer section belongs on the target. Pages can serve different intents, audiences, regions, or stages of a journey. The target may have a stronger original explanation using vocabulary outside the Ranking Sensor's configured Similar Words. Preserve useful synonyms and brand language. Choose one gap that is both statistically visible and editorially appropriate, then write an original passage supported by accurate facts.
Edit the content heatmap and measure immediately Change the target’s visible content directly in the heatmap and click Measure This Change. The tool recalculates locally without publishing, recrawling, or refreshing the catalog. The edited score summary reports before-and-after raw points, while the term rows reveal changes to counts, rarity context, relationship weights, the coverage diagnostic, and length normalization. Restore Indexed Content returns the editor to the stored page text so experiments do not become confused with production content. Test one hypothesis at a time. Remove a block of unrelated boilerplate and observe the field-length effect. Add a concise missing answer and confirm that the intended term moves from zero to one occurrence. Replace an exact word with a weighted Similar Word and verify that its displayed relationship reduces its influence; use an unconfigured paraphrase to confirm that lexical credit disappears even when the prose remains semantically sound. These controlled experiments teach the team what the factor measures and what it deliberately ignores.
Improve information instead of density A low score should lead to a content question. Does the page omit a definition, prerequisite, feature, comparison, example, limitation, location, price condition, outcome, or next step that a searcher needs? Add the information that resolves uncertainty. Natural query vocabulary often appears as a consequence of answering well. If the only available edit is an isolated word with no supporting meaning, leave the score alone and investigate another factor. Also inspect high scores critically. Repeated template text can create lexical evidence without useful depth. Keyword-stuffed copy may score while damaging readability and trust. Keep headings descriptive, paragraphs coherent, and claims supportable. Use the highlights as an audit overlay, then turn them off mentally and read the page as a visitor. The final text should remain useful without any awareness of the scoring system.
Close the loop with Ranking Sensor recalculation Document the calibrated query, target URL, comparison URL, original preview, approved draft, and reason for the edit. Publish through the normal review process and wait for the catalog to capture the new content. Recalculate the Ranking Sensor so the factor can use its full collection statistics and current modeled weights. Return to the task to confirm that the production HTML TF-IDF gap changed in the expected direction. Review adjacent signals before declaring success. Query Coverage can confirm distinct-word presence, Query Proximity can evaluate word relationships, BM25 can expose saturation and length behavior, Content Depth can assess useful breadth, and semantic visualizers can inspect meaning. A durable optimization satisfies the page’s purpose across these perspectives without forcing every score to its maximum. The tool succeeds when it turns a mysterious TF-IDF task into visible evidence, a small responsible edit, and a verifiable result.
