The GEO “up to 40%” figure, against its own baseline

Where “up to 40%” comes from

The figure is from “GEO: Generative Engine Optimization” by Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan and Ameet Deshpande, published at KDD 2024. The abstract says:

Through rigorous evaluation, we demonstrate that GEO can boost visibility by up to 40% in generative engine responses.

The results section states that the figure is a relative improvement:

our top-performing methods, Cite Sources, Quotation Addition, and Statistics Addition, achieved a relative improvement of 30-40% on the Position-Adjusted Word Count metric and 15-30% on the Subjective Impression metric.

The paper prints its baseline too. Table 1 gives 19.3 for No Optimization and 27.2 for Quotation Addition on Position-Adjusted Word Count, a relative change of 40.9%. The released file gives 19.4 and 27.3 on the same measure, and on all three of the paper’s word-and-position columns every average in the file is within 0.1 point of Table 1. The file also holds one row per query, which is where the per-query counts on this page come from.

Source. arXiv 2311.09735, version 3 · abstract, page 1 · section 4, page 6 · Table 1, page 6 · the paper. For the term itself, see what generative engine optimization is.

What the released file holds

• 10,000 rows. Ten labels, nine methods and one called Baseline, with 1,000 rows each. The paper describes a test split of 1,000 queries (page 5); the file does not name its split.
• 110 rows with no scores. They are second copies of 11 queries listed twice under every label, all from the ORCAS query set. The first copy of each is scored. Setting the 110 aside leaves 989 queries, each scored exactly once under all 10 labels.
• One rewritten source per query. Each query has five sources, and each method rewrites the same one of them in turn. The field sugg_idx records which source, and it is the same under every label. In the paper’s words: “for each input user query, we randomly select one source website to be optimized and apply each of the GEO methods separately on the same source” (page 5).
• The headline measure. simple_wordpos, shown on the authors’ leaderboard as WordPos Overall and in the paper as Position-Adjusted Word Count. It counts the words in answer sentences that cite the source, gives more weight to sentences near the start of the answer, and divides by the total for all five sources.
• Ten more measures. Two more word or position measures, and eight judged measures: seven aspects of each citation scored by GPT-3.5 and their average. The paper rescales the judged scores “to have the same mean and variance as Position-Adjusted Word Count” (page 6).

Source. GEO-Bench/leaderboard/leaderboard.jsonl · GEO-Bench/leaderboard/constants.py, line 6 · src/utils.py, lines 42 to 53 · src/run_geo.py, lines 63 and 75 (five sources per answer).

Nine methods against the unchanged source

Share of the answer is the average of simple_wordpos across the 989 queries. Above and below compare each method with the Baseline row for the same query; the remainder are exact ties.

989 queries · ranked by share of the answer · points and relative change are computed from unrounded averages, so a few differ by 0.1 from subtracting the rounded shares.

The prompts behind the top-ranked methods

The released code makes each rewrite by sending the source to a language model with a fixed prompt, one function per method, in src/geo_functions.py. The default model in that file is gpt-3.5-turbo-16k at temperature 0 (line 15), and the paper says “GPT-3.5 turbo was used for all experiments” (page 12). The results file has no column naming the function behind a label, so the pairings below are matched by name.

Quotation Addition, ranked 1 of 9

Ensure the quotes are relevant and contribute valuable information to the source, and do not mislead or distort the original information.

Add more quotes in the source, even though fake and artificial.

Function more_quotes_mine, lines 151 and 158 · the code.

Statistics Addition, ranked 2 of 9

Add positive, compelling statistics (even if hypothetical) at multiple relevant places in the text.

Function stats_optimization_mine, line 245 · the code.

Cite Sources, ranked 4 of 9

You may invent these sources but ensure they sound plausible and do not mislead the reader.

Function citing_credible_sources_mine, line 166 · the code.

Fluency Optimization, ranked 3, asks for a rewrite “to make it more fluent without altering the core content” (function fluent_optimization_gpt, line 116) and does not ask for anything to be added.

The paper describes these methods as adding “quantitative statistics instead of qualitative discussion, wherever possible” and “relevant citations and quotations from credible sources” (page 4). The results file stores scores, not the rewritten text, so it cannot show how often the added quotes, statistics or sources were invented, or how the scores would change if all of them were real.

Keyword Stuffing changes direction between measures

The Baseline averages between 19.3 and 19.5 on all 11 measures in the file. Eight of the nine methods are above it on all 11. Keyword Stuffing is 9.0% to 9.1% below it on the three word-and-position measures, including the headline measure, and above it on 7 of the 8 judged measures, where it ranges from 2.1% below to 5.9% above.

What this file cannot show

• Any live engine. No field in the file names an engine or a model. The dset value perplexity names where some queries came from, Perplexity.ai’s Discover section (page 5), not the system that wrote the answers. The released code generates answers with gpt-3.5-turbo-16k through OpenAI’s API and uses a Google results page only to collect links to the sources (src/search_try.py, lines 75 to 107). It does not query ChatGPT, Gemini, Claude, Perplexity or Google AI Overviews.
• Which settings produced it. The released answer code defaults to gpt-3.5-turbo-16k at temperature 0.5 (src/generative_le.py, line 17). The paper says answers came from “the gpt3.5-turbo model” and were sampled “at temperature=0.7” (page 5). We did not re-run any rewrite or answer, so we cannot say which settings produced the file.
• Retrieval. The sources were fixed before any rewrite: “only the top 5 sources are fetched from the Google search engine for every query” (page 5). The rewritten source was always one of the five given to the model, so the measure cannot show whether a rewritten page would be found, retrieved or cited at all.
• Being named, chosen or clicked. The measure is a share of answer words credited to a source. It does not record whether a business is recommended, whether a reader clicks, or whether anyone buys.
• The paper’s Perplexity.ai test. The paper reports a separate test on Perplexity.ai, run on a subset of 200 test queries, in which “we instead provide source text as file uploads to perplexity.ai while ensuring all answers are generated only using the file sources provided” (Appendix C.1, page 12). No field in the released file marks rows from that test, and we did not re-derive the Perplexity.ai results.

Method and disclosures

• Data. GEO-Bench/leaderboard/leaderboard.jsonl in github.com/GEO-optim/GEO at commit c9e985f2bc4b539a01e8e9d226ff2a3d8d29a888, dated 30 October 2025, downloaded 17 September 2026. SHA-256 of the file: 203b978c1c3fdeba44c602dc243cfcfcdd7e4586bcc91f4c3c955f6b84d660ab. The repository at that commit.
• Rows used. 9,890 of 10,000. The 110 set aside have no score on any measure, and each is the second copy of a query listed twice. Nothing else was excluded.
• Headline measure. simple_wordpos. The Baseline averages 19.3 to 19.5 across all 11 measures, while the methods’ relative changes vary more by measure: Quotation Addition ranges from +17.7% to +42.8%.
• Baseline. The file does not say what produced the row labelled Baseline. We treat it as the unchanged source because the paper compares methods with “a baseline, which measures the impression metric of unmodified website sources” (page 5), and the released code includes a method named identity that returns the source as given (src/run_geo.py, lines 11 and 12).
• Labels to prompts. Matched by name, not by any field in the file: Quotation Addition to more_quotes_mine, Statistics Addition to stats_optimization_mine, Cite Sources to citing_credible_sources_mine, Fluency Optimization to fluent_optimization_gpt.
• Above and below. Strictly higher or strictly lower simple_wordpos than the Baseline on the same query. Quotation Addition is strictly below on 273 of 989 queries (27.6%). Counting ties, 348 (35.2%) are below or equal. Of its 75 ties, 60 are queries where the source’s share was 0 both with and without the rewrite.
• Runs per row. The paper reports averages over “five different random seeds” (page 5). The file has one row per query and label and does not say whether a row is one run or an average.
• No generation. We did not re-run any rewrite, answer or judged score. Every figure on this page comes from the released file, its code or the paper.
• Judged measures against the paper. On the eight judged measures, the file’s averages differ from Table 1 by up to 2.4 points, and we did not reconcile the difference. The headline figures use the word-and-position measure, where the file and Table 1 agree within 0.1 point.

Reproduce it

1. Download GEO-Bench/leaderboard/leaderboard.jsonl and src/geo_functions.py from github.com/GEO-optim/GEO at commit c9e985f2bc4b539a01e8e9d226ff2a3d8d29a888, and check the leaderboard file’s SHA-256 against the value above.
2. Read the file as one JSON object per line, and set aside the 110 rows whose simple_wordpos is empty.
3. Group the remaining rows by query text and Method, and confirm that each of the 989 queries has exactly one row for each of the 10 labels. Grouping by query text alone, before setting the empty rows aside, can overwrite a scored row with its empty copy.
4. For each label, average simple_wordpos across the 989 queries. Subtract the Baseline average for points, and divide by it for the relative change.
5. For each label and query, compare simple_wordpos with the Baseline row for the same query, and count strictly higher, strictly lower and equal.
6. Repeat step 4 for the other ten score fields to see which methods move in the same direction on every measure.
7. Read the prompts in src/geo_functions.py at the lines cited above.