Ismail’s Threshold Theory to Master Perplexity AI
Abstract
Undoubtedly, there is a potential impact of Large Language Models (LLMs) like ChatGPT on revolutionizing information retrieval and knowledge discovery, particularly in the context of the vast amount of electronic material available. On another strong note, the current work offers a first time ever mathematical approach to accurately fine- tune perplexity AI, which sets a giant step ahead towards accuracy of the next generations AI. Having started this world- leading discovery, the upper and lower bounds of perplexity AI, namely UBPerplexity AI LBPerplexity AI, respectively, are obtained for the first time.
Keywords:PerplexityAIChatGPTEntropyNatural language processingIncreasing decreasing testThreshold
1. Introduction
This hot burning topic needs more exploration, especially, when we investigate the evaluation of two versions of ChatGPT, which would reveal that they still lack the ability to handle the intricacies of complex topics due to limited access to the entire scientific literature. However, they will soon be valuable for assisting scientists in surveys, literature reviews, and teaching [1].
Entropy [2] as a powerful tool of fine-tuning uncertainty can efficiently impact ChatGPT-4’s advancement including other futuristic ChatGPT forms. For example, to improve the accuracy of the generated thoughts, there are some proposed filtering them based on uncertainty, such as answer entropy, which measures the uncertainty of the answers, and filters out thoughts with higher uncertainty, so filtering by answer entropy leads to slightly higher accuracy compared to filtering by the number of consistent paths used in previous work.
Perplexity [3] is a statistical metric used in Natural Language Processing (NLP) to measure how well a Language Model (LM) predicts a sample. A lower perplexity score indicates that the LM is more accurate and confident in its predictions.
In NLP, encoding refers to representing text as numerical vectors, which can be used as input for machine learning models. Embeddings capture the relationships between words or phrases in a text corpus. Entropy can steer the advancement of both existing and futuristic ChatGPT to higher levels of accuracy. Based on my world-leading mathematical discovery, Perplexity AI, can be fine-tuned through obtaining entropic thresholds corresponding to perplexity expression.
Notably, Shannonian entropic formula, will be the very basic to target, then taking a higher-level step ahead, my innovative entropy formula, namely, Ismail’s second entropy [4], will provide a novel link between Entropy Theory, Statistical Mechanics, and AI. My lecture will travel through a visionary spotlight to employing another three entropies in the literature, under my name, Ismail’s entropy 1,3, and 4 [5–7].
My innovative mathematical approach has the great potential to steer AI into a higher level of accuracy, which will be highly influential to be applied to all fields of human knowledge. In 2020, the development of LLMs like GPT-3 marked progress towards Artificial General Intelligence (AGI), capable of performing various intellectual tasks [8].
Open AI introduced ChatGPT in 2022, an AI chatbot that generates coherent sentences by analyzing patterns in text data, impacting industries like publishing, education, and science. Despite its utility as a tool, concerns arise regarding ChatGPT's potential for errors in providing accurate information [8], leading to discussions on its role in scientific authorship and the need for appropriate disclosure in research publications.
A new type of hack has been discovered [9] that exploits LLMs by using adversarial suffixes to manipulate them into generating dangerous responses. These hacks can trick LLMs like GPT-2 into providing detailed instructions for creating explosives, planning criminal activities, or generating offensive content. By assessing the perplexity [9] of queries with adversarial suffixes, researchers found that these queries have very high perplexity values, leading to the development of a detection method using Light-GBM trained on perplexity and token length to identify and prevent such attacks effectively.
A new type of hack has been discovered that exploits LLMs by using adversarial suffixes to manipulate them into generating dangerous responses. These hacks can trick LLMs like GPT-2 into providing detailed instructions for creating explosives, planning criminal activities, or generating offensive content.
By assessing the perplexity of queries with adversarial suffixes, researchers [10] found that these queries have very high perplexity values, leading to the development of a detection method using Light-GBM trained on perplexity and token length to identify and prevent such attacks effectively.
In the research, two datasets of adversarial prompts were used, with the first dataset containing 1407 machine-generated prompts created following a specific methodology outlined in a previous study by [10].
The prompts were generated using the Vicuna-7b-1.5 model and the "individual" GCG method, focusing on harmful behaviors. Each prompt was initialized with a default starter suffix of 20 exclamation marks, and the generation process required significant computational resources, as showcased by figure [10].

It highlights that the generated adversarial attacks have high perplexity values, with different clusters identified based on perplexity levels. The green cluster represents attacks with a low perplexity below 18, the red cluster includes prompts with a higher perplexity above 400, and the yellow cluster contains prompts with perplexity values in between. The authors of [11] have thoroughly using perplexity, a measure of how well a LM predicts the next word in a sequence, to debunk misinformation in an unsupervised manner, particularly during fast-evolving events like the COVID-19 outbreak. By leveraging the higher perplexity of misinformation compared to truthful statements [11], the proposed method involves extracting evidence, priming a LM, and evaluating claims based on perplexity scores to effectively debunk false information. The approach is empirically validated using COVID-19-related test sets and aims to encourage further research in combating misinformation across various topics.
The proposed approach involves using a LM to debunk false claims by providing it with evidence related to the claims and then assessing the perplexities during the debunking process. This method aims to leverage the LMs ability to analyze and evaluate the validity of various statements by measuring the perplexity values, which indicate the model's level of uncertainty or confusion regarding the information provided. By utilizing this approach, researchers can potentially identify and refute misinformation or false assertions more effectively. This is illustrated by Fig. 2 [11].

Following [12], there is a great emphasis on the importance of ensuring that a LM has accurate and relevant evidence before evaluating claims, especially for emerging events. This highlights [10] two main approaches for obtaining evidence in fact-checking tasks: using structured knowledge bases like Wikipedia and knowledge graphs or extracting evidence directly from unstructured online data. The authors of [10] aimed to combine these approaches by extracting evidence from unstructured data while filtering out noise to enhance the quality of evidence used in their work.
In the context of the debunking system [11], the threshold, denoted as th, can be adjusted to control the system's strictness. Lowering the threshold reduces False Negative (FN) errors, which are considered more critical, but this adjustment may lead to an increase in False Positive (FP) errors. Finding a balance between FN and FP frequencies [11] by selecting an appropriate threshold is crucial for achieving a more balanced and effective debunking system in real-world applications, which can be illustrated by Fig. 3 [11].

The current work reads: 1) Introduction, 2) Crosswords between perplexity ai, and shannonian entropy, 3) Results and experimental validation, and 4) Conclusion, open problems, and next research phase.
The current paper contributes to
I. The provision of a first time ever mathematical approach to accurately fine tune perplexity AI.
II. Obtaining both upper and lower bounds of perplexity AI, namely UBPerplexity AI, LBPerplexity AI, respectively, are obtained for the first-time ever III. The provision of some emerging open problems.
2. Crossroads between Perplexity AI, and Shannonian Entropy
When assessing LMs, one popular statistic to utilize is ambiguity [13]. Consider the built-in metric of perplexity in scikit-learn's implementation of the topic-modelling algorithm Latent Dirichlet Allocation. In numerous Natural Language challenges, a common configuration [13] is to have a language L and aim to coSnstruct a model M for it. A particular genre or corpus, such as English Wikipedia, Nigerian Twitter, or Shakespeare, or (at least conceptually) a general term, such as French, could constitute the language.
Specifically [13], by a language L, we mean a process for generating text. Given a history h consisting of a series of previous words in a sentence, the language L is the probability that the next word is w, as prescribed by Eq. (1).
h a series of words(w1, w2, … , wn−1),
L(w|h) = Prob(next word is w|previous words are h), (1)
A LM is a collection of probaility distributions.
If L is English, for instance, I'm ready to bet that:
I. L(dog | The swift brown fox leaps over the slow brown) ≈ 1.
II. L(ipsum | Lorem) ≈ 1.
III. L(wings | Buffalo buffalo buffalo Buffalo buffalo) ≈ 0.
In response to Eq. (2).
L(s) = L(w1|w0) × L(w2|w0w1) … × (wn|w0w1 … wn−1) = L(wk|w0w1 … wk−1). (2)
The language L gives the probability of a sentence s.
Nonetheless, it is typical practice to omit the product's initial phrase as well, or even to utilize an even longer beginning context [13].Obtaining an impeccable imitation of L in, instance, spoken American English is surprisingly simple. Simply signal any person who is fluent in English as they pass by. Typically, our goal is to teach the model to a computer, which is why machine learning was created. Thus, M will represent any LM that we have been able to construct computationally. This configuration, comprising of a language L and a model M, is highly versatile and is utilised in numerous Natural Language jobs such as speech-to-text, autocorrection, autocomplete, machine translation, and so forth.
Given the words someone has entered so far, autocomplete is the most obvious example; pick the completion with the highest probability and try to estimate what they could type next.
Eventually [13], if given a model M, the perplexity on a sentence s, reads by Eq. (3).
Perplexity(M) = (∏ ( )) . (3) M(wk|w0w1 … wk−1)
2.1 Perplexity of a Language Model M
This is the opposite of the geometric- mean of the terms in the product's denominator, as the second line makes clear. This can be understood as a per-word metric since the likelihood of each word is only computed once, contingent upon history. This indicates that sentence length has no bearing on the level of bewilderment, other things being equal.
We want our odds to be high, which means less confusion. The perplexity would be 1 and the model would accurately predict the text if all the probability were 1. Conversely, there will be more confusion with worse LMs.
The language itself sets a lower bound on bewilderment. However, this highlights a typical aspect of NLP metrics: a simple metric, such as perplexity, may not always be the best indicator of a model's actual performance. For development (validation), confusion is beneficial, but not always for evaluation. Human evaluation is still considered the gold standard in evaluation.
2.2 Entropy
While entropy is a difficult notion in physics, information theory makes sense of it very easily. Assume you have a procedure (e.g., a word-generating language L). There is a chance p that the event or the thing that happened was going to happen at every stage of the process. The unexpected number is −log(p), where you can take the logarithm in any base by altering the units. Events with little likelihood are highly surprising. There are zero surprises in events that were expected to occur (p = 1). Unfeasible events (p = 0) have infinite surprise.
The expected value of the surprise over all potential events, indexed by i, is known as the entropy, namely, H(p), as in Eq. (4):
H(p) = − ∑ pi lnpi.
2.3 Relationship between Perplexity and Entropy
Now all that remains to do is show the relationship between the two. Assuming we took the logarithm in base e, following Eq. (5):
H(L,M)
Perplexity (M) = e . (5)
If we took the logarithm in base 2, use 2 for the base, etc. In summary, therefore:
I. We construct a LM M for the real language, L, that is producing the data.
II. We calculate M′s (with respect to L) perplexity or, equivalently, cross-entropy.
III. The actual language L′s perplexity limits M′s perplexity below (see, cross-entropy).
How random our model is is measured by the perplexity. If each word has a perplexity of 3, it indicates that the model's average chance of correctly predicting the next word in the text was one in three. It is occasionally referred to as the average branching factor for this reason.
2.4 Key Features and Advantages
Accuracy: perplexity AI delivers high accuracy rates and can be fine-tuned [14].
Efficiency: the platform offers several NLP features, making it an efficient and reliable way to find information quickly [15].
User-friendly interface: perplexity AI is easy to use and navigate, with a clean and well-designed interface [16], [17].
Credibility: the platform provides citations for the information it returns, allowing users to verify the reliability of the information [18].
Real-time information: Perplexity AI can provide real-time information, outperforming other AI chatbots like ChatGPT [19–21].
2.5 Limitations
Perplexity AI’s limitations include its sensitivity to the specific test set used, which may not always provide a consistent evaluation of its performance [15]. Additionally, while it can sometimes generate incorrect answers, it makes it easy for users to check the sources themselves [16].
The threshold approach is pivotally based on the derivative test, where increasability phase exists for regions of positive derivatives, while decreasability exists in negative derivative zones, as shown by figure [19]. Notably, this is the cornerstone of this current work, namely the Increasing/Decreasing Test (IDT).

3. Results and Experimental Validation
Theorem 1. The shannonian perplexity AI, namely, as in Eq. (6):
H(p(n) N
ωb(p(n)) = b , H(p(n)) = − ∑n=1 p(n)lnp(n), b ≥ 2. (6)
I. Increases forever in a fine tuner, say λ if and only if ∃ λ > 0 satisfying Eq. (7):
−(λ+1) (λ+1)
p(n) = e , ωb(p(n)) = b ,
λ = −lnp(n) − 1, p(n) ∈ [0,1], H(0) = 0.
II. Decreases forever in ρ if and only if ∃ ρ > 0 satisfying Eq. (8):
(ρ−1) (1−ρ)
p(n) = e , ωb(p(n)) = b , ρ = lnp(n) + 1,
p(n) ∈ [0,1], H(0) = 0.
Proof: let ω(x), defines Shannonian perplexity AI, following Eq. (9):
H(p(n))
ωb(p(n)) = b ,
H(p(n)) defines the Shannoian entropy of ( Eq.(4)). (9)
This implies Eq. (10).
H(p(n)) = − ∑n=1 p(n)lnp(n). (10)
Hence, by Eq. (11):
dωb H(p(n)) N
= b (− ∑n=1(1 + lnp(n))).
dp(n) (11)
Communicating (11), and IDT, one gets Eq. (12):
dωb H(p(n)) N
> 0 ⟺ b (− ∑n=1(1 + lnp(n)))lnb > 0.
dp(n) (12)
H(p(n))
Since b > 0, we have Eq. (13).
dωb
> 0 ⟺ 1 + lnp(n) < 0. (13)
dp(n)
1 + lnp(n) < 0 translates to the existence of a positive real number, say λ, satisfying Eq. (14).
1 + lnp(n) = −λ . (14)
Or, this provides Eq. (15),
−(λ+1)
p(n) = e . (15)
The normalization property implies Eq. (16):
∑n=1 p(n) = 1. (16)
Eq. (7) and Eq. (16) we can obtain Eq. (17):
−(λ+1) (λ+1)
Ne = 1 → N = e . (17)
Eq. (17) translates to Eq. (18):
λ = ln N − 1. (18)
Wrapping up Eqs. (9), (15),(17), and (18) yields Eq. (19):
N −(λ+1) −(λ+1)
H(p(n)) −(∑n=1 e (−(λ+1))) (λ+1)Ne (λ+1)
ωb(λ) = b = b = b = b . (19)
Moreover,
dωb
H(p(n))
< 0 ⟺ b (− ∑(1 + lnp(n))) > 0. (20) dp(n)
Eq. (20) holds only if ∃ a positive real number ρ, such that, Eq. (21) holds:
1 + lnp(n) = ρ. (21)
Engaging the same logic as in (I), the proof of (II) follows.
Experimental setup
The increasability zones, let b = 2.

Observing Fig. 5, it can be seen perplexity AI increases by the progressive increase of the fine tuner, λ. Equationally, we can see that Eq. (22), will follow:
(λ+1)
lim ωb(λ) = lim 2 = ∞.
λ→∞ λ→∞ (22)
On another separate note, reading the represented data of Table 1, b = 2, the only permitted case, is for ρ = 1, otherwise, this violates the axiomatic probability logic.
Table 1. Data showcasing the impossibility of existing zones of decreasability.
This table is shown in the PDF version of the article.
Theorem 2. The upper and lower bounds of Shannonian perplexity AI, UBPerplexity AI , LBPerplexity AI,
respectively, are given by the following inequality Eq. (23) holds:
(n+H(p(n))
H(p(n) UBPerplexity AI = (1 + ) > ω(p(n)) > H(p(n) = LBPerplexity AI. (23)
H(p(n)) = − ∑ p(n)lnp(n).
Proof: following Taylor’s expansion of the exponential function, e , for x > 0, Eq. (24) holds:
e = ∑ ( ) > x. (24)
Moreover, engaging Lehmer’s inequality [19], inequality Eq. (25) holds x x 2
e ≤ (1 + ) , for x, n > 0. (25)
Since, inequality Eq. (26) is always true,
( ) (x+n) x 2 x
(1 + ) < (1 + ) holds for all x, n > 0. (26)
Setting x = H(p(n)), the proof follows.
Mathematically speaking, for all b ≥ 2, it holds by Eq. (27), that
(λ+1)
lim ωb(λ) = lim b = ∞.
λ→∞ λ→∞ (27)
Accordingly, Eq. (27) motivates the visual validation on how the base, namely, b ≥ 2 influences the increasability phases of ωb(λ), which declares the significant impact of that choice of base. This can be seen from Fig. 3.

As observed from Fig. 6, ωb(λ) drastically increases by the progressive increase of the base, b ≥ 2. This clarifies the significance of b ≥ 2, on the mathematical configuration of perplexity AI.
4. Closing Remarks with Next Phase of Research
The current influential work showcases for the first time ever the fine-tuning of AI, is revealed by using a novel revolutionary mathematical approach. The analytic findings are validated numerically. Notably, the upper and lower bounds of Perplexity AI are calculated.
The following are some emerging open problems:
I. The limitations [9] suggest that more diverse data representing real interactions with LM Models (LLMs) would be beneficial for better analysis. Additionally, the use of the default attack algorithm as a black box and the reliance on GPT-2 for perplexity measurements may introduce biases that could change with further training or different algorithms. These limitations emphasize the need for caution and further exploration in the evaluation of machine-generated adversarial attacks.
II. The work in [11] has discussed areas for improvement in a LM debunker by analyzing wrongly predicted samples, through highlighting possible challenges such as high perplexity for true claims with abnormal sentence structures and mistakes in refuting false claims by negating the paraphrased content. This triggers an open , yet unsolved till current, which is the feasibility of exploring ways to differentiate perplexity as a measure of sentence quality from a falseness indicator and improving the handling of special linguistic features like negation in evidence selection.
III. Assuming other media, such as Twitter or Spoken English, and other languages, such as Russian, the true entropy of written English text remains undetermined.
IV. Can we replace H(x), by other entropies, such as Rényi, Tsallis, or ℤ- entropies to fine tune the perplexity?
The question is still open.
V. Is it feasible to calculate strict upper and lower bounds for perplexity AI?
VI. How can we analytically fine tune ωb(λ) (Eq. (6) of Theorem 1) from a mathematical threshold perspective?
The question is still unsolved.
The next phase of research includes attempting to solve the proposed complex-still-open to solve problems, as well as the exploration of more unprecedented innovative mathematical techniques to advance AI.
Author Contributions
Ismail A. Mageed: Conceptualization, methodology, mathematical modeling, writing – original draft, review & editing.
Funding
This research was self-funded and did not receive any external financial support.
Data Availability
The data supporting the findings of this study can be made available upon reasonable request from the
Conflicts of Interest
The author declares no conflicts of interest regarding this publication.
References
- [1] Pimentel, A., Wagener, A., da Silveira, E. F., Picciani, P., Salles, B., Follmer, C., & Oliveira Jr, O. N. (2023). Challenging ChatGPT with chemistry-related subjects. ChemRxiv, 1–13. https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/646d22f3b3dd6a65309f5119/original/challenging-chat-gpt-with-chemistry-related-subjects.pdf
- [2] Li, X., & Qiu, X. (2023). Mot: Memory-of-thought enables chatgpt to self-improve. ArXiv Preprint ArXiv:2305.05181. https://arxiv.org/abs/2305.05181
- [3] Vasilatos, C., Alam, M., Rahwan, T., Zaki, Y., & Maniatakos, M. (2023). Howkgpt: investigating the detection of chatgpt-generated university student homework through context-aware perplexity analysis. ArXiv Preprint ArXiv:2305.18226.
- [4] Mageed, I. A. (2023). Fractal dimension (DF) theory of ismail’s second entropy (HQI) with potential fractal applications to chatgpt, distributed ledger technologies (DLTS) and image processing (IP). 2023 international conference on computer and applications (ICCA) (pp. 1–6). IEEE. ( )
- [5] Mageed, I. A. (2023). Fractal dimension (DF) of ismail’s fourth entropy (h iv q, a 1, a 2,.., a k ) with fractal applications to algorithms, haptics, and transportation. 2023 international conference on computer and applications (ICCA) (pp. 1–6). IEEE. https://ieeexplore.ieee.org/abstract/document/10401780/
- [6] Mageed, I. A., & Zhang, Q. (2022). An information theoretic unified global theory for a stable m/g/1 queue with potential maximum entropy applications to energy works. 2022 global energy conference (GEC) (pp. 300– 305). IEEE. https://ieeexplore.ieee.org/abstract/document/9986719/
- [7] Mageed, I. A., & Bhat, A. H. (2022). Generalized Z-Entropy (Gze) and fractal dimensions. Appl. math, 16(5), 829–834. https://www.academia.edu/download/90028778/160517.pdf
- [8] Goto, A., & Katanoda, K. (2023). Should we acknowledge ChatGPT as an author? Journal of epidemiology, 33(7), 333–334.
- [9] Alon, G., & Kamfonas, M. (2023). Detecting language model attacks with perplexity. ArXiv Preprint ArXiv:2308.14132.
- [10] Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. ArXiv Preprint ArXiv:2307.15043.
- [11] Lee, N., Bang, Y., Madotto, A., & Fung, P. (2020). Misinformation has high perplexity. ArXiv Preprint ArXiv:2006.04666.
- [12] Vigdor, N. (2020). Man fatally poisons himself while self-medicating for coronavirus, doctor says. The new york times, 24. https://www.nytimes.com/2020/03/24/us/chloroquine-poisoning-coronavirus.html
- [13] Mageed, I. A. (2024). Masterminding the giant: shannonian entropic threshold perplexity AI. Preprints, 1–8. https://doi.org/10.20944/preprints202405.1274.v1
- [14] Mageed, I. A., & Zhang, Q. (2023). Threshold theorems for the tsallisian and rényian (TR) cumulative distribution functions (CDFs) of the Heavy-tailed stable M/G/1 queue with tsallisian and rényian entropic applications to satellite images (SIs). Electronic journal of computer science and information technology, 9(1), 41–
- [47] https://ejcsit.uniten.edu.my/index.php/ejcsit/article/view/161
- [15] Gravina, A. G., Pellegrino, R., Palladino, G., Imperio, G., Ventura, A., & Federico, A. (2024). Charting new AI education in gastroenterology: cross-sectional evaluation of ChatGPT and perplexity AI in medical residency exam. Digestive and liver disease, 56(8), 1304–1311. https://www.sciencedirect.com/science/article/pii/S1590865824003025
- [16] Uppalapati, V. K., & Nag, D. S. (2024). A comparative analysis of ai models in complex medical decision-making scenarios: evaluating ChatGPT, Claude AI, bard, and perplexity. Cureus, 16(1). https://doi.org/10.7759%2Fcureus.52485
- [17] Deike, M. (2024). Evaluating the performance of ChatGPT and perplexity AI in business reference. Journal of business & finance librarianship, 29(2), 125–154. DOI:10.1080/08963568.2024.2317534
- [18] Xu, Z., & Sheng, V. S. (2024). Detecting ai-generated code assignments using perplexity of large language models [presentation]. Proceedings of the aaai conference on artificial intelligence (Vol. 38, pp. 23155–23162).
- [19] Liu, X., & Kong, L. (2024). AI text detection method based on perplexity features with strided sliding window. Working notes of clef, 3740, 1–6. https://ceur-ws.org/Vol-3740/paper-261.pdf
- [20] Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., & Paul, M. (2024). Perplexed by perplexity: perplexity-based data pruning with small reference models. ArXiv Preprint ArXiv:2405.20541. https://arxiv.org/abs/2405.20541
- [21] Fitria, T. N. (2024). Using Chatbot-based artificial intelligence (AI) for writing an english essay: the ability of ChatGPT, perplexity AI, and ChatSonic. Journal of language intelligence and culture, 6(2), 103–128. https://jlic.uinkhas.ac.id/index.php/jlic/article/view/139
- [22] Mageed, I. A., Zhang, Q., & Modu, B. (2023). The linearity theorem of rényian and tsallisian maximum entropy solutions of the heavy-tailed stable M/G/1 queueing system entailed with potential queueing-theoretic applications to cloud computing and IoT. Electronic journal of computer science and information technology, 9(1), 15–23.