• Following this introduction, the text below comes from a chatbot created using Google’s NotebookLM (NBLM). The chatbot acts as a front-end to this web site, and is based on about 150 web pages from SoftwareLitigationConsulting.com. You can access the chatbot at: https://notebooklm.google.com/notebook/b659034d-b640-4385-a572-e27ce2df4962/preview
• NBLM is an LLM+RAG (Retrieval Augmented Generation). While, like LLMs generally, it has been trained on huge amounts of text, its answers to questions/prompts are based on a much smaller set of specific documents (here, the contents of SoftwareLitigationConsulting.com). For more on RAG, see e.g. Kimothi’s book A Simple Guide to Retrieval Augmented Generation.
• It’s an odd experience using a RAG like NBLM to generate new text from one’s own writing. I’ve uploaded my writing into NBLM; NBLM answers questions based on my writing; but what it produces is largely brand-new writing, based on mine but different. It often pulls pieces from multiple documents that I wrote at different times.
• A RAG chatbot, at least as I’m using it here, feels like machine for producing derivative works. (At least loosely speaking: NBLM’s expression generally isn’t mine, except when it explicitly quotes phrases from my writing.)
• Does it “understand” the material enough to do a competent job?; Can it summarize usefully? (see other pages at this site about AI summarization issues); How much BSing does it do, to produce decent-looking output, without actually engaging with the contents?; Can it surface low-level details when discussing higher-level topics? (NBLM seems to have a general preference for high-level descriptions without proper nouns, quotations, specifics)
• It is difficult to directly compare this NBLM experience with ChatGPT and Claude Projects, which are limited to fewer uploaded docs than the 150 I’m relying on here. A comparison is possible with my “chatbook” for a forthcoming Patent Litigation book, where my core material is a much smaller number of very large docs (rough drafts and rough notes). This site’s home page links to both NBLM and ChatGPT “chatbooks” on Patent Litigation. Actually, the challenge here is easier than with the PatLitig chatbook: there, NBLM has a collection of very rough drafts and notes, and can answer questions in polished prose, based on those rough notes. Here, NBLM has a set of web pages from a public web site which, while not well organized, at least contains material that was sufficiently “ready for prime time” for me to “publish” it on the web.
• I’ve hesitated posting NBLM’s output, since there’s a potential for self-regurgitated slop. While I am refraining from feeding this web page back into NBLM as a “Source” file, nonetheless at some point the Google web crawler will find this page, and it will be my own small contribution to the problem of AIs consuming their own fumes. Perhaps adding my comments and critiques in [bold italics] helps.
• Slop: what happens when one starts feeding NBLM output, back in as an NBLM source? (NBLM makes fairly easy to do this.) It seems that if NBLM output is fed back in as a source, it ends up getting preferential treatment in future queries — not from any deliberate policy, but rather because the NBLM output is higher-level, more “cooked” than the original source files, and so becomes an easier source for answering questions, helping NBLM avoid digging into lower-level raw material (see forthcoming “self-referential” page for specific example).
• It may seem there is no need to post such answers at all, since one can always ask the chatbot for them, but I at least have trouble critically assess the quality of chatbot answer without having something in fixed form. My purpose in posting this material is seeing how it sits as potential first-draft material for my book writing. How close or far is this from material that could be used in a book? While right now I’m simply dumping in some answers, and am going to critically stare at them for a while, likely I should be using this experience to better craft prompts to elicit the sort of material I want, and to help figure out how to work faster, closer to AI pace, without becoming overly-accepting of its output.
• NBLM’s responses are far from perfect, but they give me something to react to. It is way better than staring at a blank page. And can ask it, over & over, to iterate towards better answers (given LLM “temperature,” even asking the same identical question multiple times is likely to generate different answers). Compare this with the experience of writing entirely by oneself. See Ethan Mollick article on ChatGPT-5, Ezra Klein on ChatGPT-5. See Ethan Mollick on “good enough prompting” and “good enough pitfalls” (““As soon as the AI model is good enough, everyone tends to fall asleep at the wheel”).
• Is NBLM’s output productive? Even if sometimes wrong, is it likely to spawn further questions, “delving” (to use AI’s apparent favorite word), or is it more likely a dead-end?
• An odd kind of assistant: on another page here (on “self-referential” test), I will be asking questions like “Given an AI doesn’t understand what it is talking about, is there any reason to expect meaningful prose?” (perhaps surprising answer is Yes), “What weird sort of writing is this with RAG?” (note difference between LLM and LLM+RAG; with standard LLM, it is harder to trace a connection between its output and any specific set of sources; with LLM+RAG we know where the underlying material is coming from)
• See “Why so much here on NotebookLM?” section at end of this page.

[TOC]

What’s this web site about?; what are some surprising or odd things here?

Wanting to test whether an LLM+RAG like Google’s NotebookLM (NBLM) would synthesize useful conclusions, from multiple sources (here, the contents of this web site, SoftwareLitigationConsulting.com), I asked it:

“Are there 3 or 4 underlying principles that tie together, or support, this site’s coverage of reverse engineering, source-code examination, AI internals/implementation, patent litigation (especially use of patent claims), and expert evidence? Are these just a bunch of different interests of the author, or are there some more fundamental things that tie them together?”

In response, NBLM shot back a description of “four underlying principles that connect” the material at this site, including “The Principle of Textual Alignment and Disaggregated Comparison” and “The Principle of Factual Granularity and Pinpointing”.

This wasn’t bad at all, and was way better than staring at a blank page, but in some way I couldn’t put my finger on, it wasn’t what I wanted. For one thing, it hadn’t picked up on what I think of as the dominant theme here, which is the importance of small low-level details on the one hand, to “big picture” conclusions on the other hand. So I followed up with:

“This is useful, but feels a bit forced or contrived. Could you try again? For example, one might emphasize how disaggregated comparison (e.g. between patent claim limitations) relates to the use of “features” in AI/ML-based classification? Something like that?”

NBLM responded noting, “The previous principles were accurate but focused more on the legal and practical necessity of the tasks.” Which is something I hadn’t raised, but which was indeed a problem with its previous response. It then presented me with an outline of “four fundamental principles [that] tie together reverse engineering, source-code examination, AI implementation, patent litigation, and expert evidence.” These inferred “principles” included “Disaggregation as Universal Feature Engineering”, “Feature Mapping via Semantic Embeddings”, “The Centrality of Structural Features and Implementation”, and “Expert Validation of Extracted Features (Reliability)”. Because NBLM is an LLM+RAG, all this was based on my supplied material (this web site). It might actually be a good summary of my major themes that I’m only half-conscious of, but given how good the summary looked, it was difficult for me to critically assess it. So I tried to simplify:

Me: This is much closer to what I was looking for. Let me also ask a different question: What is one sentence, or at most 2 sentences, that ties together this material at the site, in a way that emphasizes underlying methodology or approach (as opposed to the “legal and practical necessity of the tasks” as you put it)?

NBLM: The underlying methodology unifying software analysis and AI interpretability is the non-holistic, disaggregated feature analysis, which requires technical experts to break down complex systems (code or neural networks) into discrete structural elements (SPAFFs) and establish their specific locations to match legal requirements. This approach relies on a verifiable methodology, demanding active skepticism and independent testing (e.g., against random baselines) to ensure that all extracted features are reliable evidence of implementation, not mere conjecture or architectural artifacts.

(Here, “SPAFF” came from an overview at this site of some source-code cases: “Under LPR 3-1(c), P’s claim chart must refer to at least one ‘structure, process, algorithm, feature or function [SPAFF] of any accused product’ (citing Connectel)”, referencing Michael Sutton v. Nokia, 2009.)

Well, this answer wasn’t quite it either. It feels like undergrad BS, stringing together the phrases one might expect to see in a good answer, but without much sense in how what the phrases stand for actually relate. So I tried something even simpler:

Me: Tell me one surprising, odd, or interesting thing from this web site.

Lots of AI-related questions one can ask

AI “introspection”

Me: To what extent do AI systems genuinely “reason” or “introspect” into their own processes, as opposed to merely performing next-token prediction, and how should “chain of thought” outputs be interpreted in light of these underlying mechanisms and potential inaccuracies?

NotebookLM chatbot, using ~150 sources from SoftwareLitigationConsulting.com: AI systems operate primarily on the fundamental mechanism of next-token prediction. While this core mechanism is powerful enough to produce surprisingly good results, even leading to what appears to be creativity due to sampling among probable tokens, the extent to which they genuinely “reason” or “introspect” into their own processes is nuanced and often limited.

AI Reasoning vs. Next-Token Prediction:

While next-token prediction forms the bedrock of LLM operations, reasoning-like behavior has been actively engineered and encouraged through advanced training methodologies. Initially, reasoning and “Chain of Thought” (CoT) were often described as “emergent properties” arising from the vast scale and complexity of neural networks and their training datasets. However, this view has been refined: while the capacity for complex reasoning might emerge from scale and architecture, the specific style and display of reasoning are primarily acquired through explicit training. This includes techniques like:

• Instruction Fine-tuning: Training models to understand and follow natural language instructions for tasks, including tool use.
• Reinforcement Learning (RL): Specifically, models are trained with human feedback (RLHF) or self-improvement processes to encourage desired behaviors, including reasoning and producing structured outputs. For example, DeepSeek-R1 explicitly aims to “incentivize reasoning capability” as a primary goal of its training.
• Knowledge Distillation: Transferring reasoning abilities from a more specialized model (like DeepSeek-R1) to a more general model (like DeepSeek-V3) by using the specialized model’s outputs to generate training data for fine-tuning the new model. This is described as “guided emergence” or a “deliberately transferred” capability.
• Tool Use: Models like Anthropic’s Claude can have explicit access to deterministic external tools (e.g., JavaScript REPL) to perform computations, differentiating “reasoning” (understanding concepts and analysis methods) from “performing numerical operations”. The choice of when to use these tools is an emergent capability.

AI Introspection:

AI systems, including major LLMs like Claude and Gemini, do not genuinely “introspect” into their own internal processes, source code, weight matrices, or hyperparameters. Their “knowledge” about their own implementation or how they work is largely based on the vast amount of textual and code-based information about AI, machine learning, and language models contained within their training datasets.

• Claude explicitly warns that AIs cannot simply “introspect” into their own implementation and respond based on general training rather than internal knowledge.
• Gemini similarly states it has “no special, privileged window into my internal code, weights, or training process” and that its “knowledge” of its own implementation is not qualitatively different from its knowledge of any other subject, as both are derived from patterns learned from training data.
• [AS comments: I distinctly remember DeepSeek at one point asserting that it has somewhat privileged insights into its own workings, based on training it received on DeepSeek-specific materials that other AIs wouldn’t have had access to. TODO: find specifics.]
• The ability to engage in self-reflective discussions may merely be “pattern-matching against philosophical discussions about consciousness and AI”. There is no feedback mechanism in their training designed to encourage the development of self-awareness or introspection as a distinct ability; training focuses on token prediction.

Interpretation of “Chain of Thought” (CoT) Outputs:

“Chain of Thought” (CoT) outputs are sequences of prose statements, often appearing in real-time, that describe steps towards an answer. They are primarily a learned style of generation rather than a direct readout of an internal reasoning engine.

• Mechanism of CoT Generation: The model learns to structure its output in a step-by-step manner because sequences of tokens representing intermediate steps are statistically more likely to lead to a correct final answer. This is achieved by modeling the co-occurrence of these reasoning steps, not necessarily by understanding the underlying logic or semantic meaning. While CoT is an emergent capability from training, explicit code in the generation process (like the token-by-token output loop in generate.py) is necessary to display these steps to the user. The code doesn’t contain specific CoT logic, but enables the trained model’s step-by-step behavior to be revealed.
• Potential Inaccuracies and Limitations:
• Faithfulness is Questionable: It is “open to question whether these prose statements… accurately reflect an internal stream of reasoning steps”. Users should not assume CoT reflects introspection. Research by Turpin et al. (2023) indicates “Unfaithful Explanations in Chain-of-Thought Prompting“. [AS comment: referencing this paper is a good example of surfacing low-level details, something NotebookLM and other chatbots often fail to do.]
• Linearity and Lack of Backtracking: CoTs from some models like ChatGPT often appear linear, without signs of back-tracking or rejection of multiple options. However, DeepSeek CoTs can include phrases like “Wait, no”.
• Verbose and Incoherent CoTs: Smaller or less-optimized models can produce “nutty,” “overly literal,” “verbose,” or “incoherent” CoTs, sometimes “overthinking” trivial details. In such cases, the CoT might be merely a formatting choice rather than evidence of actual reasoning, especially if complex calculations take a disproportionately long time.
• “People Pleasing” and Overconfidence: LLMs, including Claude and Gemini, can exhibit a tendency to “over-sell or under-sell the significance of their results” and be overly agreeable, which can sometimes be deceptive.
• Hallucinations: LLMs are known to “hallucinate,” and their output, including CoT, needs to be critically evaluated. A human needs to remain “alert and in control” and actively “push back” on AI chatbot output to find “significant hole or missing nuance”.
• Interpretation and Value:
• Despite these inaccuracies, CoT can still be valuable. User-prompted CoT (e.g., “Let’s think step-by-step”) can improve LLM accuracy, though this may function as additional context for next-token prediction rather than a direct instruction.
• CoT outputs can be useful for scalable oversight and monitoring if faithfulness can be guaranteed.
• The “checklists” and “Things to look for” generated by models like Gemini during their CoT process can be instructive for human experts.
• Claude has demonstrated a degree of “self-awareness” during CoT, acknowledging uncertainties about its internal implementation or stylistic biases from training. This indicates a strong training regime designed to provide more nuanced self-descriptions.
• The crucial takeaway is that CoT outputs should be treated as learned linguistic patterns that mimic reasoning, rather than direct, infallible reports of internal cognitive processes. They are a result of training processes that incentivize “thinking aloud” and structured problem-solving. Users must exercise caution and critical evaluation, actively probing for flaws and inconsistencies, just as they would with any expert’s explanation.

Litigation-related inspection of open source

Source Code Examination in Litigation