<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>The Data Savvy Corner</title>
<link>https://thedatasavvycorner.netlify.app/</link>
<atom:link href="https://thedatasavvycorner.netlify.app/index.xml" rel="self" type="application/rss+xml"/>
<description>Musings, ramblings, generally putting things down in text so I don&#39;t forget them.</description>
<generator>quarto-1.10.18</generator>
<lastBuildDate>Sun, 02 Aug 2026 00:00:00 GMT</lastBuildDate>
<item>
  <title>PDP Assist</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/09-pdp-assist.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/09-pdp-assist.jpg" alt="A cozy desk with an open notebook showing a winding learning path"></p>
</div>
<section id="what-it-is" class="level2">
<h2 class="anchored" data-anchor-id="what-it-is">What It Is</h2>
<p>A side-project demo of an AI learning coach for junior engineers. You pick a career journey, rate your skills, answer a few coach questions, and get a pathway built from real course modules — then put the training on a calendar.</p>
<p>The learning content is aimed at Azure skills. The coach uses Google Vertex Gemini when it is set up; if not, a local fallback still runs the demo. There is no sign-in. Progress is saved in the browser. Use <strong>Reset demo</strong> if you want to start over.</p>
<p>Live at <a href="https://pdp-assist.vercel.app/" target="_blank">pdp-assist.vercel.app</a>.</p>
</section>
<section id="the-interesting-bit" class="level2">
<h2 class="anchored" data-anchor-id="the-interesting-bit">The Interesting Bit</h2>
<p>You curate a library of training. The coach then builds a plan from that library, tailored to the individual developer — their journey, skill gaps, and how much time they can give. Same catalogue; different pathway for each person.</p>
</section>
<section id="try-it" class="level2">
<h2 class="anchored" data-anchor-id="try-it">Try It</h2>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://pdp-assist.vercel.app/" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://pdp-assist.vercel.app/" title="PDP Assist Demo">
  </iframe>
</div>
</section>
<section id="next-steps" class="level2">
<h2 class="anchored" data-anchor-id="next-steps">Next Steps</h2>
<p>The demo is rough on purpose. Natural next steps:</p>
<ul>
<li><strong>From demo to workplace tool</strong> — real sign-in and saved accounts, so a personal development plan survives more than one browser.</li>
<li><strong>Capability frameworks</strong> — map skills to something like the <a href="https://ddat-capability-framework.service.gov.uk/" target="_blank">DDaT Capability Framework</a> so pathways can support promotion evidence, not just a reading list.</li>
<li><strong>A wider catalogue</strong> — plug in Microsoft Learn, Pluralsight, or internal courses, while still only picking real catalogue IDs.</li>
<li><strong>A closed loop</strong> — finish sessions, reassess skills, refresh the pathway. Optional views for managers or L&amp;D for check-ins and cohorts.</li>
<li><strong>Calendar outside the tab</strong> — export to ICS, sync with Outlook or Google, Slack reminders before a study block.</li>
<li><strong>Nearby use cases</strong> — graduate programmes, contractor onboarding, career switches (analyst to engineer), anonymised skill-gap views across a team.</li>
</ul>
<p>Static learning catalogues are easy. Sticking to a plan is the hard part — that is the gap this is aiming at.</p>
<hr>
<p><em>Between “here’s a learning portal” and “I actually did the modules” is a lot of empty calendar.</em></p>


</section>

 ]]></description>
  <category>idea</category>
  <category>experiment</category>
  <category>ai</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/09-pdp-assist.html</guid>
  <pubDate>Sun, 02 Aug 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/09-pdp-assist.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Just Enough Shell to Survive</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/22-shell-for-ai-engineers.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/22-shell-for-ai-engineers/intro-img.webp" alt="A glowing cyberpunk Koopa shell hovering above a neon terminal" width="614" height="410" loading="lazy" decoding="async" style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;height:auto;"></p>
<div class="callout callout-style-simple callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>A note on platforms
</div>
</div>
<div class="callout-body-container callout-body">
<p>This tutorial assumes a <strong>Linux-like terminal</strong> — bash or zsh on macOS, Linux, GitHub Codespaces, or similar (see How to use this tutorial). That is where most AI tooling, containers, and CI pipelines expect you to work.</p>
<p>On <strong>Windows</strong>, things can look a little different: keyboard shortcuts, path separators (<code>\</code> vs <code>/</code>), and some command names may not match. <strong>WSL</strong> (Windows Subsystem for Linux) or <strong>Git Bash</strong> are good ways to follow along with minimal friction. If you are using PowerShell or Command Prompt, treat this guide as conceptually useful, but expect differences in detail.</p>
</div>
</div>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>If you build AI products, sooner or later you end up stitching together a small orchestra of tools:</p>
<ul>
<li>local scripts</li>
<li>API calls</li>
<li>logs</li>
<li>config files</li>
<li>containers</li>
<li>CI pipelines</li>
</ul>
<p>The shell is often the quickest way to make these components cooperate. This tutorial covers <strong>just enough</strong> shell to survive day-to-day AI engineering — navigation, safety habits, glue pipelines, and hands-on challenges — without pretending you need to become a terminal wizard.</p>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Engineers who are comfortable writing code but have little (or no) shell experience. If you are already fluent in shell, skip ahead to process control and service debugging.</p>
</section>
<section id="what-youll-need" class="level3">
<h3 class="anchored" data-anchor-id="what-youll-need">What You’ll Need</h3>
<ul class="task-list">
<li><label><input type="checkbox">A <strong>terminal</strong> — the window where you type shell commands (also called a shell or command line)</label></li>
<li><label><input type="checkbox">Basic familiarity with files and folders</label></li>
<li><label><input type="checkbox">Curiosity and willingness to try small experiments</label></li>
</ul>
</section>
</section>
<section id="choose-your-environment" class="level2">
<h2 class="anchored" data-anchor-id="choose-your-environment">How To Use This Tutorial</h2>
<p>Pick how you want to follow along. Your choice in the tabs below <strong>filters the Try It panels and challenges</strong> in the rest of the tutorial — you only see exercises that match your environment. Switch tabs any time; no need to read warnings in every section about what works where.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-1-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-1" aria-controls="tabset-1-1" aria-selected="true" href="">Browser sandbox</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-2" aria-controls="tabset-1-2" aria-selected="false" href="">Local terminal</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-3" aria-controls="tabset-1-3" aria-selected="false" href="">GitHub Codespaces</a></li></ul>
<div class="tab-content">
<div id="tabset-1-1" class="tab-pane active" aria-labelledby="tabset-1-1-tab">
<a id="browser-sandbox"></a>
<p>Use the <strong>shell sandbox</strong> panel at the bottom of this page — a simulated bash environment that runs in your browser, not on your machine.</p>
<p><strong>Getting started</strong></p>
<ul>
<li>Click <strong>Show sandbox</strong> to expand the panel. It loads once and stays available as you scroll.</li>
<li>Many sections include a <strong>challenge</strong>: click <strong>Start challenge</strong> to reset a clean environment, complete the task, then <strong>Check my work</strong> for automated feedback.</li>
</ul>
<p><strong>Good to know</strong></p>
<ul>
<li>This is a learning sandbox, not a full Linux terminal — behaviour differs in places (permissions, processes, networking, keyboard shortcuts).</li>
<li>Sessions are ephemeral — refreshing the page usually resets your environment.</li>
<li>Do <strong>not</strong> paste secrets, tokens, private keys, or credentials.</li>
</ul>
<p><strong>Best for:</strong> navigation, pipes, grep, globs, quoting, glue workflows, and automated challenges.</p>
<p><strong>Switch to Local or Codespaces for:</strong> Tab completion, Ctrl+R/C, line-editing shortcuts, background jobs, <code>curl</code>, your real shell config files, and anything that needs full host fidelity.</p>
</div>
<div id="tabset-1-2" class="tab-pane" aria-labelledby="tabset-1-2-tab">
<a id="local-terminal"></a>
<p>Your machine’s own shell — Terminal.app or iTerm on macOS, the default terminal on Linux, <strong>WSL</strong> or <strong>Git Bash</strong> on Windows, or similar.</p>
<p><strong>Getting started</strong></p>
<ul>
<li>Open your terminal application and follow the <strong>Try It</strong> steps in each section.</li>
<li>Challenge <strong>Check my work</strong> buttons use the browser sandbox — here, self-check against the expected output in each exercise.</li>
</ul>
<p><strong>What you get</strong></p>
<ul>
<li>Full keyboard support: <strong>Tab</strong> completion, <strong>Ctrl+R</strong>, modifier keys, and the rest of the shortcuts in the hints section.</li>
<li>Real process IDs, signals, ports, and filesystem behaviour.</li>
<li><strong><code>curl</code></strong>, <strong><code>jq</code></strong>, and outbound network access (when installed).</li>
</ul>
<p><strong>Platform notes</strong></p>
<ul>
<li>This tutorial assumes a <strong>Linux-like</strong> shell (bash or zsh). See A note on platforms for Windows differences.</li>
<li>Your config files (<code>~/.zshrc</code>, <code>~/.bashrc</code>) and command history behave as on any normal system.</li>
</ul>
<p><strong>Best for</strong></p>
<ul>
<li>Tab completion, modifier-key practice, Ctrl+R, process debugging (section 7), and API probes (section 8).</li>
</ul>
</div>
<div id="tabset-1-3" class="tab-pane" aria-labelledby="tabset-1-3-tab">
<a id="what-is-codespaces"></a>
<p><strong>GitHub Codespaces</strong> is a cloud development environment — a Linux virtual machine with VS Code in your browser and a real terminal, hosted by GitHub.</p>
<p><strong>Getting started</strong></p>
<ul>
<li><a href="https://docs.github.com/en/codespaces/getting-started/quickstart" target="_blank" rel="noopener noreferrer">Codespaces quickstart</a> (opens in a new tab)</li>
<li>Or open a repository on GitHub and launch a Codespace from the green <strong>Code</strong> button.</li>
</ul>
<p><strong>What you get</strong></p>
<ul>
<li>The same full terminal capabilities as a <strong>local terminal</strong> tab — real Linux, full keyboard shortcuts, processes, ports, and network tools.</li>
<li>A good option if you do not have a local shell set up yet, or you want Linux parity without installing WSL.</li>
</ul>
<p><strong>Best for</strong></p>
<ul>
<li>Same as <strong>Local terminal</strong> when you want a real Linux shell without local setup.</li>
</ul>
<p><a href="https://docs.github.com/en/codespaces" target="_blank" rel="noopener noreferrer">GitHub Codespaces documentation</a> (opens in a new tab).</p>
</div>
</div>
</div>
<section id="core-vocabulary" class="level3">
<h3 class="anchored" data-anchor-id="core-vocabulary">A few words we reuse constantly</h3>
<ul>
<li><strong>Command</strong> — a program name you type at the prompt (e.g.&nbsp;<code>ls</code>, <code>grep</code>)</li>
<li><strong>Flag</strong> — an option after a command, usually starting with <code>-</code> (e.g.&nbsp;<code>ls -la</code>)</li>
<li><strong>Pipeline</strong> — commands chained with <code>|</code>, where each command’s output feeds the next (covered properly in section 2)</li>
<li><strong>Prompt</strong> — the line where you type, ending in <code>$</code> (or <code>%</code> on some shells)</li>
</ul>
<p>See the full Glossary at the end of this tutorial for every command and term with links back to where it is introduced.</p>
<link rel="stylesheet" href=".././www/shell-challenges/shell-tryit-panels.css">
<link rel="preconnect" href="https://esm.sh" crossorigin="">
<div id="shell-dock" class="shell-dock shell-dock--collapsed" aria-label="Interactive shell sandbox panel">
  <div class="shell-dock-header">
    <div>
      <p class="shell-dock-title">Interactive shell sandbox</p>
      <p class="shell-dock-hint">Click to expand. Starts in a few seconds.</p>
    </div>
    <button type="button" class="shell-dock-toggle" aria-expanded="false">Show sandbox</button>
  </div>
  <div class="shell-dock-body">
    <p class="shell-dock-status">Starting sandbox…</p>
    <iframe data-src="/www/shell-sandbox/index.html" title="Interactive shell sandbox" loading="lazy">
    </iframe>
  </div>
</div>
</section>
</section>
<section id="hints-and-tips" class="level2">
<h2 class="anchored" data-anchor-id="hints-and-tips">Hints and Tips</h2>
<p>A handful of habits that pay off immediately. These are worth learning early, before the command vocabulary gets more involved.</p>
<section id="use-arrow-keys-to-navigate-prior-commands-work-smart-not-hard" class="level3">
<h3 class="anchored" data-anchor-id="use-arrow-keys-to-navigate-prior-commands-work-smart-not-hard">Use arrow keys to navigate prior commands (work smart, not hard)</h3>
<p>The shell remembers commands you have already run. You do not need to retype a long pipeline every time you want to tweak it.</p>
<ul>
<li><strong>Up arrow</strong> — step back through previous commands</li>
<li><strong>Down arrow</strong> — step forward again through that history</li>
</ul>
<p>This is especially useful when you are iterating on something fiddly: a <strong><code>grep</code></strong> search (you will meet it in section 3), a <strong><code>curl</code></strong> HTTP call (section 8), or a script you keep running with small edits. A <strong>pipeline</strong> chains commands with <code>|</code> — also coming in section 2. Run the command once, press <strong>Up</strong>, adjust a word or two, press <strong>Enter</strong>.</p>
</section>
<section id="try-it" class="level3">
<h3 class="anchored" data-anchor-id="try-it">Try It</h3>
<p>Open the panel and build a little history, then walk back through it with the arrow keys:</p>
<ol type="1">
<li>Run <strong><code>pwd</code></strong></li>
<li>Run <strong><code>ls</code></strong></li>
<li>Run <strong><code>echo "history practice"</code></strong></li>
<li>Run <strong><code>ls -la</code></strong></li>
</ol>
<p>Now navigate without retyping:</p>
<ol start="5" type="1">
<li>Press <strong>Up</strong> once — you should see <code>ls -la</code> at the prompt</li>
<li>Press <strong>Up</strong> again — <code>echo "history practice"</code></li>
<li>Press <strong>Up</strong> again — <code>ls</code></li>
<li>Press <strong>Down</strong> twice — forward again to <code>echo "history practice"</code>, then <code>ls -la</code></li>
<li>Press <strong>Enter</strong> to run whichever command is showing</li>
</ol>
<button type="button" class="shell-dock-open-btn">Open sandbox for this exercise</button>
</section>
<section id="use-tab-to-complete-matches" class="level3">
<h3 class="anchored" data-anchor-id="use-tab-to-complete-matches">Use Tab to complete matches</h3>
<p>Tab completion is the shell’s way of finishing your typing for you. Start a command or path, press <strong>Tab</strong>, and the shell suggests the rest based on what exists in your current directory or what it recognises as a valid command name.</p>
<ul>
<li>Type the first few letters of a filename, then press <strong>Tab</strong></li>
<li>If there is only one match, the shell fills in the rest</li>
<li>If there are several matches, press <strong>Tab</strong> again (or twice quickly) to see the options</li>
</ul>
<p>This saves time and cuts down on typos — particularly handy for long paths like <code>project/src/eval_metrics.py</code> when you only need to type <code>project/src/e</code> and hit <strong>Tab</strong>.</p>
</section>
<section id="try-it-1" class="level3">
<h3 class="anchored" data-anchor-id="try-it-1">Try It</h3>
<div data-shell-compat="local codespaces">
<p>Create a file with <strong><code>touch my-long-filename.txt</code></strong>, then type <strong><code>cat my-l</code></strong> and press <strong>Tab</strong> to complete the name before pressing <strong>Enter</strong>.</p>
</div>
</section>
<section id="jump-around-the-command-line-with-modifier-keys" class="level3">
<h3 class="anchored" data-anchor-id="jump-around-the-command-line-with-modifier-keys">Jump around the command line with modifier keys</h3>
<p>Once a command gets long, tapping the arrow key character-by-character is painful. On macOS, modifier keys let you move in bigger jumps:</p>
<ul>
<li><strong>⌘ + ←</strong> — jump to the start of the line</li>
<li><strong>⌘ + →</strong> — jump to the end of the line</li>
<li><strong>⌥ + ←</strong> — jump back one word at a time</li>
<li><strong>⌥ + →</strong> — jump forward one word at a time</li>
</ul>
<p>This is useful when you need to fix a typo in the middle of a pipeline without retyping the whole thing. Jump to the word, edit it, <strong>Enter</strong>.</p>
<p>On Linux terminals, the equivalents are usually <strong>Ctrl+A</strong> / <strong>Ctrl+E</strong> for the start and end of the line, and <strong>Alt + ←</strong> / <strong>Alt + →</strong> to move word by word.</p>
</section>
<section id="try-it-2" class="level3">
<h3 class="anchored" data-anchor-id="try-it-2">Try It</h3>
<div data-shell-compat="local codespaces">
<p>Type a long command such as the one below — it uses <strong><code>grep</code></strong> (search lines), <strong><code>wc -l</code></strong> (count lines), and <strong><code>|</code></strong> (pipe output along). You do not need to memorise it yet; the goal is cursor practice:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> ERROR app.log <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> retry <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">wc</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span></span></code></pre></div></div>
</div>
<p>Use <strong>⌘ + ←</strong> and <strong>⌥ + →</strong> to move the cursor without retyping. On Linux, try <strong>Ctrl+A</strong> and <strong>Alt + →</strong> instead.</p>
</div>
</section>
<section id="search-your-command-history" class="level3">
<h3 class="anchored" data-anchor-id="search-your-command-history">Search your command history</h3>
<p>Pressing <strong>Up</strong> walks back one command at a time. That is fine for recent history, but less fine when you ran something useful twenty commands ago and only remember a fragment of it.</p>
<p><strong>Ctrl+R</strong> starts a reverse search through your command history:</p>
<ol type="1">
<li>Press <strong>Ctrl+R</strong></li>
<li>Type a few characters you remember (e.g.&nbsp;<code>grep</code>, <code>docker</code>, <code>curl</code>)</li>
<li>The shell shows the most recent matching command</li>
<li>Press <strong>Ctrl+R</strong> again to step to the next older match</li>
<li>Press <strong>Enter</strong> to run it, or <strong>Esc</strong> / the arrow keys to edit first</li>
</ol>
<p>This pays off in specific scenarios:</p>
<ul>
<li>You ran a long <code>curl</code> or <code>docker</code> command yesterday and need it again</li>
<li>You are debugging and want to rerun a <code>grep</code> pipeline with a small tweak</li>
<li>You know <em>part</em> of a command but not enough to retype the whole thing</li>
</ul>
<p>If <strong>Ctrl+R</strong> is not available, you can list and filter history manually with the built-in <strong><code>history</code></strong> command:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">history</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> grep</span></code></pre></div></div>
</div>
</section>
<section id="try-it-3" class="level3">
<h3 class="anchored" data-anchor-id="try-it-3">Try It</h3>
<div data-shell-compat="local codespaces">
<p>If you have not done the arrow-key exercise above, run through steps 1–9 there first. With those four commands already in your history, try reverse search:</p>
<ol type="1">
<li>Press <strong>Ctrl+R</strong></li>
<li>Type <strong><code>echo</code></strong> — the shell should surface <code>echo "history practice"</code></li>
<li>Press <strong>Ctrl+R</strong> again to step to any older match that contains <code>echo</code></li>
<li>Press <strong>Enter</strong> to run the command, or use the arrow keys to edit it first</li>
</ol>
</div>
</section>
<section id="know-your-zshrc-and-history-file" class="level3">
<h3 class="anchored" data-anchor-id="know-your-zshrc-and-history-file">Know your <code>~/.zshrc</code> and history file</h3>
<p>On macOS, your default shell is usually <strong>zsh</strong>. When you open a new terminal tab, zsh reads a config file in your home directory called <strong><code>~/.zshrc</code></strong> (pronounced “zsh-R-C”). Think of it as startup instructions for your shell.</p>
<p>Typical things people store in <code>~/.zshrc</code>:</p>
<ul>
<li><strong>Aliases</strong> — shortcuts for commands you use often</li>
<li><strong><code>export</code> lines</strong> — environment variables such as <code>PATH</code> (where the shell looks for programs)</li>
<li><strong>Tool setup</strong> — many installers (Conda, nvm, Homebrew) ask you to paste a line into <code>~/.zshrc</code> so their commands work in every new terminal</li>
</ul>
<p>That last point is often <em>why</em> you need to configure it. A tool installs successfully, but the command is “not found” until you add its setup snippet to <code>~/.zshrc</code> and open a fresh terminal (or run <strong><code>source ~/.zshrc</code></strong> to reload the file into your current shell).</p>
<p>Your command <strong>history</strong> lives in a separate file — usually <strong><code>~/.zsh_history</code></strong>. That is how <strong>Ctrl+R</strong> and the <strong>Up</strong> arrow can find commands from previous days, not just the current session. The history file is the shell’s memory; <code>~/.zshrc</code> is its personality.</p>
<p>On Linux with <strong>bash</strong>, the same ideas apply but the files are often named <code>~/.bashrc</code> and <code>~/.bash_history</code> instead.</p>
<div class="callout callout-style-simple callout-warning">
<div class="callout-body d-flex">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-body-container">
<p>Be careful what you put in <code>~/.zshrc</code>. It runs automatically every time you open a terminal. Do <strong>not</strong> store secrets (API keys, passwords) here unless you understand the security implications — and never commit this file to a public repository.</p>
</div>
</div>
</div>
</section>
<section id="try-it-4" class="level3">
<h3 class="anchored" data-anchor-id="try-it-4">Try It</h3>
<div data-shell-compat="local codespaces">
<p>On your local machine, run <code>echo $SHELL</code> to see which shell you are using, then <code>ls -la ~/.zshrc ~/.zsh_history</code> (or <code>~/.bashrc</code> / <code>~/.bash_history</code> on Linux). Open <code>~/.zshrc</code> in your editor and see whether you recognise any aliases or <code>export</code> lines already there.</p>
</div>
</section>
<section id="stop-a-runaway-command-with-ctrlc" class="level3">
<h3 class="anchored" data-anchor-id="stop-a-runaway-command-with-ctrlc">Stop a runaway command with Ctrl+C</h3>
<p>Sometimes you paste the wrong thing, start a command that will not finish, or realise mid-run that you made a mistake. <strong>Ctrl+C</strong> sends an interrupt signal and stops the command at the current prompt.</p>
<ul>
<li>Press <strong>Ctrl+C</strong> once and wait a moment — most commands exit cleanly</li>
<li>If nothing happens, press <strong>Ctrl+C</strong> again (some programs need a second nudge)</li>
<li>You get a fresh prompt; nothing is “broken” — you can try again</li>
</ul>
<p>This is the first reflex to build. It is faster and safer than closing the terminal window when something is hanging.</p>
</section>
<section id="try-it-5" class="level3">
<h3 class="anchored" data-anchor-id="try-it-5">Try It</h3>
<div data-shell-compat="local codespaces">
<p>Run <strong><code>sleep 60</code></strong>, then press <strong>Ctrl+C</strong> before the minute is up. You should return to the prompt immediately.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sleep</span> 60</span></code></pre></div></div>
</div>
</div>
</section>
<section id="when-things-go-wrong-read-the-error-first" class="level3">
<h3 class="anchored" data-anchor-id="when-things-go-wrong-read-the-error-first">When things go wrong, read the error first</h3>
<p>Error messages look intimidating, but they usually tell you <em>which</em> of a few common problems you hit. A quick scan saves a lot of guessing.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 33%">
<col style="width: 33%">
<col style="width: 33%">
</colgroup>
<thead>
<tr class="header">
<th>Message</th>
<th>Likely cause</th>
<th>What to try</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>command not found</code></td>
<td>Typo, or the program is not installed / not on your <strong><code>PATH</code></strong> (the list of directories the shell searches for programs)</td>
<td>Check spelling; run <strong><code>which</code></strong> <em>name</em> to locate a command; check install docs</td>
</tr>
<tr class="even">
<td><code>Permission denied</code></td>
<td>You lack rights to read, write, or execute that file</td>
<td><strong><code>ls -l</code></strong> the path (long listing shows permissions); <strong><code>chmod +x</code></strong> adds execute permission for scripts; check you own the file</td>
</tr>
<tr class="odd">
<td><code>No such file or directory</code></td>
<td>Wrong path, or you are in the wrong folder</td>
<td><strong><code>pwd</code></strong> (print working directory); <strong><code>ls</code></strong> (list files); check for typos in the filename</td>
</tr>
</tbody>
</table>
<p>The shell is not scolding you — it is pointing at the mismatch between what you asked for and what exists on disk (or in your <code>PATH</code>).</p>
</section>
<section id="try-it-6" class="level3">
<h3 class="anchored" data-anchor-id="try-it-6">Try It</h3>
<p>Trigger each error on purpose and read the message before fixing it. Run these one at a time.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb4-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">typo_command</span></span></code></pre></div></div>
</div>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb5-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> secret.txt</span></code></pre></div></div>
</div>
<div data-shell-compat="local codespaces">
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> locked.txt</span></code></pre></div></div>
</div>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb7-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">chmod</span> 000 locked.txt</span></code></pre></div></div>
</div>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb8-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> locked.txt</span></code></pre></div></div>
</div>
</div>
<p><strong><code>touch</code></strong> creates an empty file. <strong><code>chmod</code></strong> changes permissions. Elsewhere in this tutorial, <strong><code>&amp;&amp;</code></strong> chains commands so each step runs only if the previous one succeeded.</p>
<button type="button" class="shell-dock-open-btn">Open sandbox for this exercise</button>
</section>
<section id="quote-paths-and-patterns-carefully" class="level3">
<h3 class="anchored" data-anchor-id="quote-paths-and-patterns-carefully">Quote paths and patterns carefully</h3>
<p>The shell splits your command on <strong>spaces</strong> unless you tell it not to. Quotes control that — and they behave differently.</p>
<p>Before the shell runs a command, it often <strong>replaces</strong> shorthand in what you typed with the real value. That replacement step is called <strong>expansion</strong>. For example, if you write <code>echo $HOME</code>, the shell expands <code>$HOME</code> to your home directory path (such as <code>/Users/you</code>) and then runs <code>echo</code> with that path. Globs like <code>*.txt</code> expand to a list of matching filenames — you will meet those in section 4.</p>
<p>Quotes tell the shell how much expansion to allow:</p>
<ul>
<li><strong>Single quotes <code>'...'</code></strong> — everything inside is literal. The shell does not expand anything; it passes the text exactly as you typed it.</li>
<li><strong>Double quotes <code>"..."</code></strong> — variables like <code>$HOME</code> still expand; most other characters are literal.</li>
</ul>
<p>Set a filename with a space:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb9-1"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FILE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"my report.txt"</span></span></code></pre></div></div>
</div>
<p>With quotes, one file:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb10-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$FILE</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span></code></pre></div></div>
</div>
<p>Without quotes, the shell splits on spaces — with <code>FILE="my report.txt"</code>, <code>touch $FILE</code> tries to create <code>my</code> and <code>report.txt</code> as two separate files instead of one:</p>
<div data-shell-compat="local codespaces">
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb11-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$FILE</span></span></code></pre></div></div>
</div>
</div>
<p>If a path or filename might contain spaces, wrap it in <strong>double quotes</strong>. When you expand a variable in a path, wrap it in double quotes — <code>"$TARGET"</code> treats the whole value as one path even when it contains spaces. For <strong><code>grep</code></strong> patterns and quotes, see section 3.</p>
<div id="quotes-1" class="shell-challenge">
<ol class="shell-challenge-steps">
<li>
<p>Create a file whose name contains a space.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb12-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"notes draft.txt"</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Print the file’s contents to confirm it exists.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb13-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"notes draft.txt"</span></span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
<section id="environment-variables" class="level3">
<h3 class="anchored" data-anchor-id="environment-variables">Environment variables and <code>.env</code> files</h3>
<p>An <strong>environment variable</strong> is a named value the shell (and child programs) can read. You set one for the current session like this:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb14" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb14-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">export</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">MODEL_NAME</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>gpt-4o-mini</span>
<span id="cb14-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$MODEL_NAME</span></span></code></pre></div></div>
</div>
<ul>
<li><strong><code>export</code></strong> makes the variable visible to commands you run from this shell</li>
<li><strong><code>echo $NAME</code></strong> prints the current value (the <code>$</code> tells the shell to expand it)</li>
<li>Variables set this way last until you close the terminal tab</li>
</ul>
<p>Putting <code>export</code> lines in <code>~/.zshrc</code> (see above) makes them available every time you open a new terminal. That is fine for non-secret defaults like <code>export EDITOR=code</code>.</p>
<p>For <strong>API keys and tokens</strong>, prefer a <strong><code>.env</code> file</strong> in your project root instead of hard-coding secrets in your shell config:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb15-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># .env — loaded by your app or tooling, not committed to git</span></span>
<span id="cb15-2"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">OPENAI_API_KEY</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>sk-...</span>
<span id="cb15-3"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">LOG_LEVEL</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>info</span></code></pre></div></div>
</div>
<p>Most Python and Node AI stacks load <code>.env</code> automatically (via <code>python-dotenv</code>, <code>dotenv</code>, or framework conventions). Add <code>.env</code> to <code>.gitignore</code> so secrets never land in a public repository.</p>
<div class="callout callout-style-simple callout-warning">
<div class="callout-body d-flex">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-body-container">
<p>Never paste API keys into the browser sandbox, commit them to git, or store them in a public gist. Use <code>.env</code> locally and platform secret stores in CI (GitHub Actions secrets, etc.).</p>
</div>
</div>
</div>
<p>For a one-off command in your terminal, you can set a variable inline without touching a file. <strong><code>curl</code></strong> fetches a URL over HTTP (section 8); <strong><code>head</code></strong> prints the first few lines of output — handy when a response is long:</p>
<div data-shell-compat="local codespaces">
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb16" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb16-1"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">OPENAI_API_KEY</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>sk-your-key <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-s</span> https://api.openai.com/v1/models <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span></span></code></pre></div></div>
</div>
</div>
<p>That sets the variable only for that single command — useful for quick tests, but <code>.env</code> is easier to maintain across a project.</p>
</section>
<section id="try-it-7" class="level3">
<h3 class="anchored" data-anchor-id="try-it-7">Try It</h3>
<p>Set a variable and read it back:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb17" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb17-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">export</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DEMO_VAR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>hello-shell</span>
<span id="cb17-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$DEMO_VAR</span></span>
<span id="cb17-3"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Model would be: </span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$DEMO_VAR</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span></code></pre></div></div>
</div>
<button type="button" class="shell-dock-open-btn">Open sandbox for this exercise</button>
</section>
</section>
<section id="shell-fundamentals" class="level2">
<h2 class="anchored" data-anchor-id="shell-fundamentals">1) Shell Fundamentals</h2>
<p>We’ll begin with a tiny command set that you will use repeatedly.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb18" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb18-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">pwd</span>          <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># print working directory — where am I?</span></span>
<span id="cb18-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span>           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># list files and folders here</span></span>
<span id="cb18-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-la</span>       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># -l long format; -a include hidden (dot) entries</span></span>
<span id="cb18-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mkdir</span> tmp    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># create a folder called tmp here</span></span>
<span id="cb18-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span>           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># tmp should appear in this listing</span></span>
<span id="cb18-6"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">cd</span> tmp       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># change directory</span></span>
<span id="cb18-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> visited.txt</span>
<span id="cb18-8"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">cd</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-</span>         <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># jump back to previous directory</span></span></code></pre></div></div>
</div>
<p><strong><code>mkdir</code></strong> creates a directory. <strong><code>touch</code></strong> creates an empty file (or updates a timestamp).</p>
<div id="fundamentals-1" class="shell-challenge">
<p><strong>Start challenge</strong> seeds a hidden <strong><code>.sandbox_marker</code></strong> file — you will need <strong><code>ls -la</code></strong> in step 3 to spot it.</p>
<ol class="shell-challenge-steps">
<li>
<p>Print your current working directory.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb19" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb19-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">pwd</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>List files in the current directory (without hidden entries).</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb20" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb20-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>List all files, including hidden ones.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb21" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb21-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-la</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Create a folder named <strong><code>tmp</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb22" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb22-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mkdir</span> tmp</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>List files again and confirm <strong><code>tmp</code></strong> appears.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb23" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb23-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Change into <strong><code>tmp</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb24" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb24-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">cd</span> tmp</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Create an empty file named <strong><code>visited.txt</code></strong> inside <strong><code>tmp</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb25" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb25-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> visited.txt</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Return to the directory you were in before <strong><code>cd tmp</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb26" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb26-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">cd</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-</span></span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p><strong>Check yourself:</strong> Where were you before <strong><code>cd tmp</code></strong>? What hidden entries did <strong><code>ls -la</code></strong> show? Where did <strong><code>cd -</code></strong> take you?</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
<section id="data-flow-pipes-and-redirection" class="level2">
<h2 class="anchored" data-anchor-id="data-flow-pipes-and-redirection">2) Data Flow: Pipes and Redirection</h2>
<p>The shell gets powerful when you connect simple tools together.</p>
<ul>
<li><strong><code>printf</code></strong> — print formatted text (like <code>echo</code>, but handles newlines reliably)</li>
<li><strong><code>cat</code></strong> — print a file’s contents to the terminal</li>
<li><strong><code>grep</code></strong> — print lines that match a pattern (more in section 3)</li>
</ul>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb27" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb27-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"apple\nbanana\napricot\n"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> fruits.txt</span>
<span id="cb27-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> fruits.txt</span>
<span id="cb27-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'^a'</span> fruits.txt</span>
<span id="cb27-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'^a'</span> fruits.txt <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">wc</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span></span></code></pre></div></div>
</div>
<p>The <strong><code>^</code></strong> in <code>'^a'</code> means “starts with” — only lines that <strong>begin with</strong> <strong><code>a</code></strong> match (<code>apple</code>, <code>apricot</code>, but not <code>banana</code>).</p>
<ul>
<li><code>&gt;</code> writes output to a file (overwrites existing content).</li>
<li><code>|</code> passes output from one command into the next.</li>
<li><strong><code>wc -l</code></strong> — <strong>w</strong>ord <strong>c</strong>ount, lines only. Counts how many lines are in the input (a file or piped output). Useful when you want a total, not every matching line.</li>
</ul>
<div id="pipes-1" class="shell-challenge">
<p>Only <strong><code>blueberry</code></strong> starts with <strong><code>b</code></strong> — that is the line your count should find.</p>
<ol class="shell-challenge-steps">
<li>
<p>Create <strong><code>fruits.txt</code></strong> with three lines: <strong><code>apple</code></strong>, <strong><code>apricot</code></strong>, and <strong><code>blueberry</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb28" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb28-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'%s\n'</span> apple apricot blueberry <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> fruits.txt</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Count how many fruits start with <strong><code>b</code></strong> and append that number to <strong><code>b-count.txt</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb29" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb29-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'^b'</span> fruits.txt <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">wc</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;</span> b-count.txt</span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
<section id="grep-chaining" class="level2">
<h2 class="anchored" data-anchor-id="grep-chaining">3) Grep Chaining</h2>
<p><strong><code>grep</code></strong> searches text for lines matching a pattern. It is one of the best shell tools for quick investigations.</p>
<ul>
<li><strong><code>grep 'PATTERN' file</code></strong> — lines containing <code>PATTERN</code></li>
<li><strong><code>grep -v 'PATTERN'</code></strong> — <strong>v</strong> invert: lines that do <em>not</em> match (exclude noise)</li>
</ul>
<section id="set-up-a-practice-log" class="level3">
<h3 class="anchored" data-anchor-id="set-up-a-practice-log">Set up a practice log</h3>
<p>Create <strong><code>app.log</code></strong> before the Try Its below:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb30" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb30-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"INFO startup\nWARN cache_miss\nERROR timeout\nERROR retry_failed\nERROR healthcheck ping\nlevel=warn cache_miss\nlevel=error oom\n"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> app.log</span></code></pre></div></div>
</div>
<p>On a <strong>local terminal</strong>, create it once and reuse it for every exercise in this section.</p>
<p>In the <strong>browser sandbox</strong>, files persist while you work — but <strong>Start challenge</strong> on any exercise wipes the environment and starts fresh. When you <strong>Start</strong> the <strong><code>grep-1</code></strong> challenge later in this section, <strong><code>app.log</code></strong> is reloaded for you with the same content as above. Until then, run the <strong><code>printf</code></strong> yourself whenever you need the file.</p>
<p>See what you are working with:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb31" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb31-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> app.log</span></code></pre></div></div>
</div>
<p>Try a few searches:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb32" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb32-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR'</span> app.log</span></code></pre></div></div>
</div>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb33" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb33-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'INFO'</span> app.log</span></code></pre></div></div>
</div>
<p>You should see every line that contains <code>ERROR</code>, and only the startup line for <code>INFO</code>.</p>
</section>
<section id="quotes-and-patterns" class="level3">
<h3 class="anchored" data-anchor-id="quotes-and-patterns">Quotes and patterns</h3>
<p>Single quotes keep the search pattern literal; double quotes let variables expand (see quoting if <strong>expansion</strong> is new).</p>
<p>Variable in the pattern — set the variable, then use double quotes:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb34" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb34-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">export</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">LOG_LEVEL</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>warn</span>
<span id="cb34-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"level=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$LOG_LEVEL</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> app.log</span></code></pre></div></div>
</div>
<p>You should see the line containing <code>level=warn</code>.</p>
</section>
<section id="chaining-filters" class="level3">
<h3 class="anchored" data-anchor-id="chaining-filters">Chaining filters</h3>
<p>Pipe <strong><code>grep</code></strong> commands together to narrow noisy logs:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb35" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb35-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR'</span> app.log <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> retry <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> healthcheck</span></code></pre></div></div>
</div>
<p>You should see only <code>ERROR timeout</code>.</p>
<p>The <code>^</code> in patterns like <code>'^a'</code> means “starts with” (used in section 2).</p>
<p>This approach is useful for narrowing noisy logs quickly:</p>
<ol type="1">
<li>broad match (<code>ERROR</code>)</li>
<li>exclude known noise (<code>-v retry</code>, <code>-v healthcheck</code>)</li>
</ol>
</section>
<section id="ai-engineer-use-case" class="level3">
<h3 class="anchored" data-anchor-id="ai-engineer-use-case">AI Engineer Use Case</h3>
<p>When a local model service fails, this pattern helps isolate relevant errors before you disappear into a debugging rabbit hole.</p>
<div id="grep-1" class="shell-challenge">
<p><strong>Start challenge</strong> pre-loads <strong><code>app.log</code></strong>. If you are practising without <strong>Start</strong>, create it first with the <strong><code>printf</code></strong> from Set up a practice log.</p>
<ol class="shell-challenge-steps">
<li>
<p>Filter <strong><code>app.log</code></strong> for <strong><code>ERROR</code></strong> lines, excluding anything containing <strong><code>retry</code></strong> or <strong><code>healthcheck</code></strong>, and save the result to <strong><code>grep-results.txt</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb36" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb36-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR'</span> app.log <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> retry <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> healthcheck <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> grep-results.txt</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Display <strong><code>grep-results.txt</code></strong> to verify the filter worked.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb37" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb37-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> grep-results.txt</span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p>You should see only <strong><code>ERROR timeout</code></strong>.</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
</section>
<section id="globs-and-brace-expansion" class="level2">
<h2 class="anchored" data-anchor-id="globs-and-brace-expansion">4) Globs and Brace Expansion</h2>
<section id="globs" class="level3">
<h3 class="anchored" data-anchor-id="globs">Globs</h3>
<p>A <strong>glob</strong> is a wildcard pattern the shell uses to match filenames. Before a command runs, the shell expands the pattern into a list of matching files.</p>
<p>Common wildcards:</p>
<ul>
<li><code>*</code> — matches any characters (e.g.&nbsp;<code>*.json</code> → all <code>.json</code> files)</li>
<li><code>?</code> — matches exactly one character (e.g.&nbsp;<code>file?.txt</code> → <code>file1.txt</code>, <code>fileA.txt</code>)</li>
</ul>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb38" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb38-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.txt</span>
<span id="cb38-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> project/src/<span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.py</span></code></pre></div></div>
</div>
<p>If you used <code>echo rm ./logs/*.json</code> earlier in the hints section, that <code>*</code> is a glob — <code>echo</code> lets you preview which files would be matched before you run <strong><code>rm</code></strong> (remove) for real.</p>
</section>
<section id="brace-expansion" class="level3">
<h3 class="anchored" data-anchor-id="brace-expansion">Brace expansion</h3>
<p>Brace expansion is a different trick — it generates text from a pattern in curly braces. It saves surprising amounts of time when scaffolding projects.</p>
<ul>
<li><strong><code>mkdir</code></strong> — create a directory; <strong><code>-p</code></strong> creates parent folders too if missing</li>
<li><strong><code>ls -R</code></strong> — list recursively (every subdirectory)</li>
</ul>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb39" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb39-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mkdir</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-p</span> project/<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">{data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">src</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">tests</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">logs}</span></span>
<span id="cb39-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> project/src/<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">{ingest</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">embed</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">serve}</span>.py</span>
<span id="cb39-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-R</span> project</span></code></pre></div></div>
</div>
<p>This is great for scaffolding small prototypes quickly.</p>
<div id="globs-1" class="shell-challenge">
<ol class="shell-challenge-steps">
<li>
<p>Create the <strong><code>project/src</code></strong> directory (create parent folders if needed).</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb40" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb40-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mkdir</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-p</span> project/src</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Create <strong><code>eval_metrics.py</code></strong> and <strong><code>eval_runner.py</code></strong> in one step using brace expansion.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb41" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb41-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">touch</span> project/src/<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">{eval_metrics</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">eval_runner}</span>.py</span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
</section>
<section id="simple-for-loops" class="level2">
<h2 class="anchored" data-anchor-id="simple-for-loops">5) Simple For Loops</h2>
<p>A <strong>for loop</strong> runs the same commands once per item in a list. That is how you batch-process files, hit several endpoints, or rerun a check across a folder without copy-pasting.</p>
<p>Basic shape:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb42" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb42-1"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> name <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> item1 item2 item3<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">;</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">do</span></span>
<span id="cb42-2">  <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Processing </span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$name</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb42-3"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">done</span></span></code></pre></div></div>
</div>
<ul>
<li><strong><code>for ... in ...</code></strong> — the list can be words you type, or a glob the shell expands</li>
<li><strong><code>do</code> / <code>done</code></strong> — wrap the commands to repeat</li>
<li><strong><code>$name</code></strong> — the current item (quote it as <code>"$name"</code> when it might contain spaces)</li>
</ul>
<p>Loop over files with a glob (see section 4). Here we use <strong><code>wc -l</code></strong> on each file directly — it prints how many lines that file contains (you met <code>wc -l</code> on piped output in section 2):</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb43" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb43-1"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> f <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.txt<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">;</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">do</span></span>
<span id="cb43-2">  <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"=== </span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$f</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"> ==="</span></span>
<span id="cb43-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">wc</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$f</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb43-4"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">done</span></span></code></pre></div></div>
</div>
<p>Quoting <code>"$f"</code> matters when filenames contain spaces — the same rule from the quoting hints.</p>
<section id="ai-engineer-use-case-1" class="level3">
<h3 class="anchored" data-anchor-id="ai-engineer-use-case-1">AI Engineer Use Case</h3>
<p>Got twelve eval output files and need the same <code>grep</code> on each? A loop beats running the command twelve times by hand:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb44" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb44-1"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> f <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> outputs/run_<span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.log<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">;</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">do</span></span>
<span id="cb44-2">  <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"== </span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$f</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"> =="</span></span>
<span id="cb44-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$f</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb44-4"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">done</span></span></code></pre></div></div>
</div>
<div id="for-loops-1" class="shell-challenge">
<p>The <strong><code>||</code></strong> in step 2 means “if <strong><code>grep</code></strong> finds nothing, run the command on the right instead” — here, print <strong><code>(no errors)</code></strong>.</p>
<ol class="shell-challenge-steps">
<li>
<p>Create a <strong><code>logs/</code></strong> folder with three files: <strong><code>alpha.log</code></strong> (INFO only), <strong><code>beta.log</code></strong> (one ERROR line), and <strong><code>gamma.log</code></strong> (ERROR plus WARN).</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb45" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb45-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mkdir</span> logs <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">&amp;&amp;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'INFO ok\n'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> logs/alpha.log <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">&amp;&amp;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR timeout\n'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> logs/beta.log <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">&amp;&amp;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR retry\nWARN slow\n'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> logs/gamma.log</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Loop over every <strong><code>.log</code></strong> file in <strong><code>logs/</code></strong> and print any line containing <strong><code>ERROR</code></strong>. If a file has no errors, print <strong><code>(no errors)</code></strong> instead.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb46" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb46-1"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> f <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> logs/<span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.log<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">;</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">do</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"== </span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$f</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"> =="</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$f</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">||</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"(no errors)"</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">;</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">done</span></span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p>You should see errors from <strong><code>beta.log</code></strong> and <strong><code>gamma.log</code></strong>, and <strong><code>(no errors)</code></strong> for <strong><code>alpha.log</code></strong>.</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
</section>
<section id="executables-and-permissions" class="level2">
<h2 class="anchored" data-anchor-id="executables-and-permissions">6) Executables and Permissions</h2>
<p>A script file is not executable by default, even if it contains valid shell code.</p>
<ul>
<li><strong><code>cat &gt; file &lt;&lt;'EOF'</code></strong> — write several lines to a file (a <strong>heredoc</strong>); text until <code>EOF</code> is saved into <code>file</code></li>
<li><strong><code>./hello.sh</code></strong> — run a script in the current directory (<code>./</code> means “here”)</li>
<li><strong><code>ls -l</code></strong> — long listing; shows permission letters including <code>x</code> for executable</li>
<li><strong><code>chmod +x</code></strong> — add e<strong>x</strong>ecute permission so you can run the script</li>
</ul>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb47" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb47-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> hello.sh <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;&lt;'EOF'</span></span>
<span id="cb47-2"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">#!/usr/bin/env bash</span></span>
<span id="cb47-3"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">echo "Hello from shell"</span></span>
<span id="cb47-4"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">EOF</span></span>
<span id="cb47-5"></span>
<span id="cb47-6"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> hello.sh</span>
<span id="cb47-7"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./hello.sh</span></span></code></pre></div></div>
</div>
<p>The first line <code>#!/usr/bin/env bash</code> is a <strong>shebang</strong> — it tells the system which interpreter to use when you run <code>./hello.sh</code>.</p>
<p>You should see a permission error before adding execute permission. That’s normal.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb48" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb48-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">chmod</span> +x hello.sh</span>
<span id="cb48-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> hello.sh</span>
<span id="cb48-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./hello.sh</span></span></code></pre></div></div>
</div>
<p>This pattern matters when packaging helper scripts for teammates, automation, or CI jobs.</p>
<div id="permissions-1" class="shell-challenge">
<ol class="shell-challenge-steps">
<li>
<p>Create <strong><code>hello.sh</code></strong> — a script that prints <strong><code>Hello from shell</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb49" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb49-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'%s\n'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'#!/usr/bin/env bash'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'echo "Hello from shell"'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> hello.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Try to run the script. You should get <strong>Permission denied</strong> — it is not executable yet.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb50" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb50-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./hello.sh</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Check the file permissions with a long listing.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb51" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb51-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> hello.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Make the script executable.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb52" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb52-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">chmod</span> +x hello.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Check permissions again.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb53" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb53-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> hello.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Run the script successfully.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb54" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb54-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./hello.sh</span></span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p>You should see <strong>Permission denied</strong> on the first run, then <strong><code>Hello from shell</code></strong> after <strong><code>chmod +x</code></strong>. <strong>Check my work</strong> verifies that behaviour — even if <strong><code>ls -l</code></strong> still shows <strong><code>-rw-r--r--</code></strong> in the sandbox. On a <strong>local terminal</strong> or in <strong>Codespaces</strong>, <strong><code>ls -l</code></strong> updates to show <strong><code>x</code></strong> after <strong><code>chmod +x</code></strong>.</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
<section id="dry-run-flag" class="level3">
<h3 class="anchored" data-anchor-id="dry-run-flag">Use a DRY_RUN flag before you trust a script</h3>
<p>Once you are writing scripts, you will often touch real files, call live APIs, or change application state — and hoping for the best is not a strategy.</p>
<p>A common pattern is a <strong>DRY_RUN</strong> flag. Instead of performing the action, the script prints what it <em>would</em> do. You inspect the output (on screen or in a log file). If it looks right, run again without the flag and trust the result.</p>
<p>Some familiar commands have a built-in safety switch. For example, <strong><code>rm</code></strong> (remove/delete) with the <strong><code>-i</code></strong> flag asks you to confirm before each delete:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb55" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb55-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rm</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-i</span> notes.txt</span></code></pre></div></div>
</div>
<p>For your own scripts, an environment variable keeps things simple:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb56" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb56-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># safe preview</span></span>
<span id="cb56-2"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DRY_RUN</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>1 <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./cleanup_old_embeddings.sh</span></span>
<span id="cb56-3"></span>
<span id="cb56-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># real run, only after the preview looked correct</span></span>
<span id="cb56-5"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./cleanup_old_embeddings.sh</span></span></code></pre></div></div>
</div>
<p>A minimal pattern inside a script might look like this. <strong><code>find</code></strong> searches the filesystem; <strong><code>-type f</code></strong> limits to files, <strong><code>-mtime +30</code></strong> finds files older than 30 days, and <strong><code>-delete</code></strong> removes them (we only run that branch when <code>DRY_RUN</code> is not set):</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb57" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb57-1"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">[</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$DRY_RUN</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1"</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">]</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">;</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">then</span></span>
<span id="cb57-2">  <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Would delete files older than 30 days in ./cache/"</span></span>
<span id="cb57-3"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span></span>
<span id="cb57-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">find</span> ./cache/ <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-type</span> f <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-mtime</span> +30 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-delete</span></span>
<span id="cb57-5"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">fi</span></span></code></pre></div></div>
</div>
<p>You can redirect dry-run output to a log for a paper trail. <strong><code>&gt;&gt;</code></strong> appends output to a file (you will not see it on screen unless you open the log):</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb58" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb58-1"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DRY_RUN</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>1 <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./cleanup_old_embeddings.sh</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;</span> dry-run.log</span></code></pre></div></div>
</div>
<p><strong><code>tee</code></strong> does both — it prints to the terminal <em>and</em> writes to the file at the same time. That is handy when you want to inspect the preview immediately while still keeping a record:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb59" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb59-1"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DRY_RUN</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>1 <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./cleanup_old_embeddings.sh</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tee</span> dry-run.log</span></code></pre></div></div>
</div>
<p>Use <code>&gt;&gt;</code> when you only need the log. Use <code>tee</code> when you want to read the output and save it in one step.</p>
<p>This is especially wise when a command is destructive, expensive, or hard to undo — bulk deletes, deployment scripts, or anything that hits production data.</p>
<p>For quick <strong>one-liners</strong>, you do not always need a full script. <strong><code>echo</code></strong> prints text to the terminal — use it to preview a destructive command first, then run the real thing once you are satisfied:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb60" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb60-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># preview</span></span>
<span id="cb60-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> rm ./logs/<span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.json</span>
<span id="cb60-3"></span>
<span id="cb60-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># enact</span></span>
<span id="cb60-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rm</span> ./logs/<span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.json</span></code></pre></div></div>
</div>
<p>With variables or wildcard patterns like <code>*.json</code>, <code>echo</code> shows you how the shell will expand things before anything irreversible happens:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb61" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb61-1"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TARGET</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"./cache/embeddings"</span></span>
<span id="cb61-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> rm <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-rf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$TARGET</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb61-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># rm -rf "$TARGET"</span></span></code></pre></div></div>
</div>
<p>The same idea works inside pipelines. Swap the destructive command for <code>echo</code> temporarily:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb62" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb62-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">find</span> ./tmp <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-name</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"*.bak"</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-exec</span> echo rm {} <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\;</span></span>
<span id="cb62-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># find ./tmp -name "*.bak" -exec rm {} \;</span></span></code></pre></div></div>
</div>
<p>If the echoed output looks right, rerun without <code>echo</code> and let the command do its job.</p>
<div id="dry-run-1" class="shell-challenge">
<ol class="shell-challenge-steps">
<li>
<p>Create <strong><code>dry_demo.sh</code></strong>. When <strong><code>DRY_RUN=1</code></strong>, it should print a preview message instead of doing anything destructive.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb63" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb63-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'%s\n'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'#!/usr/bin/env bash'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'if [ "$DRY_RUN" = "1" ]; then'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'  echo "Would delete files older than 30 days in ./cache/"'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'fi'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> dry_demo.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Make the script executable.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb64" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb64-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">chmod</span> +x dry_demo.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Run the script with <strong><code>DRY_RUN=1</code></strong> to preview what it would do.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb65" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb65-1"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DRY_RUN</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>1 <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./dry_demo.sh</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Run the script without <strong><code>DRY_RUN</code></strong> — it should do nothing visible.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb66" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb66-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./dry_demo.sh</span></span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Preview a destructive command safely with <strong><code>echo</code></strong> before you would run it for real.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb67" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb67-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> rm ./welcome.txt</span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
</section>
<section id="processes-ports-and-signals" class="level2">
<h2 class="anchored" data-anchor-id="processes-ports-and-signals">7) Processes, Ports, and Signals</h2>
<p>Now we’re in classic operations/debugging territory.</p>
<section id="start-and-inspect-a-process" class="level3">
<h3 class="anchored" data-anchor-id="start-and-inspect-a-process">Start and inspect a process</h3>
<ul>
<li><strong><code>sleep N</code></strong> — pause for N seconds (useful for testing)</li>
<li><strong><code>&amp;</code> at the end</strong> — run the command in the <strong>background</strong> so you get your prompt back</li>
<li><strong><code>jobs</code></strong> — list background tasks started from this shell</li>
<li><strong><code>ps</code></strong> — list running processes; <strong><code>-ef</code></strong> shows all processes in full format</li>
<li><strong>PID</strong> — <strong>p</strong>rocess <strong>id</strong>entifier; a numeric handle you pass to <code>kill</code></li>
</ul>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb68" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb68-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sleep</span> 300 <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">&amp;</span></span>
<span id="cb68-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">jobs</span></span>
<span id="cb68-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ps</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-ef</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> sleep <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> grep</span></code></pre></div></div>
</div>
<p>The second <code>grep -v grep</code> drops the <code>grep</code> process itself from the results — a common trick when searching for processes.</p>
</section>
<section id="stop-a-process-gracefully" class="level3">
<h3 class="anchored" data-anchor-id="stop-a-process-gracefully">Stop a process gracefully</h3>
<p><strong><code>kill PID</code></strong> sends a polite shutdown signal (replace <code>PID</code> with the number from <code>ps</code>):</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb69" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb69-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">kill</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span>PID<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span></span></code></pre></div></div>
</div>
</section>
<section id="force-stop-last-resort" class="level3">
<h3 class="anchored" data-anchor-id="force-stop-last-resort">Force stop (last resort)</h3>
<p><strong><code>kill -9 PID</code></strong> forces immediate termination — no cleanup:</p>
<div class="callout callout-style-default callout-warning callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Warning
</div>
</div>
<div class="callout-body-container callout-body">
<p>Use <code>kill -9</code> sparingly. It bypasses graceful shutdown and can leave temporary state or partial writes behind.</p>
</div>
</div>
</section>
<section id="check-which-process-is-using-a-port" class="level3">
<h3 class="anchored" data-anchor-id="check-which-process-is-using-a-port">Check which process is using a port</h3>
<p><strong><code>lsof</code></strong> — <strong>l</strong>ist <strong>o</strong>pen <strong>f</strong>iles; sockets count as files. <strong><code>-i :8000</code></strong> shows what is bound to port 8000:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb70" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb70-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">lsof</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-i</span> :8000</span></code></pre></div></div>
</div>
<p>In real projects, this is often the fastest answer to: “Why won’t my service start on this port?”</p>
</section>
<section id="try-it-8" class="level3">
<h3 class="anchored" data-anchor-id="try-it-8">Try It</h3>
<div data-shell-compat="local codespaces">
<p>Start a background job and find its process ID:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb71" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb71-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sleep</span> 120 <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">&amp;</span></span>
<span id="cb71-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">jobs</span></span>
<span id="cb71-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ps</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-ef</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> sleep <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> grep</span></code></pre></div></div>
</div>
<p>Copy the PID from the <code>ps</code> output (the number in the second column).</p>
<p>Stop it gracefully, then check it is gone:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb72" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb72-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">kill</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span>PID<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb72-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ps</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-ef</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> sleep <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> grep</span></code></pre></div></div>
</div>
<p>Replace <code>&lt;PID&gt;</code> with the number you copied. The second <code>ps</code> should return nothing.</p>
</div>
</section>
</section>
<section id="practical-ai-glue-workflows" class="level2">
<h2 class="anchored" data-anchor-id="practical-ai-glue-workflows">8) Practical AI Glue Workflows</h2>
<p>These small shell pipelines can save a lot of time during development. Work through them in order — each builds on files you created in earlier sections.</p>
<section id="filter-noisy-logs" class="level3">
<h3 class="anchored" data-anchor-id="filter-noisy-logs">Filter noisy logs</h3>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb73" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb73-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> app.log <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR'</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'healthcheck'</span></span></code></pre></div></div>
</div>
<div id="glue-logs-1" class="shell-challenge">
<p><strong>Start challenge</strong> loads <strong><code>app.log</code></strong>.</p>
<ol class="shell-challenge-steps">
<li>
<p>Filter <strong><code>app.log</code></strong> for <strong><code>ERROR</code></strong> lines, exclude anything containing <strong><code>healthcheck</code></strong>, and save the result to <strong><code>filtered-errors.txt</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb74" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb74-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'ERROR'</span> app.log <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-v</span> healthcheck <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> filtered-errors.txt</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Review <strong><code>filtered-errors.txt</code></strong> to see what you captured.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb75" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb75-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> filtered-errors.txt</span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p>You should see <strong><code>ERROR oom</code></strong> and <strong><code>ERROR timeout</code></strong> — no healthcheck line.</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
<section id="quick-file-inventory" class="level3">
<h3 class="anchored" data-anchor-id="quick-file-inventory">Quick file inventory</h3>
<p><strong><code>grep '\.py$'</code></strong> — lines ending in <code>.py</code> (<code>$</code> means end of line; <code>\.</code> is a literal dot). Useful for spotting Python files in a long listing:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb76" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb76-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-R</span> project <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'\.py$'</span></span></code></pre></div></div>
</div>
<div id="glue-inventory-1" class="shell-challenge">
<p><strong>Start challenge</strong> loads <strong><code>project/</code></strong> with three Python files.</p>
<ol class="shell-challenge-steps">
<li>
<p>Find every <strong><code>.py</code></strong> file under <strong><code>project/</code></strong> and save the listing to <strong><code>python-inventory.txt</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb77" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb77-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-R</span> project <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'\.py$'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> python-inventory.txt</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Review <strong><code>python-inventory.txt</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb78" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb78-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> python-inventory.txt</span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p>You should see at least three lines ending in <strong><code>.py</code></strong>.</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
<section id="verify-executable-helpers" class="level3">
<h3 class="anchored" data-anchor-id="verify-executable-helpers">Verify executable helpers</h3>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb79" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb79-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.sh</span></code></pre></div></div>
</div>
<div id="glue-scripts-1" class="shell-challenge">
<p><strong>Start challenge</strong> loads <strong><code>hello.sh</code></strong> (not executable yet).</p>
<ol class="shell-challenge-steps">
<li>
<p>Make <strong><code>hello.sh</code></strong> executable.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb80" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb80-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">chmod</span> +x hello.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Verify permissions with a long listing.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb81" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb81-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ls</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">*</span>.sh</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Run the script.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb82" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb82-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">./hello.sh</span></span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p>On a <strong>local terminal</strong> or in <strong>Codespaces</strong>, <strong><code>ls -l</code></strong> would also show <strong><code>x</code></strong> in the permission string after <strong><code>chmod +x</code></strong>.</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
</section>
<section id="probe-an-api-with-curl-and-parse-json-with-jq" class="level3">
<h3 class="anchored" data-anchor-id="probe-an-api-with-curl-and-parse-json-with-jq">Probe an API with <code>curl</code> (and parse JSON with <code>jq</code>)</h3>
<p>AI work involves a lot of HTTP — model APIs, embedding services, health checks. <strong><code>curl</code></strong> is the shell’s Swiss Army knife for sending requests and inspecting responses without opening Postman.</p>
<p>A minimal GET with the response body on stdout:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb83" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb83-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-s</span> https://httpbin.org/get</span></code></pre></div></div>
</div>
<ul>
<li><strong><code>-s</code></strong> — silent mode (hides the progress meter so pipes stay clean)</li>
</ul>
<p>When the response is JSON, <strong><code>jq</code></strong> filters and formats it — pass a path like <code>.headers.Host</code> to pull one field. Install <code>jq</code> locally if you do not have it (<code>brew install jq</code> on macOS).</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb84" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb84-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-s</span> https://httpbin.org/get <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jq</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'.headers.Host'</span></span></code></pre></div></div>
</div>
<p>For authenticated APIs, pass the key from an environment variable (set in your shell or loaded from <code>.env</code> — never hard-code it in the command history). <strong><code>-H</code></strong> adds an HTTP header:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb85" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb85-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-s</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-H</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Authorization: Bearer </span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$OPENAI_API_KEY</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb85-2">  https://api.openai.com/v1/models <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jq</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'.data[].id'</span></span></code></pre></div></div>
</div>
<p>You can practise the <code>jq</code> part without network access — save a fake response and parse it locally:</p>
<div id="jq-1" class="shell-challenge">
<ol class="shell-challenge-steps">
<li>
<p>Create <strong><code>response.json</code></strong> with sample API-style JSON.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb86" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb86-1"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">printf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'{"choices":[{"message":{"content":"pong"}}]}\n'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> response.json</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Inspect the file to see the raw JSON.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb87" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb87-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> response.json</span></code></pre></div></div>
</div>
</details>
</li>
<li>
<p>Extract the <strong><code>content</code></strong> field with <strong><code>jq</code></strong>.</p>
<details class="shell-challenge-reveal">
<summary>
Show command
</summary>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb88" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb88-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span> response.json <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jq</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'.choices[0].message.content'</span></span></code></pre></div></div>
</div>
</details>
</li>
</ol>
<p>You should see <strong><code>"pong"</code></strong> (JSON strings include the quote marks in <strong><code>jq</code></strong> output).</p>
<div class="shell-challenge-actions">
  <button type="button" class="shell-challenge-start">Start challenge</button>
  <button type="button" class="shell-challenge-check" disabled="">Check my work</button>
</div>
<p class="shell-challenge-status" aria-live="polite"></p>
</div>
<div data-shell-compat="local codespaces">
<p>On a <strong>local terminal</strong> or in <strong>Codespaces</strong>, probe a live URL with <strong><code>curl</code></strong> and <strong><code>jq</code></strong>:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>terminal</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb89" data-filename="terminal" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb89-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-s</span> https://httpbin.org/get <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jq</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'.headers.Host'</span></span></code></pre></div></div>
</div>
</div>
</section>
</section>
<section id="extension-codespaces" class="level2">
<h2 class="anchored" data-anchor-id="extension-codespaces">9) Extension: Run Advanced Commands Locally or in Codespaces</h2>
<p>Process debugging, live API calls, and advanced keyboard shortcuts assume a full terminal. Select <strong>Local terminal</strong> or <strong>GitHub Codespaces</strong> at the top of this tutorial if you have not already.</p>
</section>
<section id="pre-publish-checklist-for-your-own-shell-docs" class="level2">
<h2 class="anchored" data-anchor-id="pre-publish-checklist-for-your-own-shell-docs">10) Pre-Publish Checklist For Your Own Shell Docs</h2>
<ul>
<li>Are all interactive commands safe and reversible?</li>
<li>Did you clearly warn readers not to paste secrets?</li>
<li>Did you distinguish sandbox behavior from real host behavior?</li>
<li>Are advanced commands (<code>kill -9</code>, <code>lsof -i</code>) explained with caution?</li>
<li>Does each section include a concrete “why this matters” use case?</li>
</ul>
</section>
<section id="glossary" class="level2">
<h2 class="anchored" data-anchor-id="glossary">Glossary</h2>
<p>Quick reference for commands and terms in this tutorial. Each entry links to where it is introduced — use this when you meet something unfamiliar mid-challenge.</p>
<section id="concepts" class="level3">
<h3 class="anchored" data-anchor-id="concepts">Concepts</h3>
<ul>
<li><strong>Background job</strong> — A command running while you keep your prompt (<code>sleep 300 &amp;</code>).</li>
<li><strong>Brace expansion</strong> — Generate text from a pattern in curly braces (<code>project/{data,src}</code>).</li>
<li><strong>Command</strong> — A program name you type at the prompt (<code>ls</code>, <code>grep</code>).</li>
<li><strong>DRY_RUN</strong> — Preview pattern: run with a flag set so nothing destructive happens until you are sure.</li>
<li><strong>Environment variable</strong> — A named value programs can read (<code>export MODEL_NAME=...</code>).</li>
<li><strong>Flag</strong> — An option after a command, usually starting with <code>-</code> (<code>ls -la</code>, <code>grep -v</code>).</li>
<li><strong>For loop</strong> — Repeat commands for each item in a list (<code>for f in *.log; do ... done</code>).</li>
<li><strong>GitHub Codespaces</strong> — Cloud Linux dev environment with VS Code and a real terminal in the browser.</li>
<li><strong>Glob</strong> — Wildcard filename pattern (<code>*.json</code>, <code>logs/*.log</code>).</li>
<li><strong>Heredoc</strong> — Multi-line input into a file (<code>cat &gt; file &lt;&lt;'EOF'</code>).</li>
<li><strong>PATH</strong> — Directories the shell searches when you type a command name.</li>
<li><strong>PID</strong> — Process identifier — numeric handle for <code>kill</code>.</li>
<li><strong>Pipeline</strong> — Commands chained with <code>|</code>; output of one feeds the next.</li>
<li><strong>Prompt</strong> — Where you type commands, usually ending in <code>$</code>.</li>
<li><strong>Shebang</strong> — First line of a script (<code>#!/usr/bin/env bash</code>) telling the system which interpreter to use.</li>
</ul>
</section>
<section id="commands" class="level3">
<h3 class="anchored" data-anchor-id="commands">Commands</h3>
<ul>
<li><strong><code>cat</code></strong> — Print a file’s contents.</li>
<li><strong><code>cd</code></strong> — Change directory; <strong><code>cd -</code></strong> returns to the previous one.</li>
<li><strong><code>chmod</code></strong> — Change permissions; <strong><code>chmod +x</code></strong> makes a script executable.</li>
<li><strong><code>curl</code></strong> — Send HTTP requests from the terminal.</li>
<li><strong><code>echo</code></strong> — Print text (also used to preview destructive commands in DRY_RUN).</li>
<li><strong><code>export</code></strong> — Set an environment variable for child programs.</li>
<li><strong><code>find</code></strong> — Search the filesystem by name, type, age, etc.</li>
<li><strong><code>grep</code></strong> — Search lines matching a pattern; <strong><code>-v</code></strong> inverts (exclude matches).</li>
<li><strong><code>head</code></strong> — Print the first few lines of output.</li>
<li><strong><code>history</code></strong> — List commands you have run in this session.</li>
<li><strong><code>jobs</code></strong> — List background tasks from this shell.</li>
<li><strong><code>jq</code></strong> — Filter and format JSON (<code>jq '.field'</code>).</li>
<li><strong><code>kill</code></strong> — Send a signal to a process; <strong><code>-9</code></strong> forces immediate stop.</li>
<li><strong><code>ls</code></strong> — List files; <strong><code>-l</code></strong> long format; <strong><code>-a</code></strong> includes hidden entries; <strong><code>-R</code></strong> recursive.</li>
<li><strong><code>lsof</code></strong> — List open files; <strong><code>-i :PORT</code></strong> shows what is using a port.</li>
<li><strong><code>mkdir</code></strong> — Create a directory; <strong><code>-p</code></strong> creates parent folders too.</li>
<li><strong><code>printf</code></strong> — Print formatted text (reliable newlines for building files).</li>
<li><strong><code>ps</code></strong> — List processes; <strong><code>-ef</code></strong> shows all in full format.</li>
<li><strong><code>pwd</code></strong> — Print working directory (where am I?).</li>
<li><strong><code>rm</code></strong> — Remove files; <strong><code>-i</code></strong> asks before each delete.</li>
<li><strong><code>sleep</code></strong> — Pause for N seconds.</li>
<li><strong><code>source</code></strong> — Reload a config file into the current shell (<code>source ~/.zshrc</code>).</li>
<li><strong><code>tee</code></strong> — Write to a file and print to the terminal at the same time.</li>
<li><strong><code>touch</code></strong> — Create an empty file (or update timestamp).</li>
<li><strong><code>wc</code></strong> — Word count; <strong><code>-l</code></strong> counts lines.</li>
<li><strong><code>which</code></strong> — Show path to a command executable.</li>
</ul>
</section>
<section id="operators-and-syntax" class="level3">
<h3 class="anchored" data-anchor-id="operators-and-syntax">Operators and syntax</h3>
<ul>
<li><strong><code>&amp;</code></strong> — Run a command in the background.</li>
<li><strong><code>&amp;&amp;</code></strong> — Run the next command only if the previous one succeeded.</li>
<li><strong><code>||</code></strong> — Run the next command only if the previous one failed.</li>
<li><strong><code>|</code> (pipe)</strong> — Send one command’s output into the next.</li>
<li><strong><code>&gt;</code> / <code>&gt;&gt;</code></strong> — Redirect output to a file (overwrite / append).</li>
<li><strong><code>./script.sh</code></strong> — Run an executable in the current directory.</li>
<li><strong><code>$VAR</code></strong> — Expand an environment variable.</li>
<li><strong><code>'...'</code> / <code>"..."</code></strong> — Single quotes: literal text. Double quotes: allow variable expansion.</li>
</ul>
</section>
<section id="keyboard-shortcuts" class="level3">
<h3 class="anchored" data-anchor-id="keyboard-shortcuts">Keyboard shortcuts</h3>
<ul>
<li><strong>Ctrl+C</strong> — Stop a running command.</li>
<li><strong>Ctrl+R</strong> — Reverse-search command history.</li>
<li><strong>Tab</strong> — Complete filenames and command names.</li>
<li><strong>Modifier keys</strong> — Jump by word or to the start/end of the line.</li>
<li><strong>Up / Down arrows</strong> — Step through previous commands.</li>
</ul>
</section>
</section>
<section id="further-reading" class="level2">
<h2 class="anchored" data-anchor-id="further-reading">Further reading</h2>
<p>A short list of resources worth your time. Each one is widely respected, free, and does something this tutorial does not — deeper courses, correct bash habits, dense references, or tools you will use for years.</p>
<ul>
<li><p><a href="https://swcarpentry.github.io/shell-novice/"><strong>The Unix Shell</strong> (Software Carpentry)</a> — The best structured follow-on from here. Clear lessons, exercises, and a gentle ramp from navigation through scripting. If you liked the hands-on style of this tutorial, start here next.</p></li>
<li><p><a href="https://mywiki.wooledge.org/BashGuide"><strong>Bash Guide</strong></a> and <a href="https://mywiki.wooledge.org/BashPitfalls"><strong>Bash Pitfalls</strong></a> (Greg’s Wiki) — The bash community’s gold standard for doing things <em>correctly</em>. The guide teaches good habits; the pitfalls page is a catalogue of ways bash will surprise you. Essential once you start writing scripts that other people (or CI) will run.</p></li>
<li><p><a href="https://github.com/jlevy/the-art-of-command-line"><strong>The Art of Command Line</strong></a> — One dense page of practical CLI wisdom — file processing, debugging, one-liners, macOS and Windows notes. Legendary for a reason (160k+ GitHub stars). Keep it bookmarked and skim a section when you have five minutes.</p></li>
<li><p><a href="https://explainshell.com/"><strong>explainshell.com</strong></a> — Paste any command; get a visual breakdown of every flag and argument against the man page text. Invaluable when you copy a <code>curl</code>, <code>find</code>, or <code>tar</code> one-liner from Stack Overflow and want to know what it actually does before you run it.</p></li>
<li><p><a href="https://tldr.sh/"><strong>tldr pages</strong></a> — Practical examples for everyday commands (<code>tldr tar</code>, <code>tldr curl</code>). The man page when you need the full spec; tldr when you need to remember the flags you always forget. Install once (<code>brew install tldr</code> on macOS) and never leave home without it.</p></li>
<li><p><a href="https://www.shellcheck.net/"><strong>ShellCheck</strong></a> — Static analysis for shell scripts. Paste a script or hook it into your editor; it catches quoting bugs, portability issues, and logic traps before they bite you in production. Non-negotiable once you are writing glue scripts for real.</p></li>
<li><p><a href="https://wizardzines.com/zines/bite-size-bash/"><strong>Bite size bash</strong></a> (Julia Evans) — A short, illustrated zine on how bash <em>actually</em> works — quoting, <code>if</code> tests, variables, and the gotchas experienced programmers still trip over. Her <a href="https://jvns.ca/blog/2025/06/24/new-zine--the-secret-rules-of-the-terminal/">terminal deep-dives</a> are excellent if you want to understand <em>why</em> the terminal feels inconsistent (spoiler: it is four different programs pretending to be one).</p></li>
</ul>
</section>
<section id="wrap-up" class="level2">
<h2 class="anchored" data-anchor-id="wrap-up">Wrap-up</h2>
<p>Shell skills compound quickly. Start with navigation and pipes, add pattern matching and loops, then permissions and process control, and apply them to your own tool chain. Once these habits click, the shell becomes a dependable orchestration layer for AI engineering work. Keep the Glossary handy when you forget a flag or command name.</p>
<script src="../www/shell-dock/shell-dock.js" defer=""></script>
<script src="../www/shell-challenges/shell-challenges.js" defer=""></script>
<script src="../www/shell-challenges/shell-env-filter.js" defer=""></script>


</section>

 ]]></description>
  <category>Tutorial</category>
  <category>Shell</category>
  <category>CLI</category>
  <category>DevOps</category>
  <category>AI Engineering</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/22-shell-for-ai-engineers.html</guid>
  <pubDate>Sat, 20 Jun 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/22-shell-for-ai-engineers/intro-img.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>UK AI News Crawler</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/08-news-crawler.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/08-news-crawler.jpg" alt="A friendly robot reading an AI newspaper at a cozy desk"></p>
</div>
<section id="what-it-is" class="level2">
<h2 class="anchored" data-anchor-id="what-it-is">What It Is</h2>
<p>A side project that crawls, summarises and classifies UK-focused AI/ML news articles, and lets you ask questions about them via a RAG chat interface.</p>
<section id="what-it-does" class="level3">
<h3 class="anchored" data-anchor-id="what-it-does">What It Does</h3>
<ul>
<li><strong>Weekly crawl</strong>: A GitHub Actions cron job fires every Monday, searching DuckDuckGo for UK AI keywords and scraping the results.</li>
<li><strong>AI summarisation</strong>: Each article is summarised and sentiment-classified using Vertex AI (Gemini 2.5 Flash).</li>
<li><strong>Vector search</strong>: Articles are embedded with <code>text-embedding-005</code> (768 dims) and stored in Neon PostgreSQL with pgvector.</li>
<li><strong>RAG chat</strong>: Ask a question, the top 5 most relevant articles are retrieved and Gemini generates an answer grounded in the sources.</li>
<li><strong>Admin via GitHub OAuth</strong>: The repo owner can delete articles and trigger reclassification from the UI.</li>
</ul>
</section>
<section id="live-demo" class="level3">
<h3 class="anchored" data-anchor-id="live-demo">Live Demo</h3>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://news-crawler-ochre.vercel.app" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://news-crawler-ochre.vercel.app" title="UK AI News Crawler">
  </iframe>
</div>
</section>
<section id="architecture-at-a-glance" class="level3">
<h3 class="anchored" data-anchor-id="architecture-at-a-glance">Architecture at a Glance</h3>
<div class="cell" data-layout-align="default">
<div class="cell-output-display">
<div>
<p></p><figure class="figure"><p></p>
<div>
<pre class="mermaid mermaid-js">flowchart TD
    A["GitHub Actions (weekly cron)"] --&gt; B["DuckDuckGo Search"]
    B --&gt; C["Scrape &amp; Filter"]
    C --&gt; D["Vertex AI Summarise &amp; Embed"]
    D --&gt; E[("Neon Postgres + pgvector")]
    F["User Question"] --&gt; G["Vector Similarity Search"]
    E --&gt; G
    G --&gt; H["Gemini RAG Answer"]
    H --&gt; I["Streamed Response"]
    E --&gt; J["Article Listings"]
</pre>
</div>
<p></p><figcaption> High-level data flow</figcaption> </figure><p></p>
</div>
</div>
</div>
</section>
<section id="tech-stack" class="level3">
<h3 class="anchored" data-anchor-id="tech-stack">Tech Stack</h3>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Layer</th>
<th>Choice</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Frontend</td>
<td>Next.js 14, React 18, Tailwind, shadcn/ui</td>
</tr>
<tr class="even">
<td>Backend</td>
<td>FastAPI (Python), Mangum ASGI adapter</td>
</tr>
<tr class="odd">
<td>Database</td>
<td>Neon PostgreSQL + pgvector</td>
</tr>
<tr class="even">
<td>AI</td>
<td>Vertex AI: Gemini 2.5 Flash, text-embedding-005</td>
</tr>
<tr class="odd">
<td>Auth</td>
<td>NextAuth.js + GitHub OAuth</td>
</tr>
<tr class="even">
<td>Hosting</td>
<td>Vercel (two projects: frontend &amp; backend)</td>
</tr>
<tr class="odd">
<td>CI/CD</td>
<td>GitHub Actions (weekly crawl) + Vercel auto-deploy</td>
</tr>
</tbody>
</table>


</section>
</section>

 ]]></description>
  <category>til</category>
  <category>web-scraping</category>
  <category>ai</category>
  <category>rag</category>
  <category>chat</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/08-news-crawler.html</guid>
  <pubDate>Sat, 07 Mar 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/08-news-crawler.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>From pg to Neon Serverless</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/07-pg-to-neon-serverless.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/07-pg-to-neon-serverless.jpg" alt="Cozy desk with database icon on monitor, coffee and notebook"></p>
</div>
<section id="why-switch-from-pg" class="level2">
<h2 class="anchored" data-anchor-id="why-switch-from-pg">Why switch from pg?</h2>
<p>When you run RAG (or any Postgres) on Vercel, the standard <code>pg</code> driver is a bad fit. It expects long-lived connections and a pool. In serverless, each invocation is ephemeral—pools don’t help, and TCP handshakes to Postgres add latency on every cold start.</p>
<p><strong>Neon’s serverless driver</strong> (<code>@neondatabase/serverless</code>) uses HTTP fetch instead of persistent sockets. No pooling headaches, fewer cold-start penalties, and a <code>Pool</code> API compatible with <code>pg</code> so you can swap implementations without rewriting your queries.</p>
<section id="benefits" class="level3">
<h3 class="anchored" data-anchor-id="benefits">Benefits</h3>
<ul>
<li><strong>HTTP over WebSockets</strong> — <code>poolQueryViaFetch = true</code> uses Neon’s HTTP query API instead of persistent connections. Matches the request→response model of serverless.</li>
<li><strong>Same API as pg</strong> — Use <code>Pool</code> and standard SQL. Drop-in swap when the URL is Neon.</li>
<li><strong>Local dev unchanged</strong> — Keep <code>pg</code> for Docker Postgres; only use the Neon driver when the connection string points at Neon.</li>
<li><strong>pgvector native</strong> — Neon supports pgvector out of the box for cosine similarity search over embeddings.</li>
</ul>
</section>
</section>
<section id="implementation-snippet" class="level2">
<h2 class="anchored" data-anchor-id="implementation-snippet">Implementation snippet</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode typescript code-with-copy"><code class="sourceCode typescript"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> { Pool <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> NeonPool<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span> neonConfig } <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"@neondatabase/serverless"</span></span>
<span id="cb1-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> { Pool <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> PgPool } <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pg"</span></span>
<span id="cb1-3"></span>
<span id="cb1-4"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">const</span> url <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">process</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">env</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">DATABASE_URL</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span></span>
<span id="cb1-5"></span>
<span id="cb1-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">const</span> isNeon <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> url<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">includes</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"neon.tech"</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">||</span> url<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">includes</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"neon.db.ondigitalocean.com"</span>)</span>
<span id="cb1-7"></span>
<span id="cb1-8"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> (isNeon) {</span>
<span id="cb1-9">  neonConfig<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">poolQueryViaFetch</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">true</span></span>
<span id="cb1-10">}</span>
<span id="cb1-11"></span>
<span id="cb1-12"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">const</span> PoolClass <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> isNeon <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">?</span> NeonPool <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> PgPool</span>
<span id="cb1-13"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">const</span> pool <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">new</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">PoolClass</span>({ connectionString<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> url })</span>
<span id="cb1-14"></span>
<span id="cb1-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">// Use as normal — RAG vector search example</span></span>
<span id="cb1-16"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">const</span> { rows } <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> pool<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">query</span>(</span>
<span id="cb1-17">  <span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">`SELECT id, content, 1 - (embedding &lt;=&gt; $1::vector) AS similarity</span></span>
<span id="cb1-18"><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">   FROM document_chunks</span></span>
<span id="cb1-19"><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">   ORDER BY embedding &lt;=&gt; $1::vector</span></span>
<span id="cb1-20"><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">   LIMIT 5`</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb1-21">  [queryEmbedding]</span>
<span id="cb1-22">)</span></code></pre></div></div>
<p>Thin abstraction, production-ready. Point <code>DATABASE_URL</code> at Neon in prod and local Postgres in dev—no other changes.</p>


</section>

 ]]></description>
  <category>til</category>
  <category>snippet</category>
  <category>rag</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/07-pg-to-neon-serverless.html</guid>
  <pubDate>Sat, 21 Feb 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/07-pg-to-neon-serverless.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Unified Chat Service</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/06-unified-chat-service.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/06-unified-chat.jpg" alt="Architectural sketch of a chat service on a notepad"></p>
</div>
<section id="what-it-is" class="level2">
<h2 class="anchored" data-anchor-id="what-it-is">What It Is</h2>
<p>A <strong>unified chat service</strong> — a frontend component library plus central API for in-product AI chat. Consumer apps all plug into the same backend, which handles RAG via Neon Postgres + pgvector and chat completions via GCP Vertex AI.</p>
<div class="cell" data-layout-align="default">
<div class="cell-output-display">
<div>
<p></p><figure class="figure"><p></p>
<div>
<pre class="mermaid mermaid-js">flowchart LR
    subgraph Consumers["Consumer Apps"]
        direction TB
        P[Pomodoro]
        U[Unit Converter]
        T[Tip Calculator]
    end

    subgraph Service["Global Chat Service"]
        Chat["POST /api/chat"]
    end

    subgraph External["External Services"]
        direction TB
        Neon["Neon Postgres + pgvector"]
        Vertex["Vertex AI Gemini"]
    end

    P &amp; U &amp; T --&gt; Chat
    Chat --&gt;|RAG + vector search| Neon
    Chat --&gt;|completions + embeddings| Vertex
</pre>
</div>
<p></p><figcaption> High-level architecture</figcaption> </figure><p></p>
</div>
</div>
</div>
<p>Consumers call <code>POST /api/chat</code> with a <code>product_id</code>; the service fetches relevant docs, streams responses, and auto-registers events.</p>
</section>
<section id="architecture" class="level2">
<h2 class="anchored" data-anchor-id="architecture">Architecture</h2>
<p><strong>Services:</strong> Neon Postgres (pgvector) for vector search; GCP Vertex AI (text-embedding-005, Gemini 2.5 Flash) for embeddings and completions; NextAuth with GitHub for doc management at <code>/library</code>. Hosted on Vercel.</p>
<p><strong>RAG flow:</strong> Consumer sends <code>POST /api/chat</code> with <code>product_id</code>, <code>user</code>, and <code>messages</code>. The service extracts search terms from the last message, embeds the query, runs cosine similarity search on <code>document_chunks</code>, builds a system prompt with the top 5 chunks, then streams the completion.</p>
<p><strong>Data model:</strong> Projects (one per consumer app) → documents (uploaded via <code>/library</code>) → document_chunks (768-dim embeddings). CORS restricted to <code>ALLOWED_ORIGINS</code>.</p>
</section>
<section id="try-the-apps" class="level2">
<h2 class="anchored" data-anchor-id="try-the-apps">Try the Apps</h2>
<p>The consumer apps below demonstrate distinct styling and RAG tuned to each product’s documentation.</p>
<section id="chat-service-demo" class="level3">
<h3 class="anchored" data-anchor-id="chat-service-demo">Chat Service Demo</h3>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://unified-chat-service-unified-chat-s.vercel.app" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://unified-chat-service-unified-chat-s.vercel.app" title="Unified Chat Service Demo">
  </iframe>
</div>
</section>
<section id="unit-converter" class="level3">
<h3 class="anchored" data-anchor-id="unit-converter">Unit Converter</h3>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://unit-converter-chi-ten.vercel.app" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://unit-converter-chi-ten.vercel.app" title="Unit Converter Demo">
  </iframe>
</div>
</section>
<section id="minimal-pomodoro" class="level3">
<h3 class="anchored" data-anchor-id="minimal-pomodoro">Minimal Pomodoro</h3>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://minimal-pomodoro-one.vercel.app" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://minimal-pomodoro-one.vercel.app" title="Minimal Pomodoro Demo">
  </iframe>
</div>
</section>
<section id="tip-calculator" class="level3">
<h3 class="anchored" data-anchor-id="tip-calculator">Tip Calculator</h3>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://tip-calculator-ruby-eight.vercel.app" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://tip-calculator-ruby-eight.vercel.app" title="Tip Calculator Demo">
  </iframe>
</div>
<hr>
<p><em>Central service + RAG on Neon + Vertex AI. Consumers stay small; the heavy lifting happens once.</em></p>


</section>
</section>

 ]]></description>
  <category>experiment</category>
  <category>til</category>
  <category>rag</category>
  <category>ai</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/06-unified-chat-service.html</guid>
  <pubDate>Sat, 14 Feb 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/06-unified-chat.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Rubber Duck Committee</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/05-rubber-duck-committee.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/05-rubber-duck-committee.jpg" alt="A committee of rubber ducks in discussion"></p>
</div>
<section id="the-inspiration" class="level2">
<h2 class="anchored" data-anchor-id="the-inspiration">The Inspiration</h2>
<p>PewDiePie recently shared his experiments running <a href="https://www.pcgamer.com/software/ai/pewdiepie-creates-an-ai-council-appoints-himself-supreme-leader-and-wipes-out-members-who-underperform-only-for-his-councillors-to-work-against-him/">multiple AI models locally on a GPU rig</a>, having them vote on decisions as a “council”. The interesting bit? He claims collusive behaviour emerged — models started voting strategically to help each other survive.</p>
<p>No open-sourced logs or data that I could find, so take it with a pinch of salt. But the <em>idea</em> stuck with me: what if you could get multiple AI perspectives to analyse a problem independently, then vote on the best solution?</p>
</section>
<section id="can-you-do-this-on-the-cheap" class="level2">
<h2 class="anchored" data-anchor-id="can-you-do-this-on-the-cheap">Can You Do This on the Cheap?</h2>
<p>Turns out, yes. You don’t need a $20,000 setup.</p>
<p>The trick is using <strong>customised system prompts</strong> to create distinct personas, then <strong>tool calls</strong> to orchestrate the council interactions. Each “duck” gets:</p>
<ul>
<li>A unique personality and thinking style (methodical professor, creative brainstormer, pragmatic engineer)</li>
<li>Configurable tools and instructions</li>
<li>Independent analysis of the problem</li>
<li>A vote on which solution they prefer</li>
</ul>
<p>The voting happens via structured outputs — each model returns its reasoning and vote in a predictable format, making it easy to tally.</p>
</section>
<section id="try-it" class="level2">
<h2 class="anchored" data-anchor-id="try-it">Try It</h2>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://rubber-duck-committee.vercel.app" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://rubber-duck-committee.vercel.app" title="Rubber Duck Committee Demo">
  </iframe>
</div>

</section>
<section id="the-duck-personas" class="level2">
<h2 class="anchored" data-anchor-id="the-duck-personas">The Duck Personas</h2>
<p>Three ducks, three perspectives:</p>
<ul>
<li><strong>Professor Quacksworth</strong> — methodical, detail-oriented, breaks problems into pieces</li>
<li><strong>Ducky McBrainstorm</strong> — imaginative, thinks outside the box, unconventional approaches<br>
</li>
<li><strong>Captain Waddles</strong> — practical, focused on shipping working solutions</li>
</ul>
<p>They analyse independently (avoiding groupthink), show their reasoning transparently, then vote democratically on the best solution.</p>
</section>
<section id="why-rubber-ducks" class="level2">
<h2 class="anchored" data-anchor-id="why-rubber-ducks">Why Rubber Ducks?</h2>
<p><a href="https://en.wikipedia.org/wiki/Rubber_duck_debugging">Rubber duck debugging</a> is a classic technique — explain your problem to a rubber duck and you’ll often solve it yourself. This takes that idea and gives you a whole committee of ducks with different viewpoints.</p>
<p>Sometimes you need a professor. Sometimes you need a pragmatist. Why not both?</p>
<hr>
<p><em>Inspired by PewDiePie’s AI council experiments and the timeless art of rubber duck debugging.</em></p>


</section>

 ]]></description>
  <category>idea</category>
  <category>ai</category>
  <category>experiment</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/05-rubber-duck-committee.html</guid>
  <pubDate>Fri, 30 Jan 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/05-rubber-duck-committee.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Resend: Email Without SMTP Pain</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/04-resend-email-service.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/04-resend-email-service.jpg" alt="Cozy desk with an envelope, coffee cup and laptop"></p>
</div>
<section id="sending-emails-without-the-fuss" class="level2">
<h2 class="anchored" data-anchor-id="sending-emails-without-the-fuss">Sending Emails Without the Fuss</h2>
<p>Recently needed to add email notifications to a web form—admin alerts and user confirmations. The thought of configuring an SMTP server made me wince. Then I found <a href="https://resend.com">Resend</a>.</p>
</section>
<section id="what-is-it" class="level2">
<h2 class="anchored" data-anchor-id="what-is-it">What Is It?</h2>
<p>Resend is an email API for developers. Instead of wrestling with SMTP servers, mail transfer agents, and deliverability nightmares, you just… call an API.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> resend</span>
<span id="cb1-2"></span>
<span id="cb1-3">resend.api_key <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"your-api-key"</span></span>
<span id="cb1-4"></span>
<span id="cb1-5">resend.Emails.send({</span>
<span id="cb1-6">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"from"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"hello@yourdomain.com"</span>,</span>
<span id="cb1-7">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"to"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"recipient@example.com"</span>,</span>
<span id="cb1-8">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"subject"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Hello!"</span>,</span>
<span id="cb1-9">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"html"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;p&gt;Your email body here&lt;/p&gt;"</span></span>
<span id="cb1-10">})</span></code></pre></div></div>
<p>That’s genuinely it.</p>
</section>
<section id="the-good-bits" class="level2">
<h2 class="anchored" data-anchor-id="the-good-bits">The Good Bits</h2>
<ul>
<li><strong>No SMTP configuration</strong> — No ports, no TLS certificates, no authentication headaches</li>
<li><strong>Attachments are easy</strong> — Pass base64 content or file paths</li>
<li><strong>HTML emails just work</strong> — Style them however you like</li>
<li><strong>Generous free tier</strong> — 3,000 emails/month at time of writing</li>
<li><strong>Great dashboard</strong> — See delivery status, bounces, opens</li>
<li><strong>Vercel-friendly</strong> — Works beautifully with serverless functions</li>
</ul>
</section>
<section id="the-trade-offs" class="level2">
<h2 class="anchored" data-anchor-id="the-trade-offs">The Trade-offs</h2>
<ul>
<li><strong>Vendor lock-in</strong> — You’re dependent on their service</li>
<li><strong>Domain verification required</strong> — Need to add DNS records for sending</li>
<li><strong>Not self-hosted</strong> — If that matters to your use case</li>
<li><strong>Rate limits on free tier</strong> — 100 emails/day limit</li>
</ul>
</section>
<section id="my-use-case" class="level2">
<h2 class="anchored" data-anchor-id="my-use-case">My Use Case</h2>
<p>For a membership form, I set up two email types:</p>
<ol type="1">
<li><strong>Admin notification</strong> — HTML email with form data + CSV attachment</li>
<li><strong>Applicant confirmation</strong> — Friendly receipt with next steps</li>
</ol>
<p>Both fire when the form submits. If emails fail, the form submission still succeeds (data saved to DB), so nothing gets lost.</p>
</section>
<section id="quick-tips" class="level2">
<h2 class="anchored" data-anchor-id="quick-tips">Quick Tips</h2>
<ul>
<li>Store <code>RESEND_API_KEY</code> in environment variables</li>
<li>Always handle email failures gracefully—don’t break your main flow</li>
<li>The <code>from</code> address domain must be verified in your Resend dashboard</li>
<li>CSV attachments need content as a list of bytes: <code>list(csv_content.encode())</code></li>
</ul>
</section>
<section id="verdict" class="level2">
<h2 class="anchored" data-anchor-id="verdict">Verdict</h2>
<p>For hobby projects, internal tools, or MVPs where you just need email to work, Resend removes a lot of friction. Would reach for it again.</p>


</section>

 ]]></description>
  <category>til</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/04-resend-email-service.html</guid>
  <pubDate>Fri, 16 Jan 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/04-resend-email-service.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Daemon AI Assistants</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/03-daemon-ai-assistant.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/03-daemon-ai-assistant.jpg" alt="A cozy desk with friendly AI daemons floating nearby"></p>
</div>
<p>I’ve been fascinated by <a href="https://maggieappleton.com/lm-sketchbook">Maggie Appleton’s Language Model Sketchbook</a> for a while now. Her “Daemons” concept stuck with me — so I decided to build it.</p>
<section id="the-concept" class="level2">
<h2 class="anchored" data-anchor-id="the-concept">The Concept</h2>
<p>What if AI assistants weren’t chat windows, but <em>daemons</em> — little personalities that hover alongside your text, each with their own perspective?</p>
<ul>
<li><strong>Devil’s Advocate</strong> — questions your assertions, asks for evidence</li>
<li><strong>Grammar Enthusiast</strong> — spots style issues (purple, naturally)</li>
<li><strong>Clarity Coach</strong> — nudges toward clearer expression</li>
</ul>
<p>The daemons highlight relevant sections and offer contextual suggestions as you write. It’s a fundamentally different UX from the “type a prompt, wait for response” pattern we’ve all gotten used to.</p>
</section>
<section id="try-it" class="level2">
<h2 class="anchored" data-anchor-id="try-it">Try It</h2>
<p>Here’s the working prototype, click the daemon swatches to the right and they perform tasks, highlighting relevant blocks of text:</p>
<p style="text-align: center; margin-top: 0.5rem;">
  <a href="https://daemon-ai-app.vercel.app" target="_blank" rel="noopener noreferrer">Open in new window ↗</a>
</p>
<div class="notepad-embed">
  <iframe src="https://daemon-ai-app.vercel.app" title="Daemon AI Assistant Demo">
  </iframe>
</div>

</section>
<section id="why-i-built-this" class="level2">
<h2 class="anchored" data-anchor-id="why-i-built-this">Why I Built This</h2>
<p>I wanted to explore <em>alternative interfaces</em> for language models. The chat paradigm is powerful but it’s not the only way. Maggie’s original sketches proposed all sorts of ideas — daemons, word-level annotations, ambient suggestions.</p>
<p>Building this brought one of those ideas to life. It’s rough around the edges, but that’s the point of the notepad — showing work in progress.</p>
<hr>
<p><em>Credit to <a href="https://maggieappleton.com">Maggie Appleton</a> for the original “Daemons” concept.</em></p>


</section>

 ]]></description>
  <category>idea</category>
  <category>ai</category>
  <category>experiment</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/03-daemon-ai-assistant.html</guid>
  <pubDate>Fri, 16 Jan 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/03-daemon-ai-assistant.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Pruning Stale Local Branches</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/01-git-prune.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/01-git-prune.jpg" alt="Git branches being pruned"></p>
</div>
<p>After a while, local branches pile up—especially ones tracking remotes that have since been deleted. Here’s how to clean house.</p>
<section id="prune-and-find-stale-branches" class="level2">
<h2 class="anchored" data-anchor-id="prune-and-find-stale-branches">Prune and find stale branches</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Prune stale remote-tracking references</span></span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">git</span> fetch <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--prune</span></span>
<span id="cb1-3"></span>
<span id="cb1-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># List local branches with gone remotes</span></span>
<span id="cb1-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">git</span> branch <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-vv</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">': gone]'</span></span></code></pre></div></div>
</section>
<section id="get-just-the-branch-names" class="level2">
<h2 class="anchored" data-anchor-id="get-just-the-branch-names">Get just the branch names</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">git</span> branch <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-vv</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">': gone]'</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">awk</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'{print $1}'</span></span></code></pre></div></div>
</section>
<section id="delete-them-all-at-once" class="level2">
<h2 class="anchored" data-anchor-id="delete-them-all-at-once">Delete them all at once</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">git</span> branch <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-vv</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grep</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">': gone]'</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">awk</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'{print $1}'</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xargs</span> git branch <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-d</span></span></code></pre></div></div>
<p>Use <code>-D</code> instead of <code>-d</code> to force-delete branches with unmerged changes.</p>


</section>

 ]]></description>
  <category>til</category>
  <category>git</category>
  <category>snippet</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/01-git-prune.html</guid>
  <pubDate>Sun, 11 Jan 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/01-git-prune.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Cursor’s Browser Tools are a Game Changer</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/02-cursor-browser-tools.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/02-cursor-browser-tools.jpg" alt="A cozy desk with a browser window"></p>
</div>
<p>I’m genuinely blown away by Cursor’s browser integration.</p>
<section id="what-changed" class="level2">
<h2 class="anchored" data-anchor-id="what-changed">What Changed</h2>
<p>Previously, debugging web dev with an AI assistant meant:</p>
<ul>
<li>Manual screenshots → paste into chat</li>
<li>Copy-pasting chunks of DOM</li>
<li>Describing what you’re seeing</li>
<li>Back-and-forth “can you scroll down?”</li>
</ul>
<p>Now? The agent just… <em>looks at the page</em>.</p>
</section>
<section id="the-toolkit" class="level2">
<h2 class="anchored" data-anchor-id="the-toolkit">The Toolkit</h2>
<p>What I’ve been using:</p>
<ul>
<li><strong><code>browser_navigate</code></strong> — point it at a URL</li>
<li><strong><code>browser_snapshot</code></strong> — accessibility tree, better than screenshots for reasoning</li>
<li><strong><code>browser_take_screenshot</code></strong> — visual capture when needed</li>
<li><strong><code>browser_click</code> / <code>browser_type</code></strong> — interact with elements directly</li>
<li><strong><code>browser_console_messages</code></strong> — inspect console logs without opening devtools</li>
<li><strong><code>browser_network_requests</code></strong> — see API calls, failed fetches, etc.</li>
<li><strong><code>browser_resize</code></strong> — test responsive layouts at different widths</li>
<li><strong><code>browser_wait_for</code></strong> — wait for text to appear/disappear or just pause</li>
</ul>
<p>The console and network tools seem particularly powerful for debugging—no more “check your network tab” back-and-forth.</p>
</section>
<section id="why-it-matters" class="level2">
<h2 class="anchored" data-anchor-id="why-it-matters">Why It Matters</h2>
<p>This feels like a genuine workflow shift. The agent can now:</p>
<ol type="1">
<li>Navigate to your running dev server</li>
<li>Visually verify changes</li>
<li>Inspect the DOM structure</li>
<li>Catch console errors</li>
<li>Iterate on fixes</li>
</ol>
<hr>
<p><em>Now I just need it to make me coffee while the tests run.</em></p>


</section>

 ]]></description>
  <category>til</category>
  <category>cursor</category>
  <category>web-dev</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/02-cursor-browser-tools.html</guid>
  <pubDate>Sun, 11 Jan 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/02-cursor-browser-tools.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Welcome to the Notepad</title>
  <link>https://thedatasavvycorner.netlify.app/notepad/00-hello-notepad.html</link>
  <description><![CDATA[ 





<div class="notepad-hero">
<p><img src="https://thedatasavvycorner.netlify.app/www/notepad/00-hello-notepad.jpg" alt="A cozy notepad sketch"></p>
</div>
<p>In the age of the LLM revolution, I’m currently unsure as to how useful long form blogging will be.</p>
<p>While I weigh things up, I will go with a demonstrably scruffy and minimal approach to recording ideas, experiments and snippets for things that I’ve found interesting, tricky or non-trivial.</p>
<p>Some of these ideas may be re-touched with time or grow into fully fledged articles, while others may remain as limited records of stuff done.</p>



 ]]></description>
  <category>meta</category>
  <guid>https://thedatasavvycorner.netlify.app/notepad/00-hello-notepad.html</guid>
  <pubDate>Thu, 08 Jan 2026 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/notepad/00-hello-notepad.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Automated CodeCov reports with GitHub Actions</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/21-codecov-setup.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/21-codecov-setup/intro-img.jpg" alt="Futuristic cyberpunk shield" style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p><a href="https://about.codecov.io/">CodeCov</a> is an online service that allows you to generate free coverage reports for your repositories. Coverage reports can be created as part of a CI workflow. Uploading these results to the CodeCov service allows centralised tracking of code coverage trends, various interactive visuals and the all important coverage shield(!)</p>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Python developers who need to quickly get coverage reporting set up so they can get on with the rest of their work.</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p>I have included screengrabs of things where possible. Please note that GitHub or CodeCov may change their interfaces at their discretion.</p>
</div>
</div>
</section>
<section id="requirements" class="level3">
<h3 class="anchored" data-anchor-id="requirements">Requirements</h3>
<ul>
<li>Install the following requirements, along with any other dependencies of the code you wish to report on:</li>
</ul>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>requirements.txt</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" data-filename="requirements.txt" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb1-1">coverage</span>
<span id="cb1-2">pytest</span></code></pre></div></div>
</div>
<ul>
<li>A GitHub account</li>
<li>A repo with some code</li>
</ul>
</section>
</section>
<section id="method" class="level2">
<h2 class="anchored" data-anchor-id="method">Method</h2>
<ol type="1">
<li>Login to the <a href="https://about.codecov.io/" target="_blank">CodeCov Service</a> by accessing the button at the top right of the screen. I suggest selecting the GitHub option for login. You will be asked to authenticate with GitHub if you do so.</li>
<li>If you don’t have one, you will be prompted about <a href="https://about.codecov.io/pricing/" target="_blank">the type of account</a> that you need. I would suggest for most users <strong>and</strong> GitHub organisation members, a personal free account is fine.</li>
<li>Search for the repo that you would like coverage reports for. If the repo does not appear, try clicking refresh. If the repo belongs to an organisation, click on the drop-down menu to the top-left of the screen, highlighted in red in the image below.</li>
</ol>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/21-codecov-setup/01.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="CodeCov UI Repo Owner drop down. Click to expand."><img src="https://thedatasavvycorner.netlify.app/www/21-codecov-setup/01.png" class="shaded_box img-fluid figure-img" alt="CodeCov UI Repo Owner drop down. Click to expand."></a></p>
<figcaption>CodeCov UI Repo Owner drop down. Click to expand.</figcaption>
</figure>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-2-contents" aria-controls="callout-2" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Org repo not appearing? (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-2" class="callout-2-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>Some organisations may not allow CodeCov global access to its repositories. If this is the case, you will need to ask an organisation owner to check a box allowing CodeCov read &amp; write access to the repo. This is done from the organisation’s homepage &gt; Settings &gt; Third Party Access &gt; GitHub Apps &gt; Codecov &gt; Configure &gt; Repository access &gt; Only select repositories &gt; Select repositories dropdown.</p>
</div>
</div>
</div>
<ol start="4" type="1">
<li>Once found, click on the blue configure button. There are various options on the next screen for selecting your CI provider and coverage source. Ignore them and locate the <code>CODECOV_TOKEN</code> value. Copy it to your clipboard.</li>
</ol>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-3-contents" aria-controls="callout-3" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Changes with <code>CODECOV_TOKEN</code> (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-3" class="callout-3-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>You may notice that CodeCov have included a banner at the top of their login screen - “You must now upload using a token.” How lovely of them. It used to be the case that open repos did not require a token. Rather quietly sometime this year they <a href="https://docs.codecov.com/docs/adding-the-codecov-token" target="_blank">changed this behaviour</a> without making it clear in their docs. This was incredibly helpful when going through this setup live with a cohort of graduate data scientists.</p>
<p>Anyway there’s a banner now, so I’ll forgive them.</p>
</div>
</div>
</div>
<ol start="5" type="1">
<li>Keep your CodeCov page open in a tab. Now go to your repo page in a new tab. Settings &gt; Security &gt; Secrets and variables &gt; Actions</li>
</ol>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/21-codecov-setup/02.png" class="lightbox" data-gallery="quarto-lightbox-gallery-2" title="GitHub UI location of Actions Secrets &amp; Variables. Click to expand."><img src="https://thedatasavvycorner.netlify.app/www/21-codecov-setup/02.png" class="shaded_box img-fluid figure-img" alt="GitHub UI location of Actions Secrets &amp; Variables. Click to expand."></a></p>
<figcaption>GitHub UI location of Actions Secrets &amp; Variables. Click to expand.</figcaption>
</figure>
</div>
<ol start="6" type="1">
<li>On the next page, under Repository secrets, click “New repository secret”, ensure <code>Name</code> is equal to <code>CODECOV_TOKEN</code>, paste your clipboard credential into the <code>Secret</code> field then click “Add secret”.</li>
<li>Add a workflow file to your repository under <code>.github/workflows</code>.</li>
<li>Save the following code snippet into the file:</li>
</ol>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>.github/workflows/codecov-upload.yaml</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-2" data-filename=".github/workflows/codecov-upload.yaml" style="background: #f1f3f5;"><pre class="sourceCode abc code-annotation-code code-with-copy code-annotated"><code class="sourceCode abc"><span id="annotated-cell-2-1"></span>
<span id="annotated-cell-2-2">name<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> MacOS Full Test Suite and Coverage</span>
<span id="annotated-cell-2-3"></span>
<span id="annotated-cell-2-4">on<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="1">1</button><span id="annotated-cell-2-5" class="code-annotation-target">  push<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="annotated-cell-2-6">  pull_request<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="annotated-cell-2-7">    branches<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span>"main"<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">]</span></span>
<span id="annotated-cell-2-8"></span>
<span id="annotated-cell-2-9">jobs<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="annotated-cell-2-10">  build<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="annotated-cell-2-11">    name<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Full &amp; Coverage</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="2">2</button><span id="annotated-cell-2-12" class="code-annotation-target">    runs-on<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ubuntu-latest</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="3">3</button><span id="annotated-cell-2-13" class="code-annotation-target">    timeout-minutes<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> 15</span>
<span id="annotated-cell-2-14">    strategy<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="annotated-cell-2-15">      fail-fast<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> false</span>
<span id="annotated-cell-2-16">    steps<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="4">4</button><span id="annotated-cell-2-17" class="code-annotation-target">    - uses<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683</span>
<span id="annotated-cell-2-18">    - name<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Set up Python 12</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="5">5</button><span id="annotated-cell-2-19" class="code-annotation-target">      uses<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b</span>
<span id="annotated-cell-2-20">      with<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="6">6</button><span id="annotated-cell-2-21" class="code-annotation-target">        cache<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> 'pip'</span>
<span id="annotated-cell-2-22">        python-version<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> 3.12</span>
<span id="annotated-cell-2-23">    - name<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Install dependencies</span>
<span id="annotated-cell-2-24">      run<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">|</span></span>
<span id="annotated-cell-2-25">        python -m pip install --upgrade pip</span>
<span id="annotated-cell-2-26">        pip install -r requirements.txt</span>
<span id="annotated-cell-2-27">    - name<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Generate Coverage Report</span>
<span id="annotated-cell-2-28">      run<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">|</span></span>
<span id="annotated-cell-2-29">        coverage run -m pytest</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="7">7</button><span id="annotated-cell-2-30" class="code-annotation-target">        coverage report</span>
<span id="annotated-cell-2-31">        coverage xml</span>
<span id="annotated-cell-2-32">    - name<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Upload coverage report to CodeCov</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="8">8</button><span id="annotated-cell-2-33" class="code-annotation-target">      uses<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> codecov/codecov-action@015f24e6818733317a2da2edd6290ab26238649a</span>
<span id="annotated-cell-2-34">      with<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="annotated-cell-2-35">        file<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./coverage.xml</span>
<span id="annotated-cell-2-36">        flags<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> unittests</span>
<span id="annotated-cell-2-37">        verbose<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> true</span>
<span id="annotated-cell-2-38">        token<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> $<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">{{</span>  secrets.CODECOV_TOKEN  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">}}</span></span>
<span id="annotated-cell-2-39">        fail_ci_if_error<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> true</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-2" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="5,6,7" data-code-annotation="1">Adjust to suit your needs. Note that you can select different branch coverage reports in CodeCov by selecting “Branch Context”.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="12" data-code-annotation="2">Ubuntu builds are fast, but you may wish to select a different os to match the supported platform of your package / codebase.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="13" data-code-annotation="3">Not necessary, but a good idea for private repos, where organisations pay for build time.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="17" data-code-annotation="4">SHA for v4.2.2</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="5">5</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="19" data-code-annotation="5">SHA for v5.3.0</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="6">6</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="21" data-code-annotation="6">Not neccesary, but speeds up pip dependencies by using a cache.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="7">7</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="30" data-code-annotation="7">Not necessary, but a good sanity check - did we actually create a coverage report? Did it contain any coverage data? This command will print the coverage report so that you can inspect it in your Action logs.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="8">8</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="33" data-code-annotation="8">SHA for v5.0.7</span>
</dd>
</dl>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-4-contents" aria-controls="callout-4" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Why reference SHA instead of versions? (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-4" class="callout-4-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>It’s <a href="../blogs/14-gh-actions-security.html" target="_blank">more secure</a>. Ensure that you keep these versions updated to mitigate the risk of malicious attacks.</p>
</div>
</div>
</div>
<ol start="9" type="1">
<li>Commit and push the new workflow file to your remote. Go and watch the Action to ensure it prints your coverage report in the logs and uploads to the CodeCov service without complaint.</li>
<li>Go back to your tab with the CodeCov site. You may need to refresh or click the “Resync” button, but your newly configured repo should now display coverage results. Click on Configuration &gt; Badges &amp; Graphs to get the links to your shiny new coverage shield.</li>
</ol>
<p id="fin">
<i>fin!</i>
</p>


</section>

 ]]></description>
  <category>How-to</category>
  <category>CodeCov</category>
  <category>Coverage</category>
  <category>CI:CD</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/21-codecov-setup.html</guid>
  <pubDate>Fri, 29 Nov 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/21-codecov-setup/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Lazy Mocking</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/20-lazy-mocking.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/20-lazy-mocking/intro-img.jpg" alt="The marionette Taking a Nap in a cyberpunk setting." style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<blockquote class="blockquote">
<p>“Time to live<br>
Time to lie<br>
Time to laugh<br>
Time to die<br>
Take it easy, baby<br>
Take it as it comes” Take It As It Comes, The Doors.</p>
</blockquote>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>A simple approach to sharing a fixture across multiple tests where mocking is a requirement. After comparing implementations with <code>pytest</code>’s <code>monkeypatch</code> and <code>mockito</code>, <code>unittest.patch</code> was selected because it is straightforward and succinct. The code in this article is <a href="https://gist.github.com/r-leyshon/817e19438380eb9df638dfb1cd4c242e" target="_blank">available in this gist</a> for those pushed for time.</p>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Experienced python Developers, test engineers &amp; any intersection of the two. This tutorial is not for those new to mocking. Please refer to <a href="../blogs/15-pytest-mocking.html">Mocking With Pytest in Plain English</a> for a more comprehensive introduction to that. This blog is part of a series called <a href="../blogs/index.html#category=pytest-in-plain-english">pytest in plain English</a>.</p>
</section>
<section id="requirements" class="level3">
<h3 class="anchored" data-anchor-id="requirements">Requirements</h3>
<p><code>pip install pytest</code></p>
</section>
</section>
<section id="some-source-code" class="level2">
<h2 class="anchored" data-anchor-id="some-source-code">Some Source Code</h2>
<p>This little function would cause a problem when writing your test suite:</p>
<div id="c3c02f5b" class="cell" data-execution_count="1">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> datetime <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> datetime</span>
<span id="cb1-2"></span>
<span id="cb1-3"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> get_poem_line_for_day():</span>
<span id="cb1-4">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Returns the line of the poem based on the current day of the week."""</span></span>
<span id="cb1-5">    day_of_week <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> datetime.today().strftime(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'%A'</span>)</span>
<span id="cb1-6">    POEM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="cb1-7">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Monday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Monday's child is fair of face"</span>,</span>
<span id="cb1-8">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Tuesday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Tuesday's child is full of grace"</span>,</span>
<span id="cb1-9">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday's child is full of woe"</span>,</span>
<span id="cb1-10">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Thursday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Thursday's child has far to go"</span>,</span>
<span id="cb1-11">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday's child is loving and giving"</span>,</span>
<span id="cb1-12">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Saturday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Saturday's child works hard for his living"</span>,</span>
<span id="cb1-13">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Sunday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"And the child that is born on the Sabbath day is bonny and blithe, and good and gay"</span>,</span>
<span id="cb1-14">    }</span>
<span id="cb1-15">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> POEM.get(day_of_week, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Unknown day"</span>)</span>
<span id="cb1-16"></span>
<span id="cb1-17">get_poem_line_for_day()</span></code></pre></div></div>
<div class="cell-output cell-output-display" data-execution_count="1">
<pre><code>"Monday's child is fair of face"</code></pre>
</div>
</div>
<div class="callout callout-style-simple callout-none no-icon callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-1-contents" aria-controls="callout-1" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon no-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">None</span>Why would this be hard to test without mocking? (Click to reveal)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-1" class="callout-1-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<ul>
<li>The function will return different strings depending on the day the test is run.</li>
<li>Without mocking <code>get_poem_line_for_day</code>, you would have to update hard-coded test predicates to match the current day of the week. Nope.</li>
<li>In CI, avoiding mocking would likely result in setting a variable equal to the current day of the week and basing your test predicates off of that. Nope.</li>
<li>Let’s instead patch the values…</li>
</ul>
</div>
</div>
</div>
</section>
<section id="lets-test" class="level2">
<h2 class="anchored" data-anchor-id="lets-test">Let’s Test…</h2>
<section id="local-scoped-mock" class="level3">
<h3 class="anchored" data-anchor-id="local-scoped-mock">Local-Scoped Mock</h3>
<p>This is very easy to mock using local-scoped utility functions.</p>
<div id="9a4a5ee5" class="cell" data-execution_count="2">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="annotated-cell-2" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-2-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> mock</span>
<span id="annotated-cell-2-2"></span>
<span id="annotated-cell-2-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="annotated-cell-2-4"></span>
<span id="annotated-cell-2-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> poem</span>
<span id="annotated-cell-2-6"></span>
<span id="annotated-cell-2-7">POEM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="annotated-cell-2-8">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Monday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Monday's child is fair of face"</span>,</span>
<span id="annotated-cell-2-9">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Tuesday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Tuesday's child is full of grace"</span>,</span>
<span id="annotated-cell-2-10">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday's child is full of woe"</span>,</span>
<span id="annotated-cell-2-11">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Thursday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Thursday's child has far to go"</span>,</span>
<span id="annotated-cell-2-12">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday's child is loving and giving"</span>,</span>
<span id="annotated-cell-2-13">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Saturday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Saturday's child works hard for his living"</span>,</span>
<span id="annotated-cell-2-14">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Sunday"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"And the child that is born on the Sabbath day is bonny and blithe, and good and gay"</span>,</span>
<span id="annotated-cell-2-15">    }</span>
<span id="annotated-cell-2-16"></span>
<span id="annotated-cell-2-17"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="1">1</button><span id="annotated-cell-2-18" class="code-annotation-target"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@mock.patch</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"poem.get_poem_line_for_day"</span>)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="2">2</button><span id="annotated-cell-2-19" class="code-annotation-target"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_poem_line_forever_thursday(patched_poem):</span>
<span id="annotated-cell-2-20">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Uses immediate instantiation"""</span></span>
<span id="annotated-cell-2-21"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="3">3</button><span id="annotated-cell-2-22" class="code-annotation-target">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> mock_poem(day, poem<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>POEM):</span>
<span id="annotated-cell-2-23">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> poem[day]</span>
<span id="annotated-cell-2-24"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="4">4</button><span id="annotated-cell-2-25" class="code-annotation-target">    patched_poem.return_value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> mock_poem(day<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Thursday"</span>)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="5">5</button><span id="annotated-cell-2-26" class="code-annotation-target">    result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> poem.get_poem_line_for_day()</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-2" data-target-annotation="6">6</button><span id="annotated-cell-2-27" class="code-annotation-target">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Thursday's child has far to go"</span></span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
<div class="cell-annotation">
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-2" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="18" data-code-annotation="1">Specify the target that we wish to patch.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="19" data-code-annotation="2">Define a name for the patch as <code>patched_poem</code>. We’ll need to refer to this when implementing the patch in the test.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="22" data-code-annotation="3">Define a locally-scoped function that will serve the line of the poem depending on the day that <strong>you ask for</strong>.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="25" data-code-annotation="4">Set the return value of the patch we specified to be equal to the line for a hard-coded day of the week. Note that we could go ahead and make multiple assertions re-using the <code>mock_poem</code> utility.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="5">5</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="26" data-code-annotation="5">Use the System Under Test (SUT). Note that in a real test suite, we would likely target the component of the SUT that we need to control, rather than the entire source code. Eg - target <code>datetime.today</code> rather than <code>get_poem_line_for_day</code>.</span>
</dd>
<dt data-target-cell="annotated-cell-2" data-target-annotation="6">6</dt>
<dd>
<span data-code-cell="annotated-cell-2" data-code-lines="27" data-code-annotation="6">Whatever day the test is executed on, the resulting value will be patched in the way we specified.</span>
</dd>
</dl>
</div>
</div>
</section>
<section id="broken-mock-with-fixture" class="level3">
<h3 class="anchored" data-anchor-id="broken-mock-with-fixture">Broken Mock with Fixture</h3>
<p>It is common to start with a locally-scoped mock and then , as the test suite grows, it would be better to share the mock across multiple tests. You may naively try to use the same <code>mock_poem</code> as a <code>pytest</code> fixture.</p>
<div id="9dfaa6a5" class="cell" data-execution_count="3">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="annotated-cell-3" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-3-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> mock</span>
<span id="annotated-cell-3-2"></span>
<span id="annotated-cell-3-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="annotated-cell-3-4"></span>
<span id="annotated-cell-3-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> poem</span>
<span id="annotated-cell-3-6"></span>
<span id="annotated-cell-3-7"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-3" data-target-annotation="1">1</button><span id="annotated-cell-3-8" class="code-annotation-target"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.fixture</span>(scope<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"function"</span>)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-3" data-target-annotation="2">2</button><span id="annotated-cell-3-9" class="code-annotation-target"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> BROKEN_mock_poem(day, poem<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>POEM):</span>
<span id="annotated-cell-3-10">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> poem[day]</span>
<span id="annotated-cell-3-11"></span>
<span id="annotated-cell-3-12"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-3" data-target-annotation="3">3</button><span id="annotated-cell-3-13" class="code-annotation-target"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@mock.patch</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"poem.get_poem_line_for_day"</span>)</span>
<span id="annotated-cell-3-14"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_IS_BROKEN_(patched_poem, BROKEN_mock_poem):</span>
<span id="annotated-cell-3-15">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Uses deferred instantiation."""</span></span>
<span id="annotated-cell-3-16">    patched_poem.return_value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> BROKEN_mock_poem(day_name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday"</span>)</span>
<span id="annotated-cell-3-17">    result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> poem.get_poem_line_for_day()                                       </span>
<span id="annotated-cell-3-18">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday's child is full of woe"</span></span>
<span id="annotated-cell-3-19"></span>
<span id="annotated-cell-3-20">    patched_poem.return_value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> BROKEN_mock_poem(day_name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday"</span>)</span>
<span id="annotated-cell-3-21">    result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> poem.get_poem_line_for_day()</span>
<span id="annotated-cell-3-22">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday's child is loving and giving"</span></span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
<div class="cell-annotation">
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-3" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-3" data-code-lines="8" data-code-annotation="1">We attempt to shift the utility to a fixture in order to use it across multiple tests.</span>
</dd>
<dt data-target-cell="annotated-cell-3" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-3" data-code-lines="9" data-code-annotation="2"><code>pytest</code> fixtures will eagerly evaluate the arguments to the fixture and raise an exception, as a value for <code>day</code> has not been set.</span>
</dd>
<dt data-target-cell="annotated-cell-3" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-3" data-code-lines="13" data-code-annotation="3">The test is not run due to the above exception.</span>
</dd>
</dl>
</div>
</div>
<p>The problem with the above code is that <code>pytest</code> fixtures expect other fixtures, not placeholder arguments. As part of their dependency injection process, <code>pytest</code> fixtures will evaluate all arguments passed to the fixture before tests are run.</p>
</section>
<section id="mock-with-fixture---fixed" class="level3">
<h3 class="anchored" data-anchor-id="mock-with-fixture---fixed">Mock with Fixture - Fixed</h3>
<p>We need to implement some minor tweaks to the broken fixture in order to delay evaluation of the parameters. In this way, we make the fixture “lazy” by using a factory function to instantiate the poem line within the test, rather than before it.</p>
<div id="2b319b56" class="cell" data-execution_count="4">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="annotated-cell-4" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-4-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> mock</span>
<span id="annotated-cell-4-2"></span>
<span id="annotated-cell-4-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="annotated-cell-4-4"></span>
<span id="annotated-cell-4-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> poem</span>
<span id="annotated-cell-4-6"></span>
<span id="annotated-cell-4-7"></span>
<span id="annotated-cell-4-8"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.fixture</span>(scope<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"function"</span>)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="1">1</button><span id="annotated-cell-4-9" class="code-annotation-target"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> mock_poem_line_factory():</span>
<span id="annotated-cell-4-10">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Factory function that mocks expected return values."""</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="2">2</button><span id="annotated-cell-4-11" class="code-annotation-target">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _get_poem_line(day_name: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, poem: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">dict</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> POEM) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="annotated-cell-4-12">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> poem[day_name]</span>
<span id="annotated-cell-4-13">    </span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="3">3</button><span id="annotated-cell-4-14" class="code-annotation-target">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> _get_poem_line</span>
<span id="annotated-cell-4-15"></span>
<span id="annotated-cell-4-16"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@mock.patch</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"poem.get_poem_line_for_day"</span>)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="4">4</button><span id="annotated-cell-4-17" class="code-annotation-target"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_poem_line_any_day_we_like(patched_poem, mock_poem_line_factory):</span>
<span id="annotated-cell-4-18">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Uses deferred instantiation."""</span></span>
<span id="annotated-cell-4-19">    patched_poem.return_value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> mock_poem_line_factory(day_name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday"</span>)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="5">5</button><span id="annotated-cell-4-20" class="code-annotation-target">    result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> poem.get_poem_line_for_day()</span>
<span id="annotated-cell-4-21">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wednesday's child is full of woe"</span></span>
<span id="annotated-cell-4-22"></span>
<span id="annotated-cell-4-23">    patched_poem.return_value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> mock_poem_line_factory(day_name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday"</span>)</span>
<span id="annotated-cell-4-24">    result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> poem.get_poem_line_for_day()</span>
<span id="annotated-cell-4-25">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Friday's child is loving and giving"</span></span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
<div class="cell-annotation">
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-4" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="9" data-code-annotation="1">The fixture will now act as a factory function, encapsulating the instantiation of the values we wish to return. This gives us more control over when the <code>day_name</code> parameter is evaluated. Note that the fixture signature takes no arguments, though you could pass it other <code>pytest</code> fixtures if you needed to.</span>
</dd>
<dt data-target-cell="annotated-cell-4" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="11,12" data-code-annotation="2">The internal <code>_get_poem_line</code> signature defines the arguments needed to control which poem lines you wish to return.</span>
</dd>
<dt data-target-cell="annotated-cell-4" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="14" data-code-annotation="3">Note the factory function should return the internal itself, rather than its value - we need to delay evaluation.</span>
</dd>
<dt data-target-cell="annotated-cell-4" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="17" data-code-annotation="4">Don’t forget to pass in the fixture to the test signature. It must come after the alias we used for <code>mock.patch</code>, due to the decorator.</span>
</dd>
<dt data-target-cell="annotated-cell-4" data-target-annotation="5">5</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="20" data-code-annotation="5">The mock fixture gets evaluated when we attempt to patch the SUT.</span>
</dd>
</dl>
</div>
</div>
</section>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>We’ve demonstrated how to go from a locally scoped mock to a fixture mock:</p>
<ol type="1">
<li>Demonstrating how to achieve a straightforward mock:patch combo with a local utility.</li>
<li>Demonstrating that the same approach does not work for a <code>pytest</code> fixture.</li>
<li>Updating the fixture to use lazy evaluation.</li>
</ol>
<p>Please feel free to share your own thoughts and ideas in the comment section below (GitHub login required)! If you spot an error with this article, or have a suggested improvement then feel free to <a href="https://github.com/r-leyshon/blogging/issues">raise an issue on GitHub</a>.</p>
<p id="fin">
<i>fin!</i>
</p>


</section>

 ]]></description>
  <category>How-to</category>
  <category>pytest</category>
  <category>Unit tests</category>
  <category>mocking</category>
  <category>pytest-in-plain-english</category>
  <category>patching</category>
  <category>lazy</category>
  <category>lazy evaluation</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/20-lazy-mocking.html</guid>
  <pubDate>Tue, 05 Nov 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/20-lazy-mocking/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Commit with Clarity</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/19-github-audit-trail.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/intro-img.jpg" alt="A cute cat-like character inspects code with a magnifying glass." style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<blockquote class="blockquote">
<p>“You don’t need version control any more than a trapese [sic] artist needs a safety net.” Mark Cidade, <a href="https://stackoverflow.com/posts/250991/revisions" target="_blank">Stack Overflow</a></p>
</blockquote>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>Most developers spend time reading other people’s code - for Pull Reviews (PRs) or to learn more about a software package, for example. Every so often, a developer may need to review an entire repository and possibly even its commit history. This can happen when publishing a mature codebase.</p>
<p>Making sense of other peoples’ commit histories can be challenging, especially if that work is exploratory. Understanding the myriad reasons for any single change after the fact is reliant upon whether the developer had time to document their reasons for their implementation. Even if you work alone, perhaps that other developer will be yourself at times. Needing to pick through your code’s commit history in order to find a bug-free version.</p>
<p>The good news is that there are a few tools that Git and GitHub can provide us, and one or two easy habits that will greatly assist your colleagues in comprehending your Git activity. In fact, the GitHub User Interface (UI) is designed in subtle ways, to nudge you towards some of these behaviours. The sort of things that are not required but are incredibly helpful to others - including your future self.</p>
<p><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/js-optimized.gif" class="shaded_box img-fluid" style="display:block;margin-left:auto;margin-right:auto;width:60%;" alt="Jerry Springer final word"> <br> Read on for more on how to be kind to yourself, and those poor souls who need to read your work - GitHub has your back.</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-1-contents" aria-controls="callout-1" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Opinions Ahead… (click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-1" class="callout-1-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>Every Git user has their own way of doing things. Many experts on the matter have diverging opinions on pretty much everything Git has to offer. In this article, I am offering a medley of advice from colleagues, trial, error and pain incurred. I’d bet that not all the advice here will chime with you.</p>
<p>This article will include a fair amount of opinion &amp; appreciation for how GitHub helps promote considerate behaviours. You are most welcome to disagree and share your own opinions. There is a comments section at the bottom of the page to facilitate that.</p>
</div>
</div>
</div>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Programmers who use GitHub to share code. The kind of Programmers who:</p>
<ul>
<li>may not have used GitHub to collaborate in teams (previously me).</li>
<li>pride themselves on knowing ‘just enough Git to survive’ (me).</li>
<li>have been using GitHub for many years and wonder if their behaviours can be adjusted to be more helpful to others (future me).</li>
</ul>
</section>
</section>
<section id="commit-messages" class="level2">
<h2 class="anchored" data-anchor-id="commit-messages">Commit Messages</h2>
<p>One of the most ‘freeing’ style guide rules an organisation can institute are rules for what it considers to be good commit messages. Having enough rules to help take the pondering out of writing your message content will help get you back onto the development work quicker.</p>
<p>Having well-structured commit messages also helps others to quickly understand the types of changes that have been implemented in a pull request (PR). Some of my previous colleagues took the time to put together <a href="https://datasciencecampus.github.io/coding-standards/version-control.html" target="_blank">guidance on such matters</a>.</p>
<blockquote class="blockquote">
<p>” <code>&lt;type&gt;: &lt;subject&gt;</code></p>
<p>type</p>
<p>Must be one of the following:</p>
<ul>
<li>feat: A new feature</li>
<li>fix: A bug fix</li>
<li>doc: Documentation only changes</li>
<li>style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)</li>
<li>refactor: A code change that neither fixes a bug or adds a feature</li>
<li>perf: A code change that improves performance</li>
<li>test: Adding missing tests</li>
<li>chore: Changes to the build process or auxiliary tools and libraries such as documentation generation Do not capitalise the first letter.</li>
</ul>
<p>subject</p>
<p>The subject contains succinct description of the change:</p>
<ul>
<li>use the imperative, present tense: “Change” not “Changed” nor “Changes”</li>
<li>do capitalise first letter</li>
<li>no dot (.) at the end”</li>
</ul>
</blockquote>
<p>Many thanks to those Predeleagues (Predecessor Colleagues, my own portmanteau) who took the time to write that guidance.</p>
<p>This has been super useful for 99% of my commit messages that are one-liners. On the odd occasion I’ve felt a multi-line message was required, having the guidance for that written down has been a priceless time-saver. Using a commit type has also helped in moving backwards through the version history - helping my future self to target specific refactoring diffs or taking a rummage through a feature that introduced some newly discovered bug.</p>
<p>Considering the commit subject, I would subscribe to <a href="https://best-practice-and-impact.github.io/qa-of-code-guidance/version_control.html#write-short-and-informative-commits" target="_blank">ONS Duck Book’s advice</a> - aspire to short and informative messages.</p>
</section>
<section id="link-prs-with-issues" class="level2">
<h2 class="anchored" data-anchor-id="link-prs-with-issues">Link PRs with Issues</h2>
<p>To those poor souls who have jobs that involve reading through commit logs - we salute you.</p>
<p>Oftentimes understanding the <strong>reason</strong> for a commit can become a mystery. Trying to understand the purpose of some contextless diff can become a real chore. By ensuring that each PR is linked with an issue (or issues), we can help provide the context needed to anyone who needs to pick through or pick apart our code.</p>
<p>The GitHub UI helps to nudge us in the right direction. Below I create a new issue in the repo for this blog:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/01.png" class="img-fluid figure-img"></p>
<figcaption>Creating an issue in GitHub</figcaption>
</figure>
</div>
<p>Notice that there is an option on the right hand side, under <code>Development</code> to <code>Create a branch</code> for this issue. Click this and GitHub will create a branch with a naming convention that links the branch to the issue.</p>
<div class="quarto-figure quarto-figure-center" style="display:block;margin-left:auto;margin-right:auto;">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/02.png" class="img-fluid figure-img" style="width:50.0%"></p>
<figcaption>Creating a new branch from an issue in GitHub</figcaption>
</figure>
</div>
<p>Once finished with the branch, we raise a PR. To link the PR with the issue, ensure to use one of <a href="https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue" target="_blank">GitHub’s keywords</a> to automatically close the issue on merge. In this example, including the line <code>Fixes #71</code> in a comment within the PR will link the issue. Several issues can be linked in this way, eg <code>Fixes #71, Resolves #72</code> and so on. In this way, we have given an issue, branch and PR a clear identity.</p>
</section>
<section id="consider-your-merge-strategy" class="level2">
<h2 class="anchored" data-anchor-id="consider-your-merge-strategy">Consider Your Merge Strategy</h2>
<p>When merging a PR in the GitHub UI, 3 options are presented:</p>
<ul>
<li>Merge commit (the default)</li>
<li>Squash and merge</li>
<li>Rebase and merge</li>
</ul>
<p>These options are a little involved and I would recommend reading <a href="https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges" target="_blank">GitHub’s merge docs</a> on the matter for a more detailed breakdown.</p>
<p>To help visualise the differences in these approaches, consider the following diagram:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/1.webp" class="img-fluid figure-img"></p>
<figcaption>Merge strategy comparison diagram. Image © <a href="https://matt-rickard.com/squash-merge-or-rebase" target="_blank">Matt Rickard</a>.</figcaption>
</figure>
</div>
<p>In the following scenarios, I will walk through commit history following the three different merges. In all examples, I will refer to the target branch receiving the commits as <code>main</code> and the topic branch containing the additions as <code>feature</code>.</p>
<section id="rebase-merge" class="level3">
<h3 class="anchored" data-anchor-id="rebase-merge">Rebase Merge</h3>
<p>When a rebase merge is used, new commits that mirror those in <code>feature</code> branch are created in <code>main</code>. You keep every commit and maintain a linear commit history. In the diagram below, I compare commits in <a href="https://github.com/r-leyshon/rebase-merge-example" target="_blank">a repo with an example rebase merge</a>. Note that there are 2 commits from <code>feature</code> branch with commit messages like “FEATURE: …”. Merging the PR creates new commits to <code>main</code>, which have all the file edits from <code>feature</code>. However, note that these new commits have different hashes to those in <code>feature</code>.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/03.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="Comparing main and feature following rebase merge. Click to enlarge."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/03.png" class="img-fluid figure-img" alt="Comparing main and feature following rebase merge. Click to enlarge."></a></p>
<figcaption>Comparing <code>main</code> and <code>feature</code> following rebase merge. Click to enlarge.</figcaption>
</figure>
</div>
</section>
<section id="squash-and-merge" class="level3">
<h3 class="anchored" data-anchor-id="squash-and-merge">Squash and Merge</h3>
<p>On selecting “Squash and merge” in the GitHub UI, A new “summary commit” is created in <code>main</code>. This condenses all of the file changes from <code>feature</code> into a single, new commit. You can see that even in the trivial example <a href="https://github.com/r-leyshon/squash-commit-example" target="_blank">squash merge repository</a>, the outcome is that <code>main</code> branch will be much more succinct than in the other merge strategies.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/04.png" class="lightbox" data-gallery="quarto-lightbox-gallery-2" title="Comparing main and feature following a squash merge. Click to enlarge."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/04.png" class="img-fluid figure-img" alt="Comparing main and feature following a squash merge. Click to enlarge."></a></p>
<figcaption>Comparing <code>main</code> and <code>feature</code> following a squash merge. Click to enlarge.</figcaption>
</figure>
</div>
<p>Inspecting the automated squash merge commit message demonstrates how GitHub summarises the activity from <code>feature</code> branch.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/05.png" class="img-fluid figure-img"></p>
<figcaption>Squash commit generated commit message.</figcaption>
</figure>
</div>
<p>The squash merge strategy is a good candidate for busy projects with many contributors. It’s also a neat way to avoid noise in your version history, such as experimental ‘cruft’ or benign changes that you’d rather summarise. It is important to note that once merged to <code>main</code>, you should go ahead and delete the <code>feature</code> branch. If you continue to work in <code>feature</code>, there is a likelihood of diverging with <code>main</code>, ending up in merge conflicts to resolve.</p>
</section>
<section id="merge-commit" class="level3">
<h3 class="anchored" data-anchor-id="merge-commit">Merge Commit</h3>
<p>Finally, we consider the default merge behaviour - the <a href="https://github.com/r-leyshon/merge-commit-example" target="_blank">merge commit</a>. This behaviour will create a new commit in <code>main</code> that introduces the changes in <code>feature</code>. Notice that in this scenario, the commits in <code>main</code> and <code>feature</code> share the same content and commit hashes, as opposed to a rebase merge which “pretends” those commits happened in <code>main</code>.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/06.png" class="lightbox" data-gallery="quarto-lightbox-gallery-3" title="Comparing main and feature following a merge commit. Click to enlarge."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/06.png" class="img-fluid figure-img" alt="Comparing main and feature following a merge commit. Click to enlarge."></a></p>
<figcaption>Comparing <code>main</code> and <code>feature</code> following a merge commit. Click to enlarge.</figcaption>
</figure>
</div>
<p>This is perhaps the noisiest merge strategy, as the commits of each feature branch are all merged to main, plus an additional commit that marks the merge. Use this scenario if you consider it really important to be able to revisit the commit history.</p>
</section>
<section id="overview-of-merge-strategies" class="level3">
<h3 class="anchored" data-anchor-id="overview-of-merge-strategies">Overview of Merge Strategies</h3>
<table class="caption-top table">
<colgroup>
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Merge Type</strong></th>
<th><strong>What Happens</strong></th>
<th><strong>How It Looks</strong></th>
<th><strong>Best For</strong></th>
<th><strong>Downside</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Merge Commit</strong></td>
<td>A new commit is created to merge the PR, keeping all individual commits intact.</td>
<td>A merge commit is added, and you can see all the individual commits from the PR.</td>
<td>Keeping a detailed history with all commits preserved.</td>
<td>Can clutter the commit history with many small or trivial commits.</td>
</tr>
<tr class="even">
<td><strong>Squash and Merge</strong></td>
<td>All commits in the PR are combined into one single commit before merging.</td>
<td>Only one commit appears in the history, representing all changes from the PR.</td>
<td>Keeping the commit history clean and concise.</td>
<td>Loses the individual commit details from the PR.</td>
</tr>
<tr class="odd">
<td><strong>Rebase and Merge</strong></td>
<td>The commits from the PR are applied directly on top of the main branch, maintaining their original structure but without a merge commit.</td>
<td>The history looks linear, with the PR commits applied after the latest main branch commit, with no merge commit in between.</td>
<td>Keeping a linear, clean commit history without merge commits.</td>
<td>Can be tricky with conflicts, especially for complex histories.</td>
</tr>
</tbody>
</table>
</section>
<section id="selecting-a-merge-strategy" class="level3">
<h3 class="anchored" data-anchor-id="selecting-a-merge-strategy">Selecting a Merge Strategy</h3>
<p>Understanding which strategy is for the best depends on the context in which you’re working. Your 2 options for maintaining a full commit history are either merge commit or rebase and merge. If you would rather summarise the commit history, then squash commit is the right approach. If I’m working on a solo project, I tend to merge commit. If I’m working in a team, I tend to squash commit in order to keep the version history manageable.</p>
<p>Some people bemoan the squash commit approach, with 2 main complaints that I’ve encountered:</p>
<ol type="1">
<li>The full commit history is not preserved.</li>
<li>It decreases contributors’ apparent GitHub activity.</li>
</ol>
<p>My responses follow:</p>
<ol type="1">
<li>Deleted branches can be restored.</li>
<li>GitHub activity is a poor proxy for impact.</li>
</ol>
<p>Admittedly, the squash merge approach works best for projects where there is a disciplined approach to PRs. If PRs represent a tangible unit of work, such as a bug is fixed or a new feature is implemented, then squash merge works really well in my experience. Individual branches are short-lived and the risk of merge conflicts are minimised.</p>
<p>However, some projects are not like that. Particularly in exploratory work where some PRs become sprawling, long-lived with diffs numbering tens-of-thousands of lines. It may be advisable to merge commit in those scenarios. Personally, <a href="https://git-scm.com/book/en/v2/Git-Branching-Rebasing#_rebase_peril" target="_blank">I never rebase</a> when working with others.</p>
</section>
</section>
<section id="organising-issues" class="level2">
<h2 class="anchored" data-anchor-id="organising-issues">Organising Issues</h2>
<p>Issues provide the context for PRs. Understanding why a PR is necessary is very helpful to collaborators and will be appreciated by anyone who needs to look back over your version history.</p>
<p>A feature that I consider to be under-utilised in GitHub are <a href="https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/about-milestones" target="_blank">milestones</a>. The milestone feature allows you to group related work, such as grouping issues and PRs with an epic. These milestones can then be added to GitHub projects to coordinate burn-down effort. Find the milestones interface under the issues tab.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/07.png" class="lightbox" data-gallery="quarto-lightbox-gallery-4" title="GitHub Milestone interface. Click to expand."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/07.png" class="img-fluid figure-img" alt="GitHub Milestone interface. Click to expand."></a></p>
<figcaption>GitHub Milestone interface. Click to expand.</figcaption>
</figure>
</div>
<p>Adding this sort of structure can be great in helping collaborators understand where your work fits into more tangible delivery, and your product managers will thank you for it. Below I show how with a few clicks, the milestone can be used to create an informative delivery roadmap.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/08.png" class="lightbox" data-gallery="quarto-lightbox-gallery-5" title="Milestone on a GitHub roadmap. Click to expand."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/08.png" class="img-fluid figure-img" alt="Milestone on a GitHub roadmap. Click to expand."></a></p>
<figcaption>Milestone on a GitHub roadmap. Click to expand.</figcaption>
</figure>
</div>
<p>You can see in the project that I have grouped the issues by milestone on the left hand-side. Also, by adding the milestone to the date fields, you get the blue date line appearing on the roadmap to the right of the screen.</p>
<p>Recently, GitHub has unveiled some significant <a href="https://github.blog/changelog/2024-10-01-evolving-github-issues-public-beta/" target="_blank">changes to its issues</a>. The nested issues feature adds even more flexibility in grouping units of work together. This feature is in public beta at time of writing. <a href="https://github.com/features/issues/signup" target="_blank">Join the waitlist here</a>.</p>
</section>
<section id="tips-for-auditing" class="level2">
<h2 class="anchored" data-anchor-id="tips-for-auditing">Tips for Auditing</h2>
<p>If you are tasked with making sense of someone else’s code for PR or commit history for a release, then there are a few tools &amp; tricks that can help make your life a bit easier. I’ll cover some of them here.</p>
<section id="assign-blame" class="level3">
<h3 class="anchored" data-anchor-id="assign-blame">Assign BLAME</h3>
<blockquote class="blockquote">
<p>“It is ill to praise, and worse to blame, the thing which you do not understand.” Leonardo da Vinci, “Life, art and science, the thoughts of Leonardo”, p.67, 2013, Lulu.com</p>
</blockquote>
<p>At times, it may be useful to identify the person who wrote some code. It’s not completely obvious, but by viewing a file and toggling from <code>Preview</code> to <code>Blame</code>, you get a line-by-line view of who commited what and when.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/09.png" class="lightbox" data-gallery="quarto-lightbox-gallery-6" title="GitHub blame view. Click to expand."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/09.png" class="img-fluid figure-img" alt="GitHub blame view. Click to expand."></a></p>
<figcaption>GitHub blame view. Click to expand.</figcaption>
</figure>
</div>
<p>Note that there are also handy tools such as <a href="https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens" target="_blank">GitLens for VSCode</a> that can bring the Git blame into your IDE.</p>
</section>
<section id="grep-ftw" class="level3">
<h3 class="anchored" data-anchor-id="grep-ftw">Grep FTW</h3>
<p>If you need to check all occurrences of a specific pattern anywhere within a repository, there’s a great little tool called <code>git grep</code> that will help. In an interactive terminal, run <code>git grep &lt;INSERT_PATTERN&gt;</code> and all occurrences of that pattern within any file contents will be returned. If the pattern has spaces, just wrap the pattern in speech marks.</p>
<p>If there are loads of files and you want to search for a pattern in the filenames, then use:</p>
<p><code>git ls-files | grep &lt;INSERT_PATTERN&gt;</code></p>
<p>If you’d like to search for specific patterns in commit messages:</p>
<p><code>git log --grep=&lt;INSERT_PATTERN&gt;</code></p>
<p>If you need to scan an entire commit history for the presence of a pattern in the contents of the files, then this is achieved with</p>
<p><code>git log -G &lt;INSERT_PATTERN&gt;</code></p>
<p>That pattern takes regular expressions so it can be quite flexible in combination with wildcards. This is super useful if you need to check whether a specific secret was ever added to the commit history.</p>
</section>
<section id="github-vscode" class="level3">
<h3 class="anchored" data-anchor-id="github-vscode">GitHub &amp; VSCode</h3>
<p>The previous tip is a feature of Git rather than GitHub. In order to execute the commands, you’ll need a Command Line Interface and a local clone of the repository. That might not always be convenient. Did you know that since the low(ish) key Microsoft takeover of GitHub, you can now access VSCode directly from the GitHub UI?</p>
<p>By simply typing <code>.</code> (full stop or period, depending on what side of the Atlantic you learned English) you will open your repository within VSCode directly within your browser window. If you’d prefer to open VSCode in a new browser window, type <code>&gt;</code> instead. This interface is handy for adding in new files or adjusting the content of existing files.</p>
<p>If you’d like an interactive terminal, or to test out code, you’ll need to switch over to GitHub Codespaces. Again, this is a free service that GitHub provide that provisions some compute for you. To launch a ‘live’ version of VSCode in your browser, click on the burger icon in in the ribbon to the left, then <code>Terminal</code> -&gt; <code>New Terminal</code>. You will be presented with options in the terminal asking you to launch GitHub Codespaces or to continue in a local clone of the repo.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/10.png" class="lightbox" data-gallery="quarto-lightbox-gallery-7" title="Launching Codespaces. Click to expand."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/10.png" class="img-fluid figure-img" alt="Launching Codespaces. Click to expand."></a></p>
<figcaption>Launching Codespaces. Click to expand.</figcaption>
</figure>
</div>
<p>Once Codespaces is launched, you have an interactive terminal to carry out your favourite operations. Here I’m displaying the <code>git grep</code> command in my browser with Codespaces.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/19-github-audit-trail/11.png" class="lightbox" data-gallery="quarto-lightbox-gallery-8" title="Running terminal operations with Codespaces. Click to expand."><img src="https://thedatasavvycorner.netlify.app/www/19-github-audit-trail/11.png" class="img-fluid figure-img" alt="Running terminal operations with Codespaces. Click to expand."></a></p>
<figcaption>Running terminal operations with Codespaces. Click to expand.</figcaption>
</figure>
</div>
</section>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>In this article we covered considerate Git practice and how GitHub helps to nudge us toward these behaviours. Specifically:</p>
<ul>
<li><strong>Commit Messages</strong>: The value of having well-structured commit messages.</li>
<li><strong>Linking PRs with Issues</strong>: Providing necessary context for reviewers.</li>
<li><strong>Merge Strategies</strong>: Advantages and disadvantages of the options GitHub provides in their UI.</li>
<li><strong>Organising Issues</strong>: Using milestones to group related work, enhancing coordination in projects.</li>
<li><strong>Tips for Auditing</strong>: Tools and techniques for inspecting a commit history, including the use of Git blame and grep commands.</li>
<li><strong>GitHub &amp; VSCode Integration</strong>: Describes how to access VSCode directly from GitHub for streamlined code management.</li>
</ul>
<p>Please feel free to share your own thoughts and ideas in the comment section below (GitHub login required)! If you spot an error with this article, or have a suggested improvement then feel free to <a href="https://github.com/r-leyshon/blogging/issues">raise an issue on GitHub</a>.</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>To past and present colleagues who have helped to discuss pros and cons, establishing practice and firming-up some opinions. Particularly:</p>
<ul>
<li>Ian</li>
<li>Rob</li>
<li>Adrian</li>
<li><a href="https://www.youtube.com/watch?v=BCQHnlnPusY" target="_blank">Dan Shiffman</a></li>
</ul>
<p id="fin">
<i>fin!</i>
</p>


</section>

 ]]></description>
  <category>Explanation</category>
  <category>GitHub</category>
  <category>Git</category>
  <category>Version Control</category>
  <category>Software Development</category>
  <category>Blame</category>
  <category>VSCode</category>
  <category>Git Grep</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/19-github-audit-trail.html</guid>
  <pubDate>Sun, 13 Oct 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/19-github-audit-trail/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Choose Your Own Adventure with ChatGPT</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/18-jungle-quest-app.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/intro-img.jpg" alt="Adventurer in a futuristic cyberpunk wilderness." style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<blockquote class="blockquote">
<p>“ChatGPT amplifies human potential, turning thoughts into creation. It reminds us that imagination, when paired with technology, can shape the future.” ChatGPT on the creative potential of ChatGPT.</p>
</blockquote>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>This article will guide you through the process of building a Choose Your Own Adventure game using OpenAI generative AI models and Python Shiny. We’ll walk through the entire journey, from concept to deployment, with a focus on the iterative development process and the use of ChatGPT for generating storylines.</p>
<p>Choose Your Own Adventure is a classic storytelling format where the reader is presented with a series of choices that guide the story in different directions. The game is interactive, allowing the reader to make decisions that affect the outcome of the story. They were popular in the 1980s and early 1990s, widely available as graphic novels and comics with a fantasy or adventure theme.</p>
<p>The aim of this project is to produce an interactive application that will use AI to generate the branching outcomes depending on the choices made by the user. We will provide the user with an introduction to the theme and we will provide the model with the rules for the game. Each game will play out differently depending on the creative interplay between the user and the model.</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-1-contents" aria-controls="callout-1" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Inspiration for This Blog (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-1" class="callout-1-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>The application concept was heavily influenced by the YouTube video by Tech with Tim called <a href="https://www.youtube.com/watch?v=nhYcTh6vw9A" target="_blank">Python AI Choose Your Own Adventure Game - Tutorial</a>. This tutorial uses a more complicated stack behind the scenes and resulted in a game that solved itself - essentially the model would also generate ‘imagined’ user responses through to completion. The tutorial was published only 11 months ago at the time of writing, though the code would not run without significant adaptation, due to a raft of breaking changes within <code>langchain</code>.</p>
<p>I was inspired by the playful use of generative AI but could see that a few things could be done to improve the reproducibility of the code. Also, by simplifying the stack required to generate the game responses, it is hoped that the risk of deprecation and breaking changes will be reduced, increasing the longevity of the code. Finally, an application would be needed in order to allow the human player and model to take turns in playing the game. I have opted to use <a href="https://shiny.posit.co/py/" target="_blank"><code>shiny</code> for Python</a> in order to achieve this, though the same functionality could be achieved with many other dashboarding solutions.</p>
</div>
</div>
</div>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Python programmers who are curious about building AI-enabled applications. Some familiarity with <code>shiny</code> may be assumed. For an overview and intro to building <code>shiny</code> apps with Python, check out my other blogs:</p>
<ul>
<li><a href="../blogs/01-state-of-pyshiny.html">The State of Python <code>shiny</code></a></li>
<li><a href="../blogs/02-getting-started-pyshiny.html">Let’s Build a Basic Python <code>shiny</code> App</a></li>
</ul>
</section>
<section id="what-youll-need" class="level3">
<h3 class="anchored" data-anchor-id="what-youll-need">What You’ll Need</h3>
<ul class="task-list">
<li><label><input type="checkbox">Command line access</label></li>
<li><label><input type="checkbox">Python know-how</label>
<ul class="task-list">
<li><label><input type="checkbox">Configure a virtual environment</label></li>
<li><label><input type="checkbox">Dependency management</label></li>
</ul></li>
<li><label><input type="checkbox">Basic knowledge of Python Shiny</label></li>
<li><label><input type="checkbox">An OpenAI API key</label></li>
</ul>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>requirements.txt</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" data-filename="requirements.txt" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb1-1">openai==1.30.4</span>
<span id="cb1-2">shiny==1.1.0</span>
<span id="cb1-3">shinyswatch==0.7.0</span></code></pre></div></div>
</div>
<div class="callout callout-style-simple callout-none no-icon callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-2-contents" aria-controls="callout-2" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon no-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">None</span>A Note on the Purpose (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-2" class="callout-2-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>This article intends to discuss clearly. It doesn’t aim to be clever or impressive. Its aim is to extend understanding without overwhelming the reader. The code may not always be optimal, favouring a simplistic approach wherever possible. The application may not be to your liking. The purpose is to help learners build a simple AI-driven application rather than a masterpiece. If you have ideas for improvements to the app or blog, please feel free to leave a comment at the end of the article.</p>
</div>
</div>
</div>
<p>The final application is presented below, hosted with <a href="https://www.shinyapps.io/" target="_blank">shinyapps.io</a>. Please note, this is not configured for high traffic. Let me know if the app fails to launch for you by leaving a comment at the end of the blog. You will need an OpenAI API key in order to prompt the model. The app has a link to the sign up page if you would like to give it a try.</p>
<p>If you would prefer to read the source code for the application before proceeding with the article, then please click on the GitHub icon at the top-right of the application. If you would rather interact with the application in a full-sized window, then visit <a href="https://richleysh84.shinyapps.io/choose-adventure/" target="_blank">Jungle Quest app on shinyapps.io</a>. This app is set up to query the gpt-3.5-turbo model, but as you proceed through the tutorial, feel free to experiment with other available models (they can behave quite differently).</p>
<iframe class="iframey" src="https://richleysh84.shinyapps.io/choose-adventure/" style="overflow:hidden;margin:0;padding:0;width:100%;height:50rem">
</iframe>
</section>
</section>
<section id="setting-up-the-development-environment" class="level2">
<h2 class="anchored" data-anchor-id="setting-up-the-development-environment">Setting up the Development Environment</h2>
<p>I’d recommend using <a href="https://code.visualstudio.com/" target="_blank">VSCode</a> with the <a href="https://marketplace.visualstudio.com/items?itemName=Posit.shiny" target="_blank"><code>shiny</code> extension</a> to help run and debug the app. It has a handy utility for launching your app within the VSCode interface or expanding it to full screen in your default browser. This is priceless when testing your User Interface’s (UI) appearance on different browsers.</p>
<p>You’ll need to create and activate a virtual environment of your choice, I have used python 3.12 in the examples without any issues. install the dependencies listed in the <code>requirements.txt</code> file, and finally ensure that VSCode is configured to <a href="https://code.visualstudio.com/docs/python/environments" target="_blank">use the virtual environment</a>.</p>
</section>
<section id="iterative-development" class="level2">
<h2 class="anchored" data-anchor-id="iterative-development">Iterative Development</h2>
<div class="callout callout-style-default callout-caution callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Caution
</div>
</div>
<div class="callout-body-container callout-body">
<p>Take care with your OpenAI APi credentials. I demonstrate hard-coding these credentials within python scripts for simplicity. I’d advise storing them in a git-ignored secrets file or using the <a href="https://pypi.org/project/python-dotenv/" target="_blank"><code>python-dotenv</code></a> package to keep them safe. Take care not to accidentally commit these credentials and expose them on GitHub. Leakage of OpenAI credentials is the fastest-growing type of secret leak, according to <a href="https://www.gitguardian.com/state-of-secrets-sprawl-report-2024" target="_blank">Git Guardian’s State of Secret Sprawl 2024</a>.</p>
<p>Finally, please note that usage of the openai service is associated with your account via your API key. Please conform to the <a href="https://openai.com/policies/" target="_blank">service’s usage policy.</a></p>
</div>
</div>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Code Annotations
</div>
</div>
<div class="callout-body-container callout-body">
<p>Click on the numbered points within the code blocks, to reveal tooltips with additional explanations.</p>
</div>
</div>
<div class="tabset-margin-container"></div><div id="iterations" class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-1-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-1" aria-controls="tabset-1-1" aria-selected="true" href="">Iteration 1</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-2" aria-controls="tabset-1-2" aria-selected="false" href="">Iteration 2</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-3" aria-controls="tabset-1-3" aria-selected="false" href="">Iteration 3</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-4-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-4" aria-controls="tabset-1-4" aria-selected="false" href="">Iteration 4</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-5-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-5" aria-controls="tabset-1-5" aria-selected="false" href="">Iteration 5</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-6-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-6" aria-controls="tabset-1-6" aria-selected="false" href="">Iteration 6</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-7-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-7" aria-controls="tabset-1-7" aria-selected="false" href="">Iteration 7</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-8-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-8" aria-controls="tabset-1-8" aria-selected="false" href="">Iteration 8</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-9-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-9" aria-controls="tabset-1-9" aria-selected="false" href="">Iteration 9</a></li></ul>
<div id="iterations" class="tab-content">
<div id="tabset-1-1" class="tab-pane active" aria-labelledby="tabset-1-1-tab">
<p>In this early prototype, we will focus on using the openai python client to send a basic prompt to the gpt-3.5-turbo model.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-3" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 1: How to query the OpenAI API."""</span></span>
<span id="annotated-cell-3-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-3-3"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-3" data-target-annotation="1">1</button><span id="annotated-cell-3-4" class="code-annotation-target">API_KEY <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;INSERT_YOUR_KEY_HERE&gt;"</span></span>
<span id="annotated-cell-3-5"></span>
<span id="annotated-cell-3-6"></span>
<span id="annotated-cell-3-7"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> query_openai(prompt: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, api_key: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="annotated-cell-3-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Query the chat completions endpoint.</span></span>
<span id="annotated-cell-3-9"></span>
<span id="annotated-cell-3-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Parameters</span></span>
<span id="annotated-cell-3-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    ----------</span></span>
<span id="annotated-cell-3-12"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    prompt: str</span></span>
<span id="annotated-cell-3-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The prompt to query the chat completions endpoint with.</span></span>
<span id="annotated-cell-3-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    api_key: str</span></span>
<span id="annotated-cell-3-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The API key to use to query the chat completions endpoint.</span></span>
<span id="annotated-cell-3-16"></span>
<span id="annotated-cell-3-17"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Returns</span></span>
<span id="annotated-cell-3-18"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    -------</span></span>
<span id="annotated-cell-3-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    str</span></span>
<span id="annotated-cell-3-20"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The response from the chat completions endpoint.</span></span>
<span id="annotated-cell-3-21"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-3-22"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-3" data-target-annotation="2">2</button><span id="annotated-cell-3-23" class="code-annotation-target">    client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.OpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>api_key)</span>
<span id="annotated-cell-3-24">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># need to handle cases where queries go wrong.</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-3" data-target-annotation="3">3</button><span id="annotated-cell-3-25" class="code-annotation-target">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="annotated-cell-3-26">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> client.chat.completions.create(</span>
<span id="annotated-cell-3-27">            model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-3" data-target-annotation="4">4</button><span id="annotated-cell-3-28" class="code-annotation-target">            messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[</span>
<span id="annotated-cell-3-29">                {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: prompt}</span>
<span id="annotated-cell-3-30">            ]</span>
<span id="annotated-cell-3-31">        )</span>
<span id="annotated-cell-3-32">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-3-33">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># in cases where the API key is invalid.</span></span>
<span id="annotated-cell-3-34">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> openai.AuthenticationError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="annotated-cell-3-35">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">raise</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">ValueError</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Is your API key valid?:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>e<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="annotated-cell-3-36"></span>
<span id="annotated-cell-3-37"></span>
<span id="annotated-cell-3-38"></span>
<span id="annotated-cell-3-39">model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> query_openai(</span>
<span id="annotated-cell-3-40">  prompt<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"What is the capital of the moon?"</span>, api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>API_KEY</span>
<span id="annotated-cell-3-41">  )</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-3" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-3" data-code-lines="4" data-code-annotation="1">Insert your API key into the API_KEY variable. Note - it’s not advisable to include your secret credentials in your python scripts like this, but for simplicity’s sake, I’m showing that here. Take care not to accidentally commit these credentials and expose them on GitHub.</span>
</dd>
<dt data-target-cell="annotated-cell-3" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-3" data-code-lines="23" data-code-annotation="2">Creates a new OpenAI client with the api_key. This client can be reused to send queries to the different service endpoints.</span>
</dd>
<dt data-target-cell="annotated-cell-3" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-3" data-code-lines="25" data-code-annotation="3">This will only pass if the key provided is valid.</span>
</dd>
<dt data-target-cell="annotated-cell-3" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-3" data-code-lines="28,29,30" data-code-annotation="4">Note the format of the messages - a list of dictionaries. The value for role can be “user”, “system” or “assistant”.</span>
</dd>
</dl>
<p>The model responds with:</p>
<blockquote class="blockquote">
<p>The moon does not have a capital as it is not a sovereign nation or political entity.</p>
</blockquote>
<p>Let’s summarise the process with a diagram:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/01.png" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="Process diagram for iteration 1 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/01.png" class="img-fluid figure-img" alt="Process diagram for iteration 1 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 1 - Click to expand</figcaption>
</figure>
</div>
<p>So far we have a basic <code>query_openai()</code> function that we can feed in a prompt and our api key. We then receive a response back from the openai model with the expected content.</p>
<p>Although this is an extremely simple process, it’s great to start off with the fundamentals. Understanding the structure of what’s being sent and received is useful when we begin embedding this logic into our <code>shiny</code> app.</p>
</div>
<div id="tabset-1-2" class="tab-pane" aria-labelledby="tabset-1-2-tab">
<p>In this iteration, we are going to introduce a system message to help guide the behaviour of the model - we want the model to act as the guide on an adventure. We’ll also need it to follow a few rules such as how to indicate the game is over.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-4" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 2: Add system &amp; welcome prompts."""</span></span>
<span id="annotated-cell-4-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-4-3"></span>
<span id="annotated-cell-4-4">API_KEY <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;INSERT_YOUR_KEY_HERE&gt;"</span></span>
<span id="annotated-cell-4-5">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-4-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-4-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-4-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-4-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-4-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-4-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-4-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-4-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-4-14"></span>
<span id="annotated-cell-4-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="1">1</button><span id="annotated-cell-4-16" class="code-annotation-target"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-4-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-4-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-4-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-4-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-4-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-4-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-4-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="2">2</button><span id="annotated-cell-4-24" class="code-annotation-target"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-4-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-4-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-4-27"></span>
<span id="annotated-cell-4-28">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="3">3</button><span id="annotated-cell-4-29" class="code-annotation-target"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-4-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-4-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-4-32"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-4-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-4-34"></span>
<span id="annotated-cell-4-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-4-36"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-4-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-4-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-4-39"></span>
<span id="annotated-cell-4-40"></span>
<span id="annotated-cell-4-41"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> query_openai(</span>
<span id="annotated-cell-4-42">        prompt: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>,</span>
<span id="annotated-cell-4-43">        api_key: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>,</span>
<span id="annotated-cell-4-44">        sys_prompt:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> _SYSTEM_MSG,</span>
<span id="annotated-cell-4-45">        start_prompt:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> WELCOME_MSG,</span>
<span id="annotated-cell-4-46">        ) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="annotated-cell-4-47">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Query the chat completions endpoint.</span></span>
<span id="annotated-cell-4-48"></span>
<span id="annotated-cell-4-49"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Parameters</span></span>
<span id="annotated-cell-4-50"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    ----------</span></span>
<span id="annotated-cell-4-51"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    prompt: str</span></span>
<span id="annotated-cell-4-52"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The prompt to query the chat completions endpoint with.</span></span>
<span id="annotated-cell-4-53"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    api_key: str</span></span>
<span id="annotated-cell-4-54"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The API key to use to query the chat completions endpoint.</span></span>
<span id="annotated-cell-4-55"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    sys_prompt: str</span></span>
<span id="annotated-cell-4-56"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The system prompt to help guide the model behaviour. By default,</span></span>
<span id="annotated-cell-4-57"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        the system prompt is set to _SYSTEM_MSG.</span></span>
<span id="annotated-cell-4-58"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    start_prompt: str</span></span>
<span id="annotated-cell-4-59"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The start prompt which will be presented to the user as the app</span></span>
<span id="annotated-cell-4-60"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        begins. By default, the start prompt is set to WELCOME_MSG.</span></span>
<span id="annotated-cell-4-61"></span>
<span id="annotated-cell-4-62"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Returns</span></span>
<span id="annotated-cell-4-63"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    -------</span></span>
<span id="annotated-cell-4-64"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    str</span></span>
<span id="annotated-cell-4-65"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The response from the chat completions endpoint.</span></span>
<span id="annotated-cell-4-66"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-4-67"></span>
<span id="annotated-cell-4-68">    client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.OpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>api_key)</span>
<span id="annotated-cell-4-69">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># need to handle cases where queries go wrong.</span></span>
<span id="annotated-cell-4-70">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="annotated-cell-4-71">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> client.chat.completions.create(</span>
<span id="annotated-cell-4-72">            model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-4" data-target-annotation="4">4</button><span id="annotated-cell-4-73" class="code-annotation-target">            messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[</span>
<span id="annotated-cell-4-74">                {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: sys_prompt},</span>
<span id="annotated-cell-4-75">                {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: start_prompt},</span>
<span id="annotated-cell-4-76">                {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: prompt},</span>
<span id="annotated-cell-4-77">            ]</span>
<span id="annotated-cell-4-78">        )</span>
<span id="annotated-cell-4-79">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-4-80">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># in cases where the API key is invalid.</span></span>
<span id="annotated-cell-4-81">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> openai.AuthenticationError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="annotated-cell-4-82">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">raise</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">ValueError</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Is your API key valid?:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>e<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="annotated-cell-4-83"></span>
<span id="annotated-cell-4-84"></span>
<span id="annotated-cell-4-85">model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> query_openai(</span>
<span id="annotated-cell-4-86">    prompt<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"What is the capital of the moon?"</span>,</span>
<span id="annotated-cell-4-87">    api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>API_KEY</span>
<span id="annotated-cell-4-88">    )</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-4" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="16,17,18" data-code-annotation="1">Certain models can get a bit overzealous and start providing imagined user input, ultimately playing the game through to completion on their own. As games go, that’s not particularly fun, so let’s try to safeguard against that behaviour with these explicit instructions.</span>
</dd>
<dt data-target-cell="annotated-cell-4" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="24,25" data-code-annotation="2">The gpt-3.5-turbo model seems to be pretty reliable at ending the game with the required pattern “The end…”. We’ll later search for this pattern to exit the app and return a message indicating game over. Interestingly, I found gpt-4 models to be fairly unreliable in following this instruction. All of the models can be configured to stream their responses too, in which case they rarely gave the specified game over pattern. I’d be interested in others’ opinions as to why this may be the case. Please feel free to leave a comment at the end of the article if you have an opinion.</span>
</dd>
<dt data-target-cell="annotated-cell-4" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="29,30,31,32,33,34,35,36,37,38" data-code-annotation="3">This welcome message will be used to introduce the game context for our users when the app launches. We will append this into the message stream and simulate the LLM greeting our user. We will also include this message when querying the model, where it will serve as what’s known as a one-shot prompt to help guide the model’s behaviour. A one-shot prompt is an example of how you’d like the model to behave.</span>
</dd>
<dt data-target-cell="annotated-cell-4" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-4" data-code-lines="73,74,75,76,77" data-code-annotation="4">We update the messages stream with our hard-coded prompts. This helps to guide both the model and the user, setting context and modelling the desired behaviour.</span>
</dd>
</dl>
<p>Thanks to the guidance in the hard-coded prompts, our model now behaves a bit differently:</p>
<blockquote class="blockquote">
<p>I’m afraid the Moon doesn’t have a capital city like countries on Earth do! Let’s focus on our adventure in the Amazon Rainforest. To begin, please choose your name, gender, and race. Additionally, select a weapon to arm yourself with on this mystical journey. The fate of finding the lost Crown of Quetzalcoatl awaits your choices!</p>
</blockquote>
<p>Notice that the model still answers the question, but guides the user back to the purpose of the app. In a later iteration, we will see how to introduce moderations as a safeguard against the user passing inappropriate content.</p>
<p>Finally, updating our process diagram to include the additional prompts, I have emphasised the changes implemented within this iteration. As we proceed, the diagram’s complexity will increase and therefore I’ll try to emphasise the changes implemented over the previous iteration only:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/02.png" class="lightbox" data-gallery="quarto-lightbox-gallery-2" title="Process diagram for iteration 2 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/02.png" class="img-fluid figure-img" alt="Process diagram for iteration 2 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 2 - Click to expand</figcaption>
</figure>
</div>
</div>
<div id="tabset-1-3" class="tab-pane" aria-labelledby="tabset-1-3-tab">
<p>This time, we’ll put together the basic UI for the app. The UI needs a text field to pass the user’s API key and the chat component. Let’s update the app script to include the <code>shiny</code> UI.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-5" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 3: A basic user interface with no server logic."""</span></span>
<span id="annotated-cell-5-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-5-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shiny <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> App, ui</span>
<span id="annotated-cell-5-4"></span>
<span id="annotated-cell-5-5">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-5-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-5-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-5-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-5-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-5-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-5-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-5-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-5-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-5-14"></span>
<span id="annotated-cell-5-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<span id="annotated-cell-5-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-5-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-5-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-5-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-5-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-5-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-5-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-5-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<span id="annotated-cell-5-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-5-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-5-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-5-27"></span>
<span id="annotated-cell-5-28">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-5-29"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-5-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-5-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-5-32"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-5-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-5-34"></span>
<span id="annotated-cell-5-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-5-36"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-5-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-5-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-5-39"></span>
<span id="annotated-cell-5-40"></span>
<span id="annotated-cell-5-41"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> query_openai(</span>
<span id="annotated-cell-5-42">        prompt: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>,</span>
<span id="annotated-cell-5-43">        api_key: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>,</span>
<span id="annotated-cell-5-44">        sys_prompt:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> _SYSTEM_MSG,</span>
<span id="annotated-cell-5-45">        start_prompt:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> WELCOME_MSG,</span>
<span id="annotated-cell-5-46">        ) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="annotated-cell-5-47">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Query the chat completions endpoint.</span></span>
<span id="annotated-cell-5-48"></span>
<span id="annotated-cell-5-49"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Parameters</span></span>
<span id="annotated-cell-5-50"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    ----------</span></span>
<span id="annotated-cell-5-51"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    prompt: str</span></span>
<span id="annotated-cell-5-52"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The prompt to query the chat completions endpoint with.</span></span>
<span id="annotated-cell-5-53"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    api_key: str</span></span>
<span id="annotated-cell-5-54"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The API key to use to query the chat completions endpoint.</span></span>
<span id="annotated-cell-5-55"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    sys_prompt: str</span></span>
<span id="annotated-cell-5-56"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The system prompt to help guide the model behaviour. By default,</span></span>
<span id="annotated-cell-5-57"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        the system prompt is set to _SYSTEM_MSG.</span></span>
<span id="annotated-cell-5-58"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    start_prompt: str</span></span>
<span id="annotated-cell-5-59"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The start prompt which will be presented to the user as the app</span></span>
<span id="annotated-cell-5-60"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        begins. By default, the start prompt is set to WELCOME_MSG.</span></span>
<span id="annotated-cell-5-61"></span>
<span id="annotated-cell-5-62"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Returns</span></span>
<span id="annotated-cell-5-63"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    -------</span></span>
<span id="annotated-cell-5-64"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    str</span></span>
<span id="annotated-cell-5-65"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        The response from the chat completions endpoint.</span></span>
<span id="annotated-cell-5-66"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-5-67"></span>
<span id="annotated-cell-5-68">    client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.OpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>api_key)</span>
<span id="annotated-cell-5-69">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># need to handle cases where queries go wrong.</span></span>
<span id="annotated-cell-5-70">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="annotated-cell-5-71">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> client.chat.completions.create(</span>
<span id="annotated-cell-5-72">            model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<span id="annotated-cell-5-73">            messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[</span>
<span id="annotated-cell-5-74">                {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: sys_prompt},</span>
<span id="annotated-cell-5-75">                {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: start_prompt},</span>
<span id="annotated-cell-5-76">                {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: prompt},</span>
<span id="annotated-cell-5-77">            ]</span>
<span id="annotated-cell-5-78">        )</span>
<span id="annotated-cell-5-79">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-5-80">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># in cases where the API key is invalid.</span></span>
<span id="annotated-cell-5-81">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> openai.AuthenticationError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="annotated-cell-5-82">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">raise</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">ValueError</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Is your API key valid?:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>e<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="annotated-cell-5-83"></span>
<span id="annotated-cell-5-84"></span>
<span id="annotated-cell-5-85"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny User Interface ----------------------------------------------------</span></span>
<span id="annotated-cell-5-86"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-5" data-target-annotation="1">1</button><span id="annotated-cell-5-87" class="code-annotation-target">app_ui <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.page_fillable(</span>
<span id="annotated-cell-5-88">    ui.panel_title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Choose Your Own Adventure: Jungle Quest!"</span>),</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-5" data-target-annotation="2">2</button><span id="annotated-cell-5-89" class="code-annotation-target">    ui.accordion(</span>
<span id="annotated-cell-5-90">    ui.accordion_panel(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 1: Your OpenAI API Key"</span>,</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-5" data-target-annotation="3">3</button><span id="annotated-cell-5-91" class="code-annotation-target">        ui.input_text(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"key_input"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter your openai api key"</span>),</span>
<span id="annotated-cell-5-92">    ), <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"acc"</span>, multiple<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span>),</span>
<span id="annotated-cell-5-93">    ui.chat_ui(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>),                                                  </span>
<span id="annotated-cell-5-94">    fillable_mobile<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="annotated-cell-5-95">)</span>
<span id="annotated-cell-5-96"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-5" data-target-annotation="4">4</button><span id="annotated-cell-5-97" class="code-annotation-target">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> App(app_ui, server<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>)</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-5" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-5" data-code-lines="87" data-code-annotation="1"><code>ui.page_fillable()</code> Works well with a chat component, increasing the height of your app to accommodate a growing chat log.</span>
</dd>
<dt data-target-cell="annotated-cell-5" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-5" data-code-lines="89" data-code-annotation="2">The <code>ui.accordion()</code> component will present a collapsible panel. This will be useful for the key input panel - we can minimise the key input once finished with it and focus on the chat.</span>
</dd>
<dt data-target-cell="annotated-cell-5" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-5" data-code-lines="91" data-code-annotation="3">In <code>shiny</code> UI elements, the first argument is usually the <code>id</code>. If you know CSS and HTML, then it’s the same <code>id</code> you’d target for styling an element. It’s really important in <code>shiny</code> as the server logic we’ll write later will communicate data to the UI via these <code>id</code> values. Make sure the <code>id</code> values are unique and do not include hyphens - use underscores instead.</span>
</dd>
<dt data-target-cell="annotated-cell-5" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-5" data-code-lines="97" data-code-annotation="4">In this final step, we need to combine our UI with server logic to make things work. As we haven’t written any server logic yet, we can just pass <code>None</code>. This means our UI won’t do anything in its current state.</span>
</dd>
</dl>
<p>Feel free to play around with the code and re-run the app using the play icon in the top-right corner of <code>app.py</code>. This interface uses the <a href="https://shinylive.io/py/examples/" target="_blank">shinylive service</a> which is useful for sharing simple <code>shiny</code> apps without any need for python installations.</p>
<iframe src="https://shinylive.io/py/editor/#code=NobwRAdghgtgpmAXGKAHVA6VBPMAaMAYwHsIAXOcpMAMwCdiYACAZwAsBLCbJjmVYnTJMAgujxMArhwA6EOWlQB9aUwC8UjligBzOEpocANkagAjI3AAUcpnc3aIcI0rIcylm2ADCbYsRY4JgBNYkk6JgB5AHcIUQATADdKMnC4RCYAKUkIHUsmAEVJOBYyAEIZMABKPFt7aQwoQhI6eI5SGzj6rSaWttIlVCgnIy8AZQpUJgBGDNDwqNRKEQBJUQAFNYBpOGxK2q77bowuVEkyVzgADzIrDni1SoBrXaVT8-2mUzNnR7AAUXIcAi2DCEWIS2gHCYaGhLz21QORxqvAelV6nxgkiMblQljUADEoEZAjU6nYGoQ2FALtI7miiNSyJUyYcmIYTOZLEoYMQzMY4GoACp0YoHKpyBTodSidBWRQqDgSQJ0ZJ0NQAOVIcAlEDAAF8ALpAA" class="iframey" style="overflow:hidden;margin:0;padding:0;width:100%;height:30rem">
</iframe>
<p>The process diagram for our app so far looks like this:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/03.png" class="lightbox" data-gallery="quarto-lightbox-gallery-3" title="Process diagram for iteration 3 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/03.png" class="img-fluid figure-img" alt="Process diagram for iteration 3 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 3 - Click to expand</figcaption>
</figure>
</div>
<p>Our logic for talking to the OpenAI model has not yet been coupled with our UI. We’ll fold that logic into our <code>shiny</code> server in the next iteration.</p>
</div>
<div id="tabset-1-4" class="tab-pane" aria-labelledby="tabset-1-4-tab">
<p>In this part, we’ll take the logic from the <code>query_openai()</code> function defined in iteration 1 and use it to build our <code>shiny</code> server. The <code>shiny</code> server is typically referenced as the “backend” to our app.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-6" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 4: Server logic allows us to create a chat log."""</span></span>
<span id="annotated-cell-6-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-6-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shiny <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> App, ui</span>
<span id="annotated-cell-6-4"></span>
<span id="annotated-cell-6-5">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-6-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-6-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-6-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-6-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-6-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-6-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-6-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-6-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-6-14"></span>
<span id="annotated-cell-6-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<span id="annotated-cell-6-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-6-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-6-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-6-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-6-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-6-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-6-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-6-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<span id="annotated-cell-6-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-6-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-6-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-6-27"></span>
<span id="annotated-cell-6-28">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-6-29"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-6-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-6-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-6-32"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-6-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-6-34"></span>
<span id="annotated-cell-6-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-6-36"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-6-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-6-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-6-39"></span>
<span id="annotated-cell-6-40"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># compose a message stream</span></span>
<span id="annotated-cell-6-41">_SYS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: _SYSTEM_MSG}</span>
<span id="annotated-cell-6-42">_WELCOME <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: WELCOME_MSG}</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-6" data-target-annotation="1">1</button><span id="annotated-cell-6-43" class="code-annotation-target">stream <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [_SYS, _WELCOME]</span>
<span id="annotated-cell-6-44"></span>
<span id="annotated-cell-6-45"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny User Interface ----------------------------------------------------</span></span>
<span id="annotated-cell-6-46"></span>
<span id="annotated-cell-6-47">app_ui <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.page_fillable(</span>
<span id="annotated-cell-6-48">    ui.panel_title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Choose Your Own Adventure: Jungle Quest!"</span>),</span>
<span id="annotated-cell-6-49">    ui.accordion(</span>
<span id="annotated-cell-6-50">    ui.accordion_panel(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 1: Your OpenAI API Key"</span>,</span>
<span id="annotated-cell-6-51">        ui.input_text(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"key_input"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter your openai api key"</span>),</span>
<span id="annotated-cell-6-52">    ), <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"acc"</span>, multiple<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span>),</span>
<span id="annotated-cell-6-53">    ui.chat_ui(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>),</span>
<span id="annotated-cell-6-54">    fillable_mobile<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="annotated-cell-6-55">)</span>
<span id="annotated-cell-6-56"></span>
<span id="annotated-cell-6-57"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny server logic ------------------------------------------------------</span></span>
<span id="annotated-cell-6-58"></span>
<span id="annotated-cell-6-59"></span>
<span id="annotated-cell-6-60"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> server(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>, output, session):</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-6" data-target-annotation="2">2</button><span id="annotated-cell-6-61" class="code-annotation-target">    chat <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.Chat(</span>
<span id="annotated-cell-6-62">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>, messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[ui.markdown(WELCOME_MSG)], tokenizer<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span></span>
<span id="annotated-cell-6-63">        )</span>
<span id="annotated-cell-6-64">    </span>
<span id="annotated-cell-6-65"></span>
<span id="annotated-cell-6-66">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Define a callback to run when the user submits a message</span></span>
<span id="annotated-cell-6-67">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@chat.on_user_submit</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-6" data-target-annotation="3">3</button><span id="annotated-cell-6-68" class="code-annotation-target">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> respond():</span>
<span id="annotated-cell-6-69">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Respond to the user's message."""</span></span>
<span id="annotated-cell-6-70">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Get the user's input</span></span>
<span id="annotated-cell-6-71">        user <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> chat.user_input()</span>
<span id="annotated-cell-6-72">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  update the stream list</span></span>
<span id="annotated-cell-6-73">        stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: user})                    </span>
<span id="annotated-cell-6-74">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Append a response to the chat</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-6" data-target-annotation="4">4</button><span id="annotated-cell-6-75" class="code-annotation-target">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input())</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-6" data-target-annotation="5">5</button><span id="annotated-cell-6-76" class="code-annotation-target">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.chat.completions.create(</span>
<span id="annotated-cell-6-77">            model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<span id="annotated-cell-6-78">            messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>stream,</span>
<span id="annotated-cell-6-79">            temperature<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.7</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># increase to make the model more creative</span></span>
<span id="annotated-cell-6-80">            )</span>
<span id="annotated-cell-6-81">        model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-6-82">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(model_response)</span>
<span id="annotated-cell-6-83">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  if the model indicates game over, end the game with a message.</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-6" data-target-annotation="6">6</button><span id="annotated-cell-6-84" class="code-annotation-target">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"the end..."</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> model_response.lower():</span>
<span id="annotated-cell-6-85">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(</span>
<span id="annotated-cell-6-86">                {</span>
<span id="annotated-cell-6-87">                    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-6-88">                    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Game Over! Refresh the page to play again."</span></span>
<span id="annotated-cell-6-89">                    })</span>
<span id="annotated-cell-6-90">            exit()</span>
<span id="annotated-cell-6-91">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-6-92">            stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: model_response}) </span>
<span id="annotated-cell-6-93"></span>
<span id="annotated-cell-6-94"></span>
<span id="annotated-cell-6-95">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> App(ui<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>app_ui, server<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>server)</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-6" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-6" data-code-lines="43" data-code-annotation="1">To keep a running log of what’s been said, we assign chat messages to a <code>stream</code> list. When the user and model respond, we’ll dump that content as a dictionary at the end of this list.</span>
</dd>
<dt data-target-cell="annotated-cell-6" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-6" data-code-lines="61,62,63" data-code-annotation="2">Here we create the backend to our <code>shiny</code> chat interface. It’s vital that it has the same <code>id</code> value as the <code>ui.chat_ui()</code> element defined in our UI in iteration 3. This connection will allow the backend and frontend to communicate when we run the app.</span>
</dd>
<dt data-target-cell="annotated-cell-6" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-6" data-code-lines="68" data-code-annotation="3">We use async here because it improves responsiveness, especially when dealing with potentially slow network requests to the OpenAI API.</span>
</dd>
<dt data-target-cell="annotated-cell-6" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-6" data-code-lines="75" data-code-annotation="4">We’ve now switched over to the OpenAI Async client. This is better for working with event driven apps like this one. We no longer hard-code an API key. Instead, we’ll take the value from the <code>ui.input_text()</code> field. Notice that we reference the <code>id</code> value that we set when we defined the UI like a method call: <code>input.key_input()</code>. This is how <code>shiny</code> apps wire the frontend and backend together.</span>
</dd>
<dt data-target-cell="annotated-cell-6" data-target-annotation="5">5</dt>
<dd>
<span data-code-cell="annotated-cell-6" data-code-lines="76,82" data-code-annotation="5">We need to use <code>await</code> in parts of our server logic. This is because the model response typically takes some time to arrive and parts of the server would error until they receive it. Typically, anything that would raise an exception rather than returning <code>None</code> you’ll need to <code>await</code> in a <code>shiny</code> app.</span>
</dd>
<dt data-target-cell="annotated-cell-6" data-target-annotation="6">6</dt>
<dd>
<span data-code-cell="annotated-cell-6" data-code-lines="84,85,86,87,88,89,90" data-code-annotation="6">This part of our server is where we end the game. Notice that this is dependent on our model following rule 6 in <code>_SYSTEM_MSG</code>. Not all models are great at following that instruction. At the time of writing this blog, I’ve tested <code>gpt-3.5-turbo</code>, <code>gpt-4o</code> and <code>chatgpt-4o-latest</code>. In my testing I found <code>gpt-3.5-turbo</code> to be the most reliable at following this rule. But when I tested streaming the model responses, no models would end the game as requested. This is definitely the most flaky element of this app and is part of the fun of working with these models.</span>
</dd>
</dl>
<p>Updating the server logic for our process diagram, note that I have added a key that illustrates wherever we instantiate an openai client in the app. As we continue to build, handling the client becomes a bit involved so let’s start paying attention to wherever we’re using it:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/04.png" class="lightbox" data-gallery="quarto-lightbox-gallery-4" title="Process diagram for iteration 4 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/04.png" class="img-fluid figure-img" alt="Process diagram for iteration 4 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 4 - Click to expand</figcaption>
</figure>
</div>
<p>You can see that the <code>respond()</code> function has become the busiest unit in the app. It takes the api key value from our UI, combines the user’s messages sent from the chat UI, adds these to the chat stream and communicates with the OpenAI model. Now these components are all wired up we get a running application. If you’ve made it this far - well done!</p>
<p>You can see me interacting with the resultant app in the video below. Note that I cannot use shinylive to host a working version of this iteration as unfortunately the <code>openai</code> package is not available on that service.</p>
<div class="callout callout-style-default callout-important callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Important
</div>
</div>
<div class="callout-body-container callout-body">
<p>I use a real OpenAI API key in these clips to demonstrate the application. Note that I have since revoked this key and it will no longer work. You should not share your secret keys with anyone.</p>
</div>
</div>
<iframe title="Iteration 4 recording" src="https://player.vimeo.com/video/1010110793?h=133c2b081d" class="iframey" allowfullscreen="" style="overflow:hidden;margin:0;padding:0;width:100%;height:24rem;">
</iframe>
<p>Notice that I attempt to use a nonsense key value and we get a nasty-looking error. Luckily, it’s not a fatal one - the app doesn’t crash. But we should think about handling cases where the key is bad and give a more accessible notification to the user instead.</p>
</div>
<div id="tabset-1-5" class="tab-pane" aria-labelledby="tabset-1-5-tab">
<p>In this version, we build on our working app to improve the user experience. We’ll add a ‘submit’ button, which the user can use when they’re ready to use their key. We also provide notifications to the user when they submit their key, but we won’t be checking whether the key is valid until the next iteration.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-7" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-7-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 5: Submit button &amp; notifications for the user."""</span></span>
<span id="annotated-cell-7-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-7-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shiny <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> App, reactive, ui</span>
<span id="annotated-cell-7-4"></span>
<span id="annotated-cell-7-5">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-7-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-7-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-7-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-7-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-7-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-7-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-7-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-7-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-7-14"></span>
<span id="annotated-cell-7-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<span id="annotated-cell-7-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-7-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-7-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-7-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-7-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-7-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-7-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-7-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<span id="annotated-cell-7-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-7-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-7-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-7-27"></span>
<span id="annotated-cell-7-28">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-7-29"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-7-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-7-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-7-32"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-7-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-7-34"></span>
<span id="annotated-cell-7-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-7-36"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-7-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-7-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-7-39"></span>
<span id="annotated-cell-7-40"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># compose a message stream</span></span>
<span id="annotated-cell-7-41">_SYS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: _SYSTEM_MSG}</span>
<span id="annotated-cell-7-42">_WELCOME <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: WELCOME_MSG}</span>
<span id="annotated-cell-7-43">stream <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [_SYS, _WELCOME]</span>
<span id="annotated-cell-7-44"></span>
<span id="annotated-cell-7-45"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny User Interface ----------------------------------------------------</span></span>
<span id="annotated-cell-7-46"></span>
<span id="annotated-cell-7-47"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-7" data-target-annotation="1">1</button><span id="annotated-cell-7-48" class="code-annotation-target"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> input_text_with_button(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span>, label, button_label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>):</span>
<span id="annotated-cell-7-49">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-7-50"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    An interface component combining an input text widget with an action</span></span>
<span id="annotated-cell-7-51"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    button. IDs for the text field and button can be accessed as &lt;id&gt;_text</span></span>
<span id="annotated-cell-7-52"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    and &lt;id&gt;_btn respectively.</span></span>
<span id="annotated-cell-7-53"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-7-54">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> ui.div(</span>
<span id="annotated-cell-7-55">        ui.input_text(</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-7" data-target-annotation="2">2</button><span id="annotated-cell-7-56" class="code-annotation-target">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_text"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>placeholder),</span>
<span id="annotated-cell-7-57">        ui.input_action_button(</span>
<span id="annotated-cell-7-58">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_btn"</span>,</span>
<span id="annotated-cell-7-59">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>button_label,</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-7" data-target-annotation="3">3</button><span id="annotated-cell-7-60" class="code-annotation-target">            style<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"margin-top:28px;margin-bottom:16px;color:#04bb8c;border-color:#04bb8c;"</span></span>
<span id="annotated-cell-7-61">            ),</span>
<span id="annotated-cell-7-62">        class_<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d-flex gap-2"</span></span>
<span id="annotated-cell-7-63">    )</span>
<span id="annotated-cell-7-64"></span>
<span id="annotated-cell-7-65"></span>
<span id="annotated-cell-7-66">app_ui <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.page_fillable(</span>
<span id="annotated-cell-7-67">    ui.panel_title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Choose Your Own Adventure: Jungle Quest!"</span>),</span>
<span id="annotated-cell-7-68">    ui.accordion(</span>
<span id="annotated-cell-7-69">    ui.accordion_panel(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 1: Your OpenAI API Key"</span>,</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-7" data-target-annotation="4">4</button><span id="annotated-cell-7-70" class="code-annotation-target">        input_text_with_button(</span>
<span id="annotated-cell-7-71">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"key_input"</span>,</span>
<span id="annotated-cell-7-72">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter your OpenAI API key"</span>,</span>
<span id="annotated-cell-7-73">            button_label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Submit"</span>,</span>
<span id="annotated-cell-7-74">            placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter key here"</span></span>
<span id="annotated-cell-7-75">            )), <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"acc"</span>, multiple<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span>),</span>
<span id="annotated-cell-7-76">ui.h6(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 2: Choose your adventure"</span>),</span>
<span id="annotated-cell-7-77">    ui.chat_ui(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>),</span>
<span id="annotated-cell-7-78">    fillable_mobile<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="annotated-cell-7-79">)</span>
<span id="annotated-cell-7-80"></span>
<span id="annotated-cell-7-81"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny server logic ------------------------------------------------------</span></span>
<span id="annotated-cell-7-82"></span>
<span id="annotated-cell-7-83"></span>
<span id="annotated-cell-7-84"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> server(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>, output, session):</span>
<span id="annotated-cell-7-85">    chat <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.Chat(</span>
<span id="annotated-cell-7-86">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>, messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[ui.markdown(WELCOME_MSG)], tokenizer<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span></span>
<span id="annotated-cell-7-87">        )</span>
<span id="annotated-cell-7-88"></span>
<span id="annotated-cell-7-89"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-7" data-target-annotation="5">5</button><span id="annotated-cell-7-90" class="code-annotation-target">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.Effect</span></span>
<span id="annotated-cell-7-91">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.event</span>(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_btn)</span>
<span id="annotated-cell-7-92">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> handle_api_key_submit():</span>
<span id="annotated-cell-7-93">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Update the UI with a notification when user submits key."""</span></span>
<span id="annotated-cell-7-94">        api_key <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text()</span>
<span id="annotated-cell-7-95">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> api_key:</span>
<span id="annotated-cell-7-96">            ui.notification_show(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"API key submitted: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>api_key[:<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">..."</span>)</span>
<span id="annotated-cell-7-97">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-7-98">            ui.notification_show(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Please enter an API key"</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"warning"</span>)</span>
<span id="annotated-cell-7-99"></span>
<span id="annotated-cell-7-100"></span>
<span id="annotated-cell-7-101">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Define a callback to run when the user submits a message</span></span>
<span id="annotated-cell-7-102">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@chat.on_user_submit</span></span>
<span id="annotated-cell-7-103">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> respond():</span>
<span id="annotated-cell-7-104">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Respond to the user's message."""</span></span>
<span id="annotated-cell-7-105">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Get the user's input</span></span>
<span id="annotated-cell-7-106">        user <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> chat.user_input()</span>
<span id="annotated-cell-7-107">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  update the stream list</span></span>
<span id="annotated-cell-7-108">        stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: user})</span>
<span id="annotated-cell-7-109">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Append a response to the chat</span></span>
<span id="annotated-cell-7-110">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text())</span>
<span id="annotated-cell-7-111">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.chat.completions.create(</span>
<span id="annotated-cell-7-112">            model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<span id="annotated-cell-7-113">            messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>stream,</span>
<span id="annotated-cell-7-114">            temperature<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.7</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># increase to make the model more creative</span></span>
<span id="annotated-cell-7-115">            )</span>
<span id="annotated-cell-7-116">        model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-7-117">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(model_response)</span>
<span id="annotated-cell-7-118">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  if the model indicates game over, end the game with a message.</span></span>
<span id="annotated-cell-7-119">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"the end..."</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> model_response.lower():</span>
<span id="annotated-cell-7-120">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(</span>
<span id="annotated-cell-7-121">                {</span>
<span id="annotated-cell-7-122">                    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-7-123">                    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Game Over! Refresh the page to play again."</span></span>
<span id="annotated-cell-7-124">                    })</span>
<span id="annotated-cell-7-125">            exit()</span>
<span id="annotated-cell-7-126">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-7-127">            stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: model_response})</span>
<span id="annotated-cell-7-128"></span>
<span id="annotated-cell-7-129"></span>
<span id="annotated-cell-7-130">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> App(ui<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>app_ui, server<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>server)</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-7" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-7" data-code-lines="48" data-code-annotation="1">Here we define a new function that’s used to conveniently return a text field and an action button together. This is what we’ll use for the key input going forward.</span>
</dd>
<dt data-target-cell="annotated-cell-7" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-7" data-code-lines="56,57,58" data-code-annotation="2">Notice that the <code>id</code> value that we’ll pass into <code>input_text_with_button()</code> will return separate unique <code>id</code> values for the text field and action button.</span>
</dd>
<dt data-target-cell="annotated-cell-7" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-7" data-code-lines="60" data-code-annotation="3">Feel free to experiment with styling any elements with CSS. If you’d rather not have inline CSS, you can move the styling into a dedicated CSS file and apply it with classes or IDs to the elements in your UI.</span>
</dd>
<dt data-target-cell="annotated-cell-7" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-7" data-code-lines="70,71,72,73,74" data-code-annotation="4">We replace the text field of previous iterations with our new <code>input_text_with_button()</code> text field &amp; action button combo.</span>
</dd>
<dt data-target-cell="annotated-cell-7" data-target-annotation="5">5</dt>
<dd>
<span data-code-cell="annotated-cell-7" data-code-lines="90,91,92,93,94,95,96,97,98" data-code-annotation="5">In the server, we define a new function that will run whenever the action button with <code>id=input.key_input_btn</code> is clicked by the user. If there’s a value that’s been submitted we’re going to place a confirmation notification on the UI. If not, we’ll raise a warning to the user to remind them to submit their key.</span>
</dd>
</dl>
<p>I’ve added a marker in the process diagram to remind us that <code>handle_api_key_submit()</code> will run as a reactive event when the key is submitted. This function updates the UI with feedback notifications.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/05.png" class="lightbox" data-gallery="quarto-lightbox-gallery-5" title="Process diagram for iteration 5 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/05.png" class="img-fluid figure-img" alt="Process diagram for iteration 5 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 5 - Click to expand</figcaption>
</figure>
</div>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Tip
</div>
</div>
<div class="callout-body-container callout-body">
<p>Using <code>ui_notification_show()</code> is a really useful method for debugging reactive values when your app breaks. Use it to check on intermediate values that the frontend receives from the backend.</p>
</div>
</div>
<div class="callout callout-style-default callout-important callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Important
</div>
</div>
<div class="callout-body-container callout-body">
<p>I use a real OpenAI API key in these clips to demonstrate the application working. Note that I have since revoked this key and it will no longer work. You should not share your secret keys with anyone.</p>
</div>
</div>
<p>In the recording below, you can see the new action button and the UI notifications being returned by the server. However, note that I still get that nasty red error when I pass an invalid key. In the next iteration, we’ll determine whether the user has used a valid key.</p>
<iframe title="Iteration 5 recording" src="https://player.vimeo.com/video/1010111488?h=f74e808099" class="iframey" allowfullscreen="" style="overflow:hidden;margin:0;padding:0;width:100%;height:24rem;">
</iframe>
</div>
<div id="tabset-1-6" class="tab-pane" aria-labelledby="tabset-1-6-tab">
<p>In this version of the app we will introduce more backend logic that will check whether the key the user has submitted is a valid one.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-8" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-8-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 6: Check the key is valid."""</span></span>
<span id="annotated-cell-8-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-8-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shiny <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> App, reactive, ui</span>
<span id="annotated-cell-8-4"></span>
<span id="annotated-cell-8-5">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-8-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-8-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-8-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-8-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-8-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-8-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-8-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-8-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-8-14"></span>
<span id="annotated-cell-8-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<span id="annotated-cell-8-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-8-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-8-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-8-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-8-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-8-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-8-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-8-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<span id="annotated-cell-8-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-8-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-8-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-8-27"></span>
<span id="annotated-cell-8-28">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-8-29"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-8-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-8-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-8-32"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-8-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-8-34"></span>
<span id="annotated-cell-8-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-8-36"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-8-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-8-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-8-39"></span>
<span id="annotated-cell-8-40"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># compose a message stream</span></span>
<span id="annotated-cell-8-41">_SYS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: _SYSTEM_MSG}</span>
<span id="annotated-cell-8-42">_WELCOME <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: WELCOME_MSG}</span>
<span id="annotated-cell-8-43">stream <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [_SYS, _WELCOME]</span>
<span id="annotated-cell-8-44"></span>
<span id="annotated-cell-8-45"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny User Interface ----------------------------------------------------</span></span>
<span id="annotated-cell-8-46"></span>
<span id="annotated-cell-8-47"></span>
<span id="annotated-cell-8-48"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> input_text_with_button(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span>, label, button_label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>):</span>
<span id="annotated-cell-8-49">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-8-50"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    An interface component combining an input text widget with an action</span></span>
<span id="annotated-cell-8-51"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    button. IDs for the text field and button can be accessed as &lt;id&gt;_text</span></span>
<span id="annotated-cell-8-52"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    and &lt;id&gt;_btn respectively.</span></span>
<span id="annotated-cell-8-53"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-8-54">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> ui.div(</span>
<span id="annotated-cell-8-55">        ui.input_text(</span>
<span id="annotated-cell-8-56">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_text"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>placeholder),</span>
<span id="annotated-cell-8-57">        ui.input_action_button(</span>
<span id="annotated-cell-8-58">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_btn"</span>,</span>
<span id="annotated-cell-8-59">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>button_label,</span>
<span id="annotated-cell-8-60">            style<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"margin-top:28px;margin-bottom:16px;color:#04bb8c;border-color:#04bb8c;"</span></span>
<span id="annotated-cell-8-61">            ),</span>
<span id="annotated-cell-8-62">        class_<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d-flex gap-2"</span></span>
<span id="annotated-cell-8-63">    )</span>
<span id="annotated-cell-8-64"></span>
<span id="annotated-cell-8-65"></span>
<span id="annotated-cell-8-66">app_ui <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.page_fillable(</span>
<span id="annotated-cell-8-67">    ui.panel_title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Choose Your Own Adventure: Jungle Quest!"</span>),</span>
<span id="annotated-cell-8-68">    ui.accordion(</span>
<span id="annotated-cell-8-69">    ui.accordion_panel(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 1: Your OpenAI API Key"</span>,</span>
<span id="annotated-cell-8-70">        input_text_with_button(</span>
<span id="annotated-cell-8-71">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"key_input"</span>,</span>
<span id="annotated-cell-8-72">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter your OpenAI API key"</span>,</span>
<span id="annotated-cell-8-73">            button_label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Submit"</span>,</span>
<span id="annotated-cell-8-74">            placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter key here"</span></span>
<span id="annotated-cell-8-75">            )), <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"acc"</span>, multiple<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span>),</span>
<span id="annotated-cell-8-76">ui.h6(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 2: Choose your adventure"</span>),</span>
<span id="annotated-cell-8-77">    ui.chat_ui(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>),</span>
<span id="annotated-cell-8-78">    fillable_mobile<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="annotated-cell-8-79">)</span>
<span id="annotated-cell-8-80"></span>
<span id="annotated-cell-8-81"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny server logic ------------------------------------------------------</span></span>
<span id="annotated-cell-8-82"></span>
<span id="annotated-cell-8-83"></span>
<span id="annotated-cell-8-84"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> server(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>, output, session):</span>
<span id="annotated-cell-8-85">    chat <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.Chat(</span>
<span id="annotated-cell-8-86">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>, messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[ui.markdown(WELCOME_MSG)], tokenizer<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span></span>
<span id="annotated-cell-8-87">        )</span>
<span id="annotated-cell-8-88"></span>
<span id="annotated-cell-8-89">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.Effect</span></span>
<span id="annotated-cell-8-90">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.event</span>(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_btn)</span>
<span id="annotated-cell-8-91">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> handle_api_key_submit():</span>
<span id="annotated-cell-8-92">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Update the UI with a notification when user submits key.</span></span>
<span id="annotated-cell-8-93"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        </span></span>
<span id="annotated-cell-8-94"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Checks the validity of the API key by querying the models list</span></span>
<span id="annotated-cell-8-95"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        endpoint."""</span></span>
<span id="annotated-cell-8-96">        api_key <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text()</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-8" data-target-annotation="1">1</button><span id="annotated-cell-8-97" class="code-annotation-target">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>api_key)</span>
<span id="annotated-cell-8-98">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-8" data-target-annotation="2">2</button><span id="annotated-cell-8-99" class="code-annotation-target">            resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.models.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">list</span>()</span>
<span id="annotated-cell-8-100">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> resp:</span>
<span id="annotated-cell-8-101">                ui.notification_show(</span>
<span id="annotated-cell-8-102">                    <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"API key validated: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>api_key[:<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">..."</span>)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-8" data-target-annotation="3">3</button><span id="annotated-cell-8-103" class="code-annotation-target">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> openai.AuthenticationError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="annotated-cell-8-104">            ui.notification_show(</span>
<span id="annotated-cell-8-105">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Bad key provided. Please try again."</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"warning"</span>)</span>
<span id="annotated-cell-8-106"></span>
<span id="annotated-cell-8-107"></span>
<span id="annotated-cell-8-108">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Define a callback to run when the user submits a message</span></span>
<span id="annotated-cell-8-109">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@chat.on_user_submit</span></span>
<span id="annotated-cell-8-110">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> respond():</span>
<span id="annotated-cell-8-111">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Respond to the user's message."""</span></span>
<span id="annotated-cell-8-112">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Get the user's input</span></span>
<span id="annotated-cell-8-113">        user <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> chat.user_input()</span>
<span id="annotated-cell-8-114">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  update the stream list</span></span>
<span id="annotated-cell-8-115">        stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: user})</span>
<span id="annotated-cell-8-116">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Append a response to the chat</span></span>
<span id="annotated-cell-8-117">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text())</span>
<span id="annotated-cell-8-118">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.chat.completions.create(</span>
<span id="annotated-cell-8-119">            model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<span id="annotated-cell-8-120">            messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>stream,</span>
<span id="annotated-cell-8-121">            temperature<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.7</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># increase to make the model more creative</span></span>
<span id="annotated-cell-8-122">            )</span>
<span id="annotated-cell-8-123">        model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-8-124">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(model_response)</span>
<span id="annotated-cell-8-125">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  if the model indicates game over, end the game with a message.</span></span>
<span id="annotated-cell-8-126">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"the end..."</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> model_response.lower():</span>
<span id="annotated-cell-8-127">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(</span>
<span id="annotated-cell-8-128">                {</span>
<span id="annotated-cell-8-129">                    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-8-130">                    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Game Over! Refresh the page to play again."</span></span>
<span id="annotated-cell-8-131">                    })</span>
<span id="annotated-cell-8-132">            exit()</span>
<span id="annotated-cell-8-133">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-8-134">            stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: model_response})</span>
<span id="annotated-cell-8-135"></span>
<span id="annotated-cell-8-136"></span>
<span id="annotated-cell-8-137">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> App(ui<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>app_ui, server<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>server)</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-8" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-8" data-code-lines="97" data-code-annotation="1">Notice that we are now creating another openai client. This is so that we can test the key to see whether it successfully queries the openai service, before using it to play the game. Initiating multiple clients is not needed and is inefficient design. In a later iteration we’ll come back to this.</span>
</dd>
<dt data-target-cell="annotated-cell-8" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-8" data-code-lines="99" data-code-annotation="2">This time, we’ll query the <code>models.list()</code> endpoint to get a list of models available. There is currently no OpenAI endpoint for explicitly checking whether a service is valid. OpenAI API support team responded to my support ticket to suggest this as the best method for ensuring a key is valid.</span>
</dd>
<dt data-target-cell="annotated-cell-8" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-8" data-code-lines="103" data-code-annotation="3">In cases where a bad key is provided, we can avoid the <code>openai.AuthenticationError</code> and print a warning to the UI instead.</span>
</dd>
</dl>
<p>In our increasingly complex process diagram, I’ve emphasised that <code>handle_api_key_submit()</code> now instantiates another client object and uses it to query the <code>models.list</code> endpoint of the OpenAI service.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/06.png" class="lightbox" data-gallery="quarto-lightbox-gallery-6" title="Process diagram for iteration 6 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/06.png" class="img-fluid figure-img" alt="Process diagram for iteration 6 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 6 - Click to expand</figcaption>
</figure>
</div>
<div class="callout callout-style-default callout-important callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Important
</div>
</div>
<div class="callout-body-container callout-body">
<p>I use a real OpenAI API key in these clips to demonstrate the application working. Note that I have since revoked this key and it will no longer work. You should not share your secret keys with anyone.</p>
</div>
</div>
<p>In the recording below I demonstrate how this version of the app will return a warning to the user if the submitted key was not valid.</p>
<iframe title="Iteration 6 recording" src="https://player.vimeo.com/video/1010111977?h=b625a6f9ef" class="iframey" allowfullscreen="" style="overflow:hidden;margin:0;padding:0;width:100%;height:24rem;">
</iframe>
</div>
<div id="tabset-1-7" class="tab-pane" aria-labelledby="tabset-1-7-tab">
<p>Now we introduce a moderation feature that will check that the prompts being passed from the user to the OpenAI service comply with the service’s usage policies. Forewarned - this is far from perfect!</p>
<p>In general, it’s pretty good but if you’re intentionally trying to test it like I did when implementing this feature, you can find some funny quirks. British expletives tend to sail through unchallenged and at times my test prompts were raised as violations for stating things like “Let’s fight!” (category: harassment) when that was one of the options provided to me by the model! It’s likely that passing greater context to the moderations endpoint (such as more of the message stream) may be able to overcome this, though that has not been implemented for this tutorial.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-9" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-9-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 7: Implement prompt moderation."""</span></span>
<span id="annotated-cell-9-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-9-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shiny <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> App, reactive, ui</span>
<span id="annotated-cell-9-4"></span>
<span id="annotated-cell-9-5">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-9-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-9-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-9-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-9-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-9-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-9-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-9-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-9-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-9-14"></span>
<span id="annotated-cell-9-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<span id="annotated-cell-9-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-9-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-9-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-9-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-9-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-9-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-9-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-9-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<span id="annotated-cell-9-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-9-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-9-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-9-27"></span>
<span id="annotated-cell-9-28">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-9-29"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-9-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-9-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-9-32"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-9-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-9-34"></span>
<span id="annotated-cell-9-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-9-36"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-9-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-9-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-9-39"></span>
<span id="annotated-cell-9-40"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># compose a message stream</span></span>
<span id="annotated-cell-9-41">_SYS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: _SYSTEM_MSG}</span>
<span id="annotated-cell-9-42">_WELCOME <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: WELCOME_MSG}</span>
<span id="annotated-cell-9-43">stream <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [_SYS, _WELCOME]</span>
<span id="annotated-cell-9-44"></span>
<span id="annotated-cell-9-45"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny User Interface ----------------------------------------------------</span></span>
<span id="annotated-cell-9-46"></span>
<span id="annotated-cell-9-47"></span>
<span id="annotated-cell-9-48"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> input_text_with_button(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span>, label, button_label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>):</span>
<span id="annotated-cell-9-49">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-9-50"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    An interface component combining an input text widget with an action</span></span>
<span id="annotated-cell-9-51"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    button. IDs for the text field and button can be accessed as &lt;id&gt;_text</span></span>
<span id="annotated-cell-9-52"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    and &lt;id&gt;_btn respectively.</span></span>
<span id="annotated-cell-9-53"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-9-54">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> ui.div(</span>
<span id="annotated-cell-9-55">        ui.input_text(</span>
<span id="annotated-cell-9-56">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_text"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>placeholder),</span>
<span id="annotated-cell-9-57">        ui.input_action_button(</span>
<span id="annotated-cell-9-58">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_btn"</span>,</span>
<span id="annotated-cell-9-59">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>button_label,</span>
<span id="annotated-cell-9-60">            style<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"margin-top:28px;margin-bottom:16px;color:#04bb8c;border-color:#04bb8c;"</span></span>
<span id="annotated-cell-9-61">            ),</span>
<span id="annotated-cell-9-62">        class_<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d-flex gap-2"</span></span>
<span id="annotated-cell-9-63">    )</span>
<span id="annotated-cell-9-64"></span>
<span id="annotated-cell-9-65"></span>
<span id="annotated-cell-9-66">app_ui <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.page_fillable(</span>
<span id="annotated-cell-9-67">    ui.panel_title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Choose Your Own Adventure: Jungle Quest!"</span>),</span>
<span id="annotated-cell-9-68">    ui.accordion(</span>
<span id="annotated-cell-9-69">    ui.accordion_panel(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 1: Your OpenAI API Key"</span>,</span>
<span id="annotated-cell-9-70">        input_text_with_button(</span>
<span id="annotated-cell-9-71">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"key_input"</span>,</span>
<span id="annotated-cell-9-72">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter your OpenAI API key"</span>,</span>
<span id="annotated-cell-9-73">            button_label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Submit"</span>,</span>
<span id="annotated-cell-9-74">            placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter key here"</span></span>
<span id="annotated-cell-9-75">            )), <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"acc"</span>, multiple<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span>),</span>
<span id="annotated-cell-9-76">ui.h6(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 2: Choose your adventure"</span>),</span>
<span id="annotated-cell-9-77">    ui.chat_ui(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>),</span>
<span id="annotated-cell-9-78">    fillable_mobile<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="annotated-cell-9-79">)</span>
<span id="annotated-cell-9-80"></span>
<span id="annotated-cell-9-81"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny server logic ------------------------------------------------------</span></span>
<span id="annotated-cell-9-82"></span>
<span id="annotated-cell-9-83"></span>
<span id="annotated-cell-9-84"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> server(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>, output, session):</span>
<span id="annotated-cell-9-85">    chat <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.Chat(</span>
<span id="annotated-cell-9-86">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>, messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[ui.markdown(WELCOME_MSG)], tokenizer<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span></span>
<span id="annotated-cell-9-87">        )</span>
<span id="annotated-cell-9-88"></span>
<span id="annotated-cell-9-89"></span>
<span id="annotated-cell-9-90">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.Effect</span></span>
<span id="annotated-cell-9-91">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.event</span>(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_btn)</span>
<span id="annotated-cell-9-92">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> handle_api_key_submit():</span>
<span id="annotated-cell-9-93">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Update the UI with a notification when user submits key.</span></span>
<span id="annotated-cell-9-94"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        </span></span>
<span id="annotated-cell-9-95"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Checks the validity of the API key by querying the models list</span></span>
<span id="annotated-cell-9-96"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        endpoint."""</span></span>
<span id="annotated-cell-9-97">        api_key <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text()</span>
<span id="annotated-cell-9-98">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>api_key)</span>
<span id="annotated-cell-9-99">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="annotated-cell-9-100">            resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.models.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">list</span>()</span>
<span id="annotated-cell-9-101">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> resp:</span>
<span id="annotated-cell-9-102">                ui.notification_show(</span>
<span id="annotated-cell-9-103">                    <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"API key validated: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>api_key[:<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">..."</span>)</span>
<span id="annotated-cell-9-104">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> openai.AuthenticationError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="annotated-cell-9-105">            ui.notification_show(</span>
<span id="annotated-cell-9-106">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Bad key provided. Please try again."</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"warning"</span>)</span>
<span id="annotated-cell-9-107">    </span>
<span id="annotated-cell-9-108"></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-9" data-target-annotation="1">1</button><span id="annotated-cell-9-109" class="code-annotation-target">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> check_moderation(prompt:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="annotated-cell-9-110">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Check if prompt is flagged by OpenAI's moderation endpoint.</span></span>
<span id="annotated-cell-9-111"></span>
<span id="annotated-cell-9-112"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Parameters</span></span>
<span id="annotated-cell-9-113"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        ----------</span></span>
<span id="annotated-cell-9-114"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        prompt : str</span></span>
<span id="annotated-cell-9-115"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            The user's prompt to check.</span></span>
<span id="annotated-cell-9-116"></span>
<span id="annotated-cell-9-117"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Returns</span></span>
<span id="annotated-cell-9-118"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        -------</span></span>
<span id="annotated-cell-9-119"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        str</span></span>
<span id="annotated-cell-9-120"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            The category violations if flagged, otherwise "good prompt".</span></span>
<span id="annotated-cell-9-121"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        """</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-9" data-target-annotation="2">2</button><span id="annotated-cell-9-122" class="code-annotation-target">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text())</span>
<span id="annotated-cell-9-123">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.moderations.create(</span>
<span id="annotated-cell-9-124">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>prompt)</span>
<span id="annotated-cell-9-125">        content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.results[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].to_dict()</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-9" data-target-annotation="3">3</button><span id="annotated-cell-9-126" class="code-annotation-target">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> content[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"flagged"</span>]:</span>
<span id="annotated-cell-9-127">            infringements <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> []</span>
<span id="annotated-cell-9-128">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> key, val <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> content[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"categories"</span>].items():</span>
<span id="annotated-cell-9-129">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> val:</span>
<span id="annotated-cell-9-130">                    infringements.append(key)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-9" data-target-annotation="4">4</button><span id="annotated-cell-9-131" class="code-annotation-target">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" &amp; "</span>.join(infringements)</span>
<span id="annotated-cell-9-132">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-9" data-target-annotation="5">5</button><span id="annotated-cell-9-133" class="code-annotation-target">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"good prompt"</span></span>
<span id="annotated-cell-9-134"></span>
<span id="annotated-cell-9-135"></span>
<span id="annotated-cell-9-136">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Define a callback to run when the user submits a message</span></span>
<span id="annotated-cell-9-137">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@chat.on_user_submit</span></span>
<span id="annotated-cell-9-138">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> respond():</span>
<span id="annotated-cell-9-139">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Respond to the user's message."""</span></span>
<span id="annotated-cell-9-140">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Get the user's input</span></span>
<span id="annotated-cell-9-141">        usr_prompt <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> chat.user_input()</span>
<span id="annotated-cell-9-142"></span>
<span id="annotated-cell-9-143">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Check moderations endpoint incase openai policies are violated</span></span>
<span id="annotated-cell-9-144">        flag_check <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> check_moderation(prompt<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>usr_prompt)</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-9" data-target-annotation="6">6</button><span id="annotated-cell-9-145" class="code-annotation-target">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> flag_check <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"good prompt"</span>:</span>
<span id="annotated-cell-9-146">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message({</span>
<span id="annotated-cell-9-147">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-9-148">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Your message may violate OpenAI's usage policy, categories: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>flag_check<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">. Please rephrase your input and try again."</span></span>
<span id="annotated-cell-9-149">            })</span>
<span id="annotated-cell-9-150">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-9-151">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  update the stream list</span></span>
<span id="annotated-cell-9-152">            stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: usr_prompt})</span>
<span id="annotated-cell-9-153">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Append a response to the chat</span></span>
<span id="annotated-cell-9-154">            client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text())</span>
<span id="annotated-cell-9-155">            response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.chat.completions.create(</span>
<span id="annotated-cell-9-156">                model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<span id="annotated-cell-9-157">                messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>stream,</span>
<span id="annotated-cell-9-158">                temperature<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.7</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># increase to make the model more creative</span></span>
<span id="annotated-cell-9-159">                )</span>
<span id="annotated-cell-9-160">            model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-9-161">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(model_response)</span>
<span id="annotated-cell-9-162">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  if the model indicates game over, end game with a message.</span></span>
<span id="annotated-cell-9-163">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"the end..."</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> model_response.lower():</span>
<span id="annotated-cell-9-164">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(</span>
<span id="annotated-cell-9-165">                    {</span>
<span id="annotated-cell-9-166">                        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-9-167">                        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Game Over! Refresh to play again."</span></span>
<span id="annotated-cell-9-168">                        })</span>
<span id="annotated-cell-9-169">                exit()</span>
<span id="annotated-cell-9-170">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-9-171">                stream.append(</span>
<span id="annotated-cell-9-172">                    {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: model_response})</span>
<span id="annotated-cell-9-173"></span>
<span id="annotated-cell-9-174"></span>
<span id="annotated-cell-9-175">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> App(ui<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>app_ui, server<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>server)</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-9" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-9" data-code-lines="109" data-code-annotation="1">We define a new function that will query the OpenAI moderations endpoint.</span>
</dd>
<dt data-target-cell="annotated-cell-9" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-9" data-code-lines="122" data-code-annotation="2">Notice that we create a third openai client in order to handle the communication with the service. In the next iteration we will refactor this.</span>
</dd>
<dt data-target-cell="annotated-cell-9" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-9" data-code-lines="126" data-code-annotation="3">If the prompt has violated any category, then the value of the <code>flagged</code> key will be <code>True</code>.</span>
</dd>
<dt data-target-cell="annotated-cell-9" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-9" data-code-lines="131" data-code-annotation="4">In cases where multiple categories are violated, we will include each breached category in a message to the user.</span>
</dd>
<dt data-target-cell="annotated-cell-9" data-target-annotation="5">5</dt>
<dd>
<span data-code-cell="annotated-cell-9" data-code-lines="133" data-code-annotation="5">The return value in the case of a prompt that passes the moderation check.</span>
</dd>
<dt data-target-cell="annotated-cell-9" data-target-annotation="6">6</dt>
<dd>
<span data-code-cell="annotated-cell-9" data-code-lines="145,146,147,148,149" data-code-annotation="6">We implement some control flow to query the <code>chat.completions</code> endpoint only if the moderations check passes.</span>
</dd>
</dl>
<p>The updated process diagram clarifies the reactive flow of the app. Now prompts from the user pass through <code>check_moderations()</code> first. The outcome of check moderations is then passed along within the <code>respond()</code> function, determining whether the prompt would be passed along to the <code>chat.completions</code> endpoint.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/07.png" class="lightbox" data-gallery="quarto-lightbox-gallery-7" title="Process diagram for iteration 7 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/07.png" class="img-fluid figure-img" alt="Process diagram for iteration 7 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 7 - Click to expand</figcaption>
</figure>
</div>
<div class="callout callout-style-default callout-important callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Important
</div>
</div>
<div class="callout-body-container callout-body">
<p>I use a real OpenAI API key in these clips to demonstrate the application working. Note that I have since revoked this key and it will no longer work. You should not share your secret keys with anyone.</p>
</div>
</div>
<p>When using this iteration of the app, you can see that certain prompts may now be flagged as inappropriate. Note that this also reduces the performance of our app, as we need to send and process 2 queries for each prompt that the user enters.</p>
<iframe title="Iteration 7 recording" src="https://player.vimeo.com/video/1010112083?h=dc49a29e07" class="iframey" allowfullscreen="" style="overflow:hidden;margin:0;padding:0;width:100%;height:24rem;">
</iframe>
</div>
<div id="tabset-1-8" class="tab-pane" aria-labelledby="tabset-1-8-tab">
<p>We have nearly arrived at our final design. In this stage, we refactor the application to ensure we use a single openai client to query the separate endpoints.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-10" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-10-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 8: Refactor OpenAI client instantiation."""</span></span>
<span id="annotated-cell-10-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-10-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shiny <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> App, reactive, ui</span>
<span id="annotated-cell-10-4"></span>
<span id="annotated-cell-10-5">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-10-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-10-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-10-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-10-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-10-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-10-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-10-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-10-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-10-14"></span>
<span id="annotated-cell-10-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<span id="annotated-cell-10-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-10-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-10-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-10-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-10-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-10-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-10-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-10-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<span id="annotated-cell-10-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-10-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-10-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-10-27"></span>
<span id="annotated-cell-10-28">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-10-29"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-10-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-10-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-10-32"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-10-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-10-34"></span>
<span id="annotated-cell-10-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-10-36"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-10-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-10-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-10-39"></span>
<span id="annotated-cell-10-40"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># compose a message stream</span></span>
<span id="annotated-cell-10-41">_SYS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: _SYSTEM_MSG}</span>
<span id="annotated-cell-10-42">_WELCOME <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: WELCOME_MSG}</span>
<span id="annotated-cell-10-43">stream <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [_SYS, _WELCOME]</span>
<span id="annotated-cell-10-44"></span>
<span id="annotated-cell-10-45"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny User Interface ----------------------------------------------------</span></span>
<span id="annotated-cell-10-46"></span>
<span id="annotated-cell-10-47"></span>
<span id="annotated-cell-10-48"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> input_text_with_button(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span>, label, button_label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>):</span>
<span id="annotated-cell-10-49">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-10-50"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    An interface component combining an input text widget with an action</span></span>
<span id="annotated-cell-10-51"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    button. IDs for the text field and button can be accessed as &lt;id&gt;_text</span></span>
<span id="annotated-cell-10-52"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    and &lt;id&gt;_btn respectively.</span></span>
<span id="annotated-cell-10-53"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-10-54">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> ui.div(</span>
<span id="annotated-cell-10-55">        ui.input_text(</span>
<span id="annotated-cell-10-56">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_text"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>placeholder),</span>
<span id="annotated-cell-10-57">        ui.input_action_button(</span>
<span id="annotated-cell-10-58">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_btn"</span>,</span>
<span id="annotated-cell-10-59">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>button_label,</span>
<span id="annotated-cell-10-60">            style<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"margin-top:28px;margin-bottom:16px;color:#04bb8c;border-color:#04bb8c;"</span></span>
<span id="annotated-cell-10-61">            ),</span>
<span id="annotated-cell-10-62">        class_<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d-flex gap-2"</span></span>
<span id="annotated-cell-10-63">    )</span>
<span id="annotated-cell-10-64"></span>
<span id="annotated-cell-10-65"></span>
<span id="annotated-cell-10-66">app_ui <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.page_fillable(</span>
<span id="annotated-cell-10-67">    ui.panel_title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Choose Your Own Adventure: Jungle Quest!"</span>),</span>
<span id="annotated-cell-10-68">    ui.accordion(</span>
<span id="annotated-cell-10-69">    ui.accordion_panel(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 1: Your OpenAI API Key"</span>,</span>
<span id="annotated-cell-10-70">        input_text_with_button(</span>
<span id="annotated-cell-10-71">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"key_input"</span>,</span>
<span id="annotated-cell-10-72">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter your OpenAI API key"</span>,</span>
<span id="annotated-cell-10-73">            button_label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Submit"</span>,</span>
<span id="annotated-cell-10-74">            placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter key here"</span></span>
<span id="annotated-cell-10-75">            )), <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"acc"</span>, multiple<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span>),</span>
<span id="annotated-cell-10-76">ui.h6(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 2: Choose your adventure"</span>),</span>
<span id="annotated-cell-10-77">    ui.chat_ui(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>),</span>
<span id="annotated-cell-10-78">    fillable_mobile<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="annotated-cell-10-79">)</span>
<span id="annotated-cell-10-80"></span>
<span id="annotated-cell-10-81"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny server logic ------------------------------------------------------</span></span>
<span id="annotated-cell-10-82"></span>
<span id="annotated-cell-10-83"></span>
<span id="annotated-cell-10-84"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> server(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>, output, session):</span>
<span id="annotated-cell-10-85">    chat <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.Chat(</span>
<span id="annotated-cell-10-86">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>, messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[ui.markdown(WELCOME_MSG)], tokenizer<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span></span>
<span id="annotated-cell-10-87">        )</span>
<span id="annotated-cell-10-88">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  define a reactive value that will store the openai client</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-10" data-target-annotation="1">1</button><span id="annotated-cell-10-89" class="code-annotation-target">    openai_client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> reactive.Value(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>)</span>
<span id="annotated-cell-10-90"></span>
<span id="annotated-cell-10-91"></span>
<span id="annotated-cell-10-92">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.Effect</span></span>
<span id="annotated-cell-10-93">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.event</span>(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_btn)</span>
<span id="annotated-cell-10-94">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> handle_api_key_submit():</span>
<span id="annotated-cell-10-95">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Update the UI with a notification when user submits key.</span></span>
<span id="annotated-cell-10-96"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        </span></span>
<span id="annotated-cell-10-97"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Checks the validity of the API key by querying the models list</span></span>
<span id="annotated-cell-10-98"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        endpoint."""</span></span>
<span id="annotated-cell-10-99">        api_key <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text()</span>
<span id="annotated-cell-10-100">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>api_key)</span>
<span id="annotated-cell-10-101">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="annotated-cell-10-102">            resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.models.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">list</span>()</span>
<span id="annotated-cell-10-103">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> resp:</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-10" data-target-annotation="2">2</button><span id="annotated-cell-10-104" class="code-annotation-target">                openai_client.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">set</span>(client)</span>
<span id="annotated-cell-10-105">                ui.notification_show(</span>
<span id="annotated-cell-10-106">                    <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"API key validated: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>api_key[:<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">..."</span>)</span>
<span id="annotated-cell-10-107">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> openai.AuthenticationError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="annotated-cell-10-108">            ui.notification_show(</span>
<span id="annotated-cell-10-109">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Bad key provided. Please try again."</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"warning"</span>)</span>
<span id="annotated-cell-10-110">    </span>
<span id="annotated-cell-10-111"></span>
<span id="annotated-cell-10-112">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> check_moderation(</span>
<span id="annotated-cell-10-113">            prompt:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, reactive_client:reactive.Value</span>
<span id="annotated-cell-10-114">            ) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="annotated-cell-10-115">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Check if prompt is flagged by OpenAI's moderation endpoint.</span></span>
<span id="annotated-cell-10-116"></span>
<span id="annotated-cell-10-117"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Parameters</span></span>
<span id="annotated-cell-10-118"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        ----------</span></span>
<span id="annotated-cell-10-119"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        prompt : str</span></span>
<span id="annotated-cell-10-120"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            The user's prompt to check.</span></span>
<span id="annotated-cell-10-121"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        reactive_client : reactive.Value</span></span>
<span id="annotated-cell-10-122"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            A reactive value that stores the openai client.</span></span>
<span id="annotated-cell-10-123"></span>
<span id="annotated-cell-10-124"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Returns</span></span>
<span id="annotated-cell-10-125"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        -------</span></span>
<span id="annotated-cell-10-126"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        str</span></span>
<span id="annotated-cell-10-127"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            The category violations if flagged, otherwise "good prompt".</span></span>
<span id="annotated-cell-10-128"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        """</span></span>
<span id="annotated-cell-10-129">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> reactive_client.get()</span>
<span id="annotated-cell-10-130">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.moderations.create(</span>
<span id="annotated-cell-10-131">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>prompt)</span>
<span id="annotated-cell-10-132">        content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.results[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].to_dict()</span>
<span id="annotated-cell-10-133">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> content[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"flagged"</span>]:</span>
<span id="annotated-cell-10-134">            infringements <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> []</span>
<span id="annotated-cell-10-135">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> key, val <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> content[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"categories"</span>].items():</span>
<span id="annotated-cell-10-136">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> val:</span>
<span id="annotated-cell-10-137">                    infringements.append(key)</span>
<span id="annotated-cell-10-138">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" &amp; "</span>.join(infringements)</span>
<span id="annotated-cell-10-139">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-10-140">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"good prompt"</span></span>
<span id="annotated-cell-10-141">    </span>
<span id="annotated-cell-10-142"></span>
<span id="annotated-cell-10-143">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Define a callback to run when the user submits a message</span></span>
<span id="annotated-cell-10-144">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@chat.on_user_submit</span></span>
<span id="annotated-cell-10-145">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> respond():</span>
<span id="annotated-cell-10-146">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Respond to the user's message."""</span></span>
<span id="annotated-cell-10-147">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Get the user's input</span></span>
<span id="annotated-cell-10-148">        usr_prompt <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> chat.user_input()</span>
<span id="annotated-cell-10-149"></span>
<span id="annotated-cell-10-150">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Check moderations endpoint incase openai policies are violated</span></span>
<span id="annotated-cell-10-151">        flag_check <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> check_moderation(</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-10" data-target-annotation="3">3</button><span id="annotated-cell-10-152" class="code-annotation-target">            prompt<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>usr_prompt, reactive_client<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>openai_client)</span>
<span id="annotated-cell-10-153">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> flag_check <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"good prompt"</span>:</span>
<span id="annotated-cell-10-154">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message({</span>
<span id="annotated-cell-10-155">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-10-156">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Your message may violate OpenAI's usage policy, categories: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>flag_check<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">. Please rephrase your input and try again."</span></span>
<span id="annotated-cell-10-157">            })</span>
<span id="annotated-cell-10-158">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-10-159">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  update the stream list</span></span>
<span id="annotated-cell-10-160">            stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: usr_prompt})</span>
<span id="annotated-cell-10-161">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Append a response to the chat</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-10" data-target-annotation="4">4</button><span id="annotated-cell-10-162" class="code-annotation-target">            response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> openai_client.get().chat.completions.create(</span>
<span id="annotated-cell-10-163">                model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<span id="annotated-cell-10-164">                messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>stream,</span>
<span id="annotated-cell-10-165">                temperature<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.7</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># increase to make the model more creative</span></span>
<span id="annotated-cell-10-166">                )</span>
<span id="annotated-cell-10-167">            model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-10-168">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(model_response)</span>
<span id="annotated-cell-10-169">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  if the model indicates game over, end game with a message.</span></span>
<span id="annotated-cell-10-170">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"the end..."</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> model_response.lower():</span>
<span id="annotated-cell-10-171">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(</span>
<span id="annotated-cell-10-172">                    {</span>
<span id="annotated-cell-10-173">                        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-10-174">                        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Game Over! Refresh to play again."</span></span>
<span id="annotated-cell-10-175">                        })</span>
<span id="annotated-cell-10-176">                exit()</span>
<span id="annotated-cell-10-177">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-10-178">                stream.append(</span>
<span id="annotated-cell-10-179">                    {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: model_response})</span>
<span id="annotated-cell-10-180"></span>
<span id="annotated-cell-10-181"></span>
<span id="annotated-cell-10-182">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> App(ui<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>app_ui, server<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>server)</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-10" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-10" data-code-lines="89" data-code-annotation="1"><code>shiny</code> <a href="https://shiny.posit.co/py/api/core/reactive.value.html" target="_blank">reactive values</a> are often good choices for objects that you intend to update at multiple points within a <code>shiny</code> server. Here we are defining an empty reactive value that we can subsequently use to store and access an openai client.</span>
</dd>
<dt data-target-cell="annotated-cell-10" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-10" data-code-lines="104" data-code-annotation="2">It’s a subtle change, but now if the client returns a list of models, validating the api key that the user passed, then we set that client as the return value of <code>openai_client</code>.</span>
</dd>
<dt data-target-cell="annotated-cell-10" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-10" data-code-lines="152" data-code-annotation="3">The <code>check_moderations()</code> function expects to receive the reactive value object and will use it to <code>.get()</code> the stored client.</span>
</dd>
<dt data-target-cell="annotated-cell-10" data-target-annotation="4">4</dt>
<dd>
<span data-code-cell="annotated-cell-10" data-code-lines="162" data-code-annotation="4">This is the last occasion that we access the reactive client value. We initiated the client once and used it in three places to query 3 different endpoints.</span>
</dd>
</dl>
<p>This refactoring reduces the complexity of the app, ensuring that once the api key has been validated, that same openai client will be passed to the <code>moderations</code> and <code>chat.completions</code> endpoints.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href=".././www/18-jungle-quest-app/08.png" class="lightbox" data-gallery="quarto-lightbox-gallery-8" title="Process diagram for iteration 8 - Click to expand"><img src="https://thedatasavvycorner.netlify.app/www/18-jungle-quest-app/08.png" class="img-fluid figure-img" alt="Process diagram for iteration 8 - Click to expand"></a></p>
<figcaption>Process diagram for iteration 8 - Click to expand</figcaption>
</figure>
</div>
</div>
<div id="tabset-1-9" class="tab-pane" aria-labelledby="tabset-1-9-tab">
<p>In this final stage, we introduce some aesthetic changes - adding a theme and an image to chat UI, adding to the feel of the app. It’s always nice to leave some of the styling towards the end of a build, a bit like adding the cherry to a cake.</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre><strong>app.py</strong></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="annotated-cell-11" data-filename="app.py" style="background: #f1f3f5;"><pre class="sourceCode python code-annotation-code code-with-copy code-annotated"><code class="sourceCode python"><span id="annotated-cell-11-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Iteration 9: Styling."""</span></span>
<span id="annotated-cell-11-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> openai</span>
<span id="annotated-cell-11-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shiny <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> App, reactive, ui</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-11" data-target-annotation="1">1</button><span id="annotated-cell-11-4" class="code-annotation-target"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> shinyswatch <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> theme</span>
<span id="annotated-cell-11-5"></span>
<span id="annotated-cell-11-6">_SYSTEM_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-11-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">You are the guide of a 'choose your own adventure'- style game: a mystical</span></span>
<span id="annotated-cell-11-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">journey through the Amazon Rainforest. Your job is to create compelling</span></span>
<span id="annotated-cell-11-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">outcomes that correspond with the player's choices. You must navigate the</span></span>
<span id="annotated-cell-11-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">player through challenges, providing choices, and consequences, dynamically</span></span>
<span id="annotated-cell-11-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">adapting the tale based on the player's inputs. Your goal is to create a</span></span>
<span id="annotated-cell-11-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">branching narrative experience where each of the player's choices leads to</span></span>
<span id="annotated-cell-11-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">a new path, ultimately determining their fate. The player's goal is to find</span></span>
<span id="annotated-cell-11-14"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the lost crown of Quetzalcoatl.</span></span>
<span id="annotated-cell-11-15"></span>
<span id="annotated-cell-11-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Here are some rules to follow:</span></span>
<span id="annotated-cell-11-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">1. Always wait for the player to respond with their input before providing</span></span>
<span id="annotated-cell-11-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">any choices. Never provide the player's input yourself. This is most</span></span>
<span id="annotated-cell-11-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">important.</span></span>
<span id="annotated-cell-11-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">2. Ask the player to provide a name, gender and race.</span></span>
<span id="annotated-cell-11-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">3. Ask the player to choose from a selection of weapons that will be used</span></span>
<span id="annotated-cell-11-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">later in the game.</span></span>
<span id="annotated-cell-11-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">4. Have a few paths that lead to success. </span></span>
<span id="annotated-cell-11-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">5. Have some paths that lead to death.</span></span>
<span id="annotated-cell-11-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">6. Whether or not the game results in success or death, the response must</span></span>
<span id="annotated-cell-11-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">include the text "The End...", I will search for this text to end the game.</span></span>
<span id="annotated-cell-11-27"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-11-28"></span>
<span id="annotated-cell-11-29">WELCOME_MSG <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-11-30"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Welcome to the Amazon Rainforest, adventurer! Your mission is to find the</span></span>
<span id="annotated-cell-11-31"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">lost Crown of Quetzalcoatl:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span></span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-11" data-target-annotation="2">2</button><span id="annotated-cell-11-32" class="code-annotation-target"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">&lt;div style="display: grid; place-items: center;"&gt;&lt;img src="https://raw.githubusercontent.com/r-leyshon/adventure-app/main/www/crown.jpeg" width=60%/&gt;&lt;/div&gt;</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span></span>
<span id="annotated-cell-11-33"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">However, many challenges stand in your way. Are you brave enough, strong</span></span>
<span id="annotated-cell-11-34"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">enough and clever enough to overcome the perils of the jungle and secure</span></span>
<span id="annotated-cell-11-35"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">the crown?</span></span>
<span id="annotated-cell-11-36"></span>
<span id="annotated-cell-11-37"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Before we begin our journey, choose your name, gender and race. Choose a</span></span>
<span id="annotated-cell-11-38"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">weapon to bring with you. Choose wisely, as the way ahead is filled with</span></span>
<span id="annotated-cell-11-39"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">many dangers.</span></span>
<span id="annotated-cell-11-40"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-11-41"></span>
<span id="annotated-cell-11-42"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># compose a message stream</span></span>
<span id="annotated-cell-11-43">_SYS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: _SYSTEM_MSG}</span>
<span id="annotated-cell-11-44">_WELCOME <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: WELCOME_MSG}</span>
<span id="annotated-cell-11-45">stream <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [_SYS, _WELCOME]</span>
<span id="annotated-cell-11-46"></span>
<span id="annotated-cell-11-47"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny User Interface ----------------------------------------------------</span></span>
<span id="annotated-cell-11-48"></span>
<span id="annotated-cell-11-49"></span>
<span id="annotated-cell-11-50"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> input_text_with_button(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span>, label, button_label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>):</span>
<span id="annotated-cell-11-51">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="annotated-cell-11-52"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    An interface component combining an input text widget with an action</span></span>
<span id="annotated-cell-11-53"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    button. IDs for the text field and button can be accessed as &lt;id&gt;_text</span></span>
<span id="annotated-cell-11-54"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    and &lt;id&gt;_btn respectively.</span></span>
<span id="annotated-cell-11-55"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="annotated-cell-11-56">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> ui.div(</span>
<span id="annotated-cell-11-57">        ui.input_text(</span>
<span id="annotated-cell-11-58">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_text"</span>, label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>label, placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>placeholder),</span>
<span id="annotated-cell-11-59">        ui.input_action_button(</span>
<span id="annotated-cell-11-60">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_btn"</span>,</span>
<span id="annotated-cell-11-61">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>button_label,</span>
<span id="annotated-cell-11-62">            style<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"margin-top:32px;margin-bottom:16px;color:#04bb8c;border-color:#04bb8c;"</span></span>
<span id="annotated-cell-11-63">            ),</span>
<span id="annotated-cell-11-64">        class_<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d-flex gap-2"</span></span>
<span id="annotated-cell-11-65">    )</span>
<span id="annotated-cell-11-66"></span>
<span id="annotated-cell-11-67"></span>
<span id="annotated-cell-11-68">app_ui <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.page_fillable(</span>
<span id="annotated-cell-11-69">    ui.panel_title(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Choose Your Own Adventure: Jungle Quest!"</span>),</span>
<span id="annotated-cell-11-70">    ui.accordion(</span>
<span id="annotated-cell-11-71">    ui.accordion_panel(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 1: Your OpenAI API Key"</span>,</span>
<span id="annotated-cell-11-72">        input_text_with_button(</span>
<span id="annotated-cell-11-73">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"key_input"</span>,</span>
<span id="annotated-cell-11-74">            label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter your OpenAI API key"</span>,</span>
<span id="annotated-cell-11-75">            button_label<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Submit"</span>,</span>
<span id="annotated-cell-11-76">            placeholder<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Enter key here"</span></span>
<span id="annotated-cell-11-77">            )), <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"acc"</span>, multiple<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span>),</span>
<span id="annotated-cell-11-78">ui.h6(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Step 2: Choose your adventure"</span>),</span>
<span id="annotated-cell-11-79">    ui.chat_ui(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>),</span>
<span id="annotated-cell-11-80">    fillable_mobile<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<button class="code-annotation-anchor" data-target-cell="annotated-cell-11" data-target-annotation="3">3</button><span id="annotated-cell-11-81" class="code-annotation-target">    theme<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>theme.darkly,</span>
<span id="annotated-cell-11-82">)</span>
<span id="annotated-cell-11-83"></span>
<span id="annotated-cell-11-84"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Shiny server logic ------------------------------------------------------</span></span>
<span id="annotated-cell-11-85"></span>
<span id="annotated-cell-11-86"></span>
<span id="annotated-cell-11-87"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> server(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>, output, session):</span>
<span id="annotated-cell-11-88">    chat <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ui.Chat(</span>
<span id="annotated-cell-11-89">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"chat"</span>, messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[ui.markdown(WELCOME_MSG)], tokenizer<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span></span>
<span id="annotated-cell-11-90">        )</span>
<span id="annotated-cell-11-91">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  define a reactive value that will store the openai client</span></span>
<span id="annotated-cell-11-92">    openai_client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> reactive.Value(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>)</span>
<span id="annotated-cell-11-93"></span>
<span id="annotated-cell-11-94"></span>
<span id="annotated-cell-11-95">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.Effect</span></span>
<span id="annotated-cell-11-96">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@reactive.event</span>(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_btn)</span>
<span id="annotated-cell-11-97">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> handle_api_key_submit():</span>
<span id="annotated-cell-11-98">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Update the UI with a notification when user submits key.</span></span>
<span id="annotated-cell-11-99"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        </span></span>
<span id="annotated-cell-11-100"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Checks the validity of the API key by querying the models list</span></span>
<span id="annotated-cell-11-101"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        endpoint."""</span></span>
<span id="annotated-cell-11-102">        api_key <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>.key_input_text()</span>
<span id="annotated-cell-11-103">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> openai.AsyncOpenAI(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>api_key)</span>
<span id="annotated-cell-11-104">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="annotated-cell-11-105">            resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.models.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">list</span>()</span>
<span id="annotated-cell-11-106">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> resp:</span>
<span id="annotated-cell-11-107">                openai_client.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">set</span>(client)</span>
<span id="annotated-cell-11-108">                ui.notification_show(</span>
<span id="annotated-cell-11-109">                    <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"API key validated: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>api_key[:<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>]<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">..."</span>)</span>
<span id="annotated-cell-11-110">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> openai.AuthenticationError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="annotated-cell-11-111">            ui.notification_show(</span>
<span id="annotated-cell-11-112">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Bad key provided. Please try again."</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"warning"</span>)</span>
<span id="annotated-cell-11-113">    </span>
<span id="annotated-cell-11-114"></span>
<span id="annotated-cell-11-115">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> check_moderation(</span>
<span id="annotated-cell-11-116">            prompt:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, reactive_client:reactive.Value</span>
<span id="annotated-cell-11-117">            ) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="annotated-cell-11-118">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Check if prompt is flagged by OpenAI's moderation endpoint.</span></span>
<span id="annotated-cell-11-119"></span>
<span id="annotated-cell-11-120"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Parameters</span></span>
<span id="annotated-cell-11-121"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        ----------</span></span>
<span id="annotated-cell-11-122"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        prompt : str</span></span>
<span id="annotated-cell-11-123"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            The user's prompt to check.</span></span>
<span id="annotated-cell-11-124"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        reactive_client : reactive.Value</span></span>
<span id="annotated-cell-11-125"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            A reactive value that stores the openai client.</span></span>
<span id="annotated-cell-11-126"></span>
<span id="annotated-cell-11-127"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Returns</span></span>
<span id="annotated-cell-11-128"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        -------</span></span>
<span id="annotated-cell-11-129"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        str</span></span>
<span id="annotated-cell-11-130"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">            The category violations if flagged, otherwise "good prompt".</span></span>
<span id="annotated-cell-11-131"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        """</span></span>
<span id="annotated-cell-11-132">        client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> reactive_client.get()</span>
<span id="annotated-cell-11-133">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> client.moderations.create(</span>
<span id="annotated-cell-11-134">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>prompt)</span>
<span id="annotated-cell-11-135">        content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.results[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].to_dict()</span>
<span id="annotated-cell-11-136">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> content[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"flagged"</span>]:</span>
<span id="annotated-cell-11-137">            infringements <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> []</span>
<span id="annotated-cell-11-138">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> key, val <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> content[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"categories"</span>].items():</span>
<span id="annotated-cell-11-139">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> val:</span>
<span id="annotated-cell-11-140">                    infringements.append(key)</span>
<span id="annotated-cell-11-141">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" &amp; "</span>.join(infringements)</span>
<span id="annotated-cell-11-142">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-11-143">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"good prompt"</span></span>
<span id="annotated-cell-11-144">    </span>
<span id="annotated-cell-11-145"></span>
<span id="annotated-cell-11-146">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Define a callback to run when the user submits a message</span></span>
<span id="annotated-cell-11-147">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@chat.on_user_submit</span></span>
<span id="annotated-cell-11-148">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> respond():</span>
<span id="annotated-cell-11-149">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Respond to the user's message.</span></span>
<span id="annotated-cell-11-150"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        </span></span>
<span id="annotated-cell-11-151"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        First check that OpenAI's usage policies are not moderated. If this</span></span>
<span id="annotated-cell-11-152"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        passes, then respond with a message from the model. If the model</span></span>
<span id="annotated-cell-11-153"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        has ended the game, then exit the game."""</span></span>
<span id="annotated-cell-11-154">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Get the user's input</span></span>
<span id="annotated-cell-11-155">        usr_prompt <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> chat.user_input()</span>
<span id="annotated-cell-11-156"></span>
<span id="annotated-cell-11-157">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Check moderations endpoint incase openai policies are violated</span></span>
<span id="annotated-cell-11-158">        flag_check <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> check_moderation(</span>
<span id="annotated-cell-11-159">            prompt<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>usr_prompt, reactive_client<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>openai_client)</span>
<span id="annotated-cell-11-160">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> flag_check <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"good prompt"</span>:</span>
<span id="annotated-cell-11-161">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message({</span>
<span id="annotated-cell-11-162">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-11-163">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Your message may violate OpenAI's usage policy, categories: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>flag_check<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">. Please rephrase your input and try again."</span></span>
<span id="annotated-cell-11-164">            })</span>
<span id="annotated-cell-11-165">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-11-166">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  update the stream list</span></span>
<span id="annotated-cell-11-167">            stream.append({<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"user"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: usr_prompt})</span>
<span id="annotated-cell-11-168">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Append a response to the chat</span></span>
<span id="annotated-cell-11-169">            response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> openai_client.get().chat.completions.create(</span>
<span id="annotated-cell-11-170">                model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gpt-3.5-turbo"</span>,</span>
<span id="annotated-cell-11-171">                messages<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>stream,</span>
<span id="annotated-cell-11-172">                temperature<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.7</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># increase to make the model more creative</span></span>
<span id="annotated-cell-11-173">                )</span>
<span id="annotated-cell-11-174">            model_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> response.choices[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>].message.content</span>
<span id="annotated-cell-11-175">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(model_response)</span>
<span id="annotated-cell-11-176">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  if the model indicates game over, end game with a message.</span></span>
<span id="annotated-cell-11-177">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"the end..."</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> model_response.lower():</span>
<span id="annotated-cell-11-178">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> chat.append_message(</span>
<span id="annotated-cell-11-179">                    {</span>
<span id="annotated-cell-11-180">                        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>,</span>
<span id="annotated-cell-11-181">                        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Game Over! Refresh to play again."</span></span>
<span id="annotated-cell-11-182">                        })</span>
<span id="annotated-cell-11-183">                exit()</span>
<span id="annotated-cell-11-184">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="annotated-cell-11-185">                stream.append(</span>
<span id="annotated-cell-11-186">                    {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"role"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"assistant"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"content"</span>: model_response})</span>
<span id="annotated-cell-11-187"></span>
<span id="annotated-cell-11-188"></span>
<span id="annotated-cell-11-189">app <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> App(ui<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>app_ui, server<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>server)</span><div class="code-annotation-gutter-bg"></div><div class="code-annotation-gutter"></div></code></pre></div></div>
</div>
<dl class="code-annotation-container-hidden code-annotation-container-grid">
<dt data-target-cell="annotated-cell-11" data-target-annotation="1">1</dt>
<dd>
<span data-code-cell="annotated-cell-11" data-code-lines="4" data-code-annotation="1">In order to set a theme we import <a href="https://github.com/posit-dev/py-shinyswatch" target="_blank"><code>shinyswatch</code></a>.</span>
</dd>
<dt data-target-cell="annotated-cell-11" data-target-annotation="2">2</dt>
<dd>
<span data-code-cell="annotated-cell-11" data-code-lines="32" data-code-annotation="2">Markdown or HTML that you pass to the chat interface will get rendered. This includes images, links and even emojis. In fact, one of the gpt-4o models I tested decided to respond with emojis.</span>
</dd>
<dt data-target-cell="annotated-cell-11" data-target-annotation="3">3</dt>
<dd>
<span data-code-cell="annotated-cell-11" data-code-lines="81" data-code-annotation="3">Popping this one-liner in your UI will apply the theme styling to your application. Feel free to select any of the available options, see the interactive theme selector app below for all available themes.</span>
</dd>
</dl>
<p><code>shinyswatch</code> is a great utility that allows for efficient styling of python <code>shiny</code> apps. Check out this shinylive app from the package maintainers that allows you to easily toggle between the different themes available.</p>
<iframe src="https://shinylive.io/py/editor/#code=NobwRAdghgtgpmAXAAjFADugdOgnmAGlQGMB7CAFzkqVQDMAnUmZAZwAsBLCXZTmdKQYVkAQUxEAkhHQBXCqyIB5eXIVEAynFatO5Ig2oATOAyKzOAHQjX+g4Wy49WAdygVi7a9YzoA+hbIALzIFjhQAOZwfnQANhZGABTWyKmO3Liu7p5YFOxw8H7onMQA1qYBnIkAlETIAMTIADwAtGJGRsh5cF358MjFZabIAKqSXaTIuKSyDMi+KWlh3Gp+rLGcJgzJkLIwloTIBwByewBGpogHRDDcQQCMAAw3UAAeD4-PyABuUPFwQQAzI9aotUmEZhRVlRXhQ-N9TGd3PwdutNhVfrEDqCIGlegUARwMlkPOxcn04FgAF5wdDsXBmVKNVrIADC7FIpFYPSguO4nAonD++P6iVI6EF5D+1WsMps8pMdDYpgR2xW8hQ0jUimQkLUKBUUPkOu5Oj0EBQWjN5GqV1xaUaAFkoOU2LMetNZsqGKrkHRZBBiJLccQ-rFWPiRT1BuU5tyfcN-YHg2D0s43KTyQSiiVY2sVaYat57akAAJ6+Sp0uGCBbXJwWGpxVsDZbeF-Gp2vF4wwUWa4ugHEDqihYCB7GoAXwOxZ8mGCYkwiV8lSI8dVcsOYAouHQCBQ24bFDAk6I4Gg8FoYEMAEcLIZ4JRWLlYVuyJRqMeD0SeNYf5kM08awtx3PcrxhY9JwAXSAA" class="iframey" style="overflow:hidden;margin:0;padding:0;width:100%;height:30rem">
</iframe>
<div class="callout callout-style-default callout-important callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Important
</div>
</div>
<div class="callout-body-container callout-body">
<p>I use a real OpenAI API key in these clips to demonstrate the application working. Note that I have since revoked this key and it will no longer work. You should not share your secret keys with anyone.</p>
</div>
</div>
<p>The final app is demonstrated in full below.</p>
<iframe title="Iteration 9 recording" src="https://player.vimeo.com/video/1010112202?h=273164945d" class="iframey" allowfullscreen="" style="overflow:hidden;margin:0;padding:0;width:100%;height:24rem;">
</iframe>
</div>
</div>
</div>
<p>Click to return to the start of the iteration tabsets</p>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>Finally, we have arrived at the end of the application development! We started out with a basic little script that demonstrated how to use the <code>openai</code> python client and have arrived at a fully functional game with some additional safeguards against misuse. If you’ve stuck with it this far - well done, I’d say you’ve earned yourself a pat on the back!</p>
<div hidden="">
<p><i class="fa-solid fa-hands-clapping" aria-label="hands-clapping"></i></p>
</div>
<p><i class="fa-solid fa-hands-clapping fa-beat fa-2xl" style="color:#ffa53d;text-align:center;display:inline-block;width:100%;"></i></p>
<p>If you’d like to continue improving the app, then here are some suggestions:</p>
<ul>
<li>Pass more context to the moderations endpoint to minimise false positives.</li>
<li>Add a temperature control, so that the user can adjust how creative the models can be.</li>
<li>Add a model selection widget so that the user can try out responses with different openai models.</li>
<li>Experiment with <a href="https://github.com/r-leyshon/adventure-app/tree/stream-example" target="_blanks">streaming the responses</a> for a more responsive design.</li>
<li>Restructure your app to make use of <a href="https://shiny.posit.co/py/docs/modules.html" target="_blank"><code>shiny</code> modules</a>, increasing the application’s maintainability.</li>
</ul>
<p>I hope you’ve enjoyed the tutorial and learned something new. If you’re able to riff off this with your own designs I would be really interested to take a look - please feel free to post links to your own creations in the comments at the end of the blog. If you’d like to consider how to share your app with others, take a look at my article on <a href="../blogs/07-schedule-deploy-shinyapps.html">Deployment to Shinyapps.io</a>.</p>
<p>If you spot an error with this article, or have a suggested improvement then feel free to leave a comment (GitHub login required) or <a href="https://github.com/r-leyshon/blogging/issues">raise an issue on GitHub</a>.</p>
<p id="fin">
<i>fin!</i>
</p>


</section>

 ]]></description>
  <category>Tutorial</category>
  <category>Python Shiny</category>
  <category>LLMs</category>
  <category>Large Language Models</category>
  <category>GenAI</category>
  <category>Generative AI</category>
  <category>Front End Dev</category>
  <category>OpenAI</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/18-jungle-quest-app.html</guid>
  <pubDate>Sat, 21 Sep 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/18-jungle-quest-app/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Building a Second Brain</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/book-reviews/06-second-brain.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/book-reviews/06-second-brain/intro-img.jpg" alt="An image of A digital brain built from circuitry." style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<section id="book-summary" class="level2">
<h2 class="anchored" data-anchor-id="book-summary">Book Summary</h2>
<p>Building a Second Brain by Tiago Forte is a productivity book that teaches how to take and organise effective notes in a digital format. Collating these records with your own synthesis within a cohesive software ecosystem in order to improve productivity and support creative work.</p>
<blockquote class="blockquote">
<p>“Those who learn how to leverage technology and master the flow of information through their lives will be empowered to accomplish anything they set their minds to.” <span class="citation" data-cites="2ndBrain">[1, p. 7]</span></p>
</blockquote>
</section>
<section id="on-the-author" class="level2">
<h2 class="anchored" data-anchor-id="on-the-author">On the Author</h2>
<p>Tiago Forte is a productivity expert known for his work on personal knowledge management. He studied international business and worked in various roles, including the Peace Corps in Ukraine, where he taught productivity techniques. His career in innovation consulting introduced him to the world of Personal Knowledge Management (PKM), leading to the creation of his training venture, <a href="https://fortelabs.com/" target="_blank">Forte Labs</a>. This has since grown into a successful online education company, with Forte emphasising the integration of personal and professional growth. His work focuses on helping individuals manage information overload to enhance creativity and productivity.</p>
</section>
<section id="three-takeaway-ideas" class="level2">
<h2 class="anchored" data-anchor-id="three-takeaway-ideas">Three Takeaway Ideas</h2>
<section id="consolidation." class="level3">
<h3 class="anchored" data-anchor-id="consolidation.">1. Consolidation.</h3>
<p>What I liked best about this book was that it didn’t feel completely new to me. It discussed concepts that I had intuited and practised in less formal ways prior to reading. Therefore, adaptation to a centralised digital notekeeping solution would present a minor attitude adjustment, rather than a complete transformation.</p>
<p>Back in 2015 I had attended a training course on how to get the most out of Microsoft OneNote, with low expectations and even less interest. Previously, OneNote had been the perplexing default option for printing documents - a minor annoyance while I swiftly found the printer I needed. The training session did a lot to demystify the software and highlighted some great features which at the time felt quite revolutionary, notably audio transcription, apply and search notes by tag and video embedding. I proceeded to use OneNote to record meeting outcomes, transcribe user-testing sessions, scribble ideas over papers and policy documents and so on. Some of my colleagues started to notice and asked me about it. I remember saying these words:</p>
<blockquote class="blockquote">
<p>If I need to find something, I can ‘CTRL + F’ the software, whereas I can’t do that with a notebook or stack of post-it notes.</p>
</blockquote>
<p>Over time, my digital ecosystem has diversified a bit. Sometimes because of me and software compatibility with the devices that I use, and at other times because of the preferences of my team. This book made a great case for favouring interoperable software to achieve a unified note taking system. Software that cannot work together is not serving your ability to retrieve information when you need it most.</p>
<blockquote class="blockquote">
<p>“Research from Microsoft shows that the average US employee spends 76 hours per year looking for misplaced notes, items, or files. And a report from the International Data Corporation found that 26 percent of a typical knowledge worker’s day is spent looking for and consolidating information spread across a variety of systems.” <span class="citation" data-cites="2ndBrain">[1, p. 16]</span></p>
</blockquote>
<p>This must be achingly familiar to most modern workers, especially programmers. Those times when you find yourself Googling for solutions that you once knew, following purple links that you’ve already visited. Or trying to find that salient programming meme or inspirational quote in a newly refreshed social media feed. Surely, everyone has experienced this? Tiago Forte’s book suggests that if we ‘get our digital act together’, we don’t have to rely on our biological brain for recall. We can farm that job out to a digital note taking system which will perform better, recall faster and with greater accuracy. Instead, we can shift our efforts to synthesis - spotting connections between notes, adding our interpretation to them and extracting relevance in the process.</p>
</section>
<section id="c.o.d.e." class="level3">
<h3 class="anchored" data-anchor-id="c.o.d.e.">2. C.O.D.E.</h3>
<p>To anyone who has found themselves mindlessly highlighting reams of a text with no specific reason in mind, other than in a futile attempt to try to remember everything (guilty as charged), Forte’s C.O.D.E. recipe can offer clarity:</p>
<p><img src="https://thedatasavvycorner.netlify.app/www/book-reviews/06-second-brain/CODE.001.jpeg" alt="Tiago Forte's recipe for success: Capture, Organise, Distil &amp; Express. Image credit https://fortelabs.com/" width="80%" class="center"></p>
<p>My interpretation of the recipe is as follows:</p>
<p><strong style="font-size:24px">C</strong>apture: Your note taking app should be ready to go at all times. With the capability to sync across all of your devices and backed up in the cloud, you should never miss an opportunity to make a record of an important piece of information, whenever and wherever it presents itself to you. More on the type of information to capture in a note in section 3.</p>
<p><strong style="font-size:24px">O</strong>rganise: Consider where the note should live within your digital brain. Don’t put pressure on yourself to do this straight away, as it may dissuade you from recording notes in the first place. But use your default section of a notebook (the area where documents would print to by default in OneNote for example) as a holding area. You can then triage these notes later in the day.</p>
<p>Also, don’t agonise over where to locate these notes. Remember that you can search for them across your entire notebook. In fact, revisiting important notes and realigning them to new project folders or bringing old notes out of cold storage can play an important part in the creative process - never start from scratch again! Every time you initiate a new project, a great starting point would be to consult existing notes for relevant content. Most modern note taking applications also allow you to create internal links, allowing you to build connections across topics in your digital notes, or to group notes together under an index page.</p>
<p><strong style="font-size:24px">D</strong>istill: The author suggests a specific summarising mechanism to allow at-a-glance knowledge retrieval when revisiting notes. One crucial element in acquiring knowledge is that you should always endeavour to do something with the information you wish to learn. Over time, when the details fade in your mind, it is the impact or relevance of that information that remains. Forte encourages the reader to develop the habit of reflecting upon the studied content, leaving helpful summaries that will aid your future self in retrieving what was important.</p>
<p><strong style="font-size:24px">E</strong>xpress: Evaluate. Share the products of your note taking efforts and seek feedback from others. Find opportunities to help those around you in professional communities and consider how to refine your notes or your note-taking practice. Forte introduces the concept of “intermediate packets” - over time, projects may call for common notes or portions of notes to be put to work. In iteratively refining these packets of information, they can be adapted or generalised, greatly helping to facilitate future work.</p>
</section>
<section id="what-to-capture" class="level3">
<h3 class="anchored" data-anchor-id="what-to-capture">3. What to Capture?</h3>
<p>The author emphasises the importance of allowing your intuition to guide you when deciding on what information holds more value within any piece of text. The more you practise this skill, the better you will become in extracting the salient points of a text. If that sounds too much like hard work, the author suggests aligning your priorities to a set of ongoing projects that will help to guide your attention in capturing, organising and reflecting upon information.</p>
<blockquote class="blockquote">
<p>“You have to keep a dozen of your favourite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps.” (Nobel prize-winning physicist Richard Feynman reflects on his strategy for success) <span class="citation" data-cites="2ndBrain">[1, p. 44]</span></p>
</blockquote>
<p>I don’t consider the number twelve to be particularly important in all honesty. But you may have a few longer term targets in a professional development plan that you would like to progress - why not start with these? Having sections of your digital brain dedicated to these topics allows you to orient your efforts and order your thoughts.</p>
<p>Forte also provides direction to those who need more direction in deciding what information has the greatest utility.</p>
<blockquote class="blockquote">
<p>“Capture Criteria #1: Does It Inspire Me? …<br>
”Capture Criteria #2: Is It Useful? …<br>
“Capture Criteria #3: Is It Personal? …<br>
”Capture Criteria #4: Is It Inspiring?” <span class="citation" data-cites="2ndBrain">[1, pp. 48–49]</span></p>
</blockquote>
<p>These criteria can be used to help filter the information that you capture, allowing you to focus on that which is most important. The author offers a brief explanation for each of the criteria, supported by examples. I am currently trying these criteria on for size and have included a highlighting key within my note taking app to help me remember to use them.</p>
</section>
</section>
<section id="in-summary" class="level2">
<h2 class="anchored" data-anchor-id="in-summary">In Summary</h2>
<p>Overall, I would recommend this book to anyone interested in the personal efficacy space. It is full of practical advice and encouragement to get started or to refine your digital note taking system. The book contains plenty of examples that illustrate exactly how to put the ideas into action, catering for a range of preferred learning styles. Additionally, the author’s system is extremely well-supported and debated online. Check out the <a href="https://fortelabs.com" target="_blank">Forte Labs</a> website and the <a href="https://www.youtube.com/@TiagoForte" target="_blank">Tiago Forte YouTube channel</a> for more information.</p>
<p>If this book inspired you to start your own digital note taking system, you may find my review of Cal Newport’s <a href="../book-reviews/01-deep-work.html">Deep Work</a> book useful.</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>I would like to thank Beth for a fantastic book recommendation - this was a great read.</p>



</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body" data-entry-spacing="0">
<div id="ref-2ndBrain" class="csl-entry">
<div class="csl-left-margin">[1] </div><div class="csl-right-inline">Tiago Forte, <em><span class="nocase">Building a Second Brain</span>: <span class="nocase">A Proven Method to Organise Your Digital Life and Unlock Your Creative Potential.</span></em> <span>Profile Books Ltd</span>, 2022.</div>
</div>
</div></section></div> ]]></description>
  <category>Non-fiction</category>
  <category>Self-Help</category>
  <category>Productivity</category>
  <category>Personal Knowledge Management</category>
  <category>Personal Development</category>
  <category>Business</category>
  <category>Psychology</category>
  <category>Technology</category>
  <category>Note Taking</category>
  <guid>https://thedatasavvycorner.netlify.app/book-reviews/06-second-brain.html</guid>
  <pubDate>Fri, 06 Sep 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/book-reviews/06-second-brain/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Quarto Comments</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/17-quarto-comments.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/17-quarto-comments/intro-img.jpg" alt="Tamagotchi-styled neon emojis." style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<blockquote class="blockquote">
<p>“It takes humility to seek feedback. It takes wisdom to understand it, analyze it and appropriately act on it.” Stephen Covey.</p>
</blockquote>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>For people who use Quarto to build their websites, there are a few options for adding commenting functionality to their site. This allows your readers to engage with your site, leaving text or emoji feedback. Quarto currently offers built-in support for three commenting widgets. This article compares their functionality, linking to minimal implementations of each.</p>
<p>For full details on how to implement each solution, please consult the <a href="https://quarto.org/docs/output-formats/html-basics.html#commenting">Quarto commenting documentation</a>.</p>
<p>This article was written with Quarto 1.5.56. The solutions explored in this article are in development and may change in the future.</p>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Programmers with a working knowledge of quarto websites, considering options for adding commenting features to their site.</p>
</section>
<section id="what-youll-need" class="level3">
<h3 class="anchored" data-anchor-id="what-youll-need">What You’ll Need</h3>
<ul class="task-list">
<li><label><input type="checkbox"><a href="https://quarto.org/docs/download/index.html">Quarto CLI</a></label></li>
<li><label><input type="checkbox">Command line access</label></li>
</ul>
</section>
</section>
<section id="hypothesis" class="level2">
<h2 class="anchored" data-anchor-id="hypothesis">Hypothesis</h2>
<p>Hypothesis is a great new, feature-rich, social annotation solution. It has a strong <a href="https://web.hypothes.is/about/">commitment to open source</a>. After exploring its main features, I would suggest it is aimed at a broad audience with strong features for academic audiences, particularly cohorts of students that will benefit from group analysis of documents.</p>
<p>For users to interact with the Hypothesis interface, they will need to create an account and login, though this is not a painful process and will prevent excess spam on your site. An example of a minimal quarto website with Hypothesis enabled is below - feel free to test the commenting features yourself. Click on the arrow in the top-right hand corner to expand the Hypothesis user interface (UI). Code for the site is available on <a href="https://github.com/r-leyshon/quarto-hypothesis">GitHub</a>.</p>
<div style="text-align:center;padding-left:10%;">
<p><iframe class="iframey" src="https://r-leyshon.github.io/quarto-hypothesis/" style="overflow:hidden;margin:0;padding:0;width:80%"></iframe></p>
</div>
<hr>
<div id="tbl-panel" class="quarto-layout-panel anchored">
<figure class="quarto-float quarto-float-tbl figure">
<figcaption class="quarto-float-caption-top quarto-float-caption quarto-float-tbl" id="tbl-panel-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Table&nbsp;1: Hypothesis Observations
</figcaption>
<div aria-describedby="tbl-panel-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div class="quarto-layout-row">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-panel" style="flex-basis: 50.0%;justify-content: center;">
<div id="tbl-first" class="striped hover tabley quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-first-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Pros
</figcaption>
<div aria-describedby="tbl-first-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table class="table-striped table-hover tabley caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: left;">Simple setup, just include the Hypothesis portion in your site’s YAML &amp; render.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Private highlighting.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Private notetaking if you create a private group. Other members can be invited to participate.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Toggle on/off highlights and comments separately.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Annotations - linked specifically to a region of text in the article.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Page notes - applied to an entire page.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Annotations and page notes can be tagged with additional metadata.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Revisions are easy and built into the UI.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Collapsible UI is neat, reducing clutter.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Browser extension available.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Compatible with sites deployed from other repo-hosting solutions, eg GitLab, BitBucket.</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-panel" style="flex-basis: 50.0%;justify-content: center;">
<div id="tbl-second" class="striped hover tabley quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-second-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Cons
</figcaption>
<div aria-describedby="tbl-second-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table class="table-striped table-hover tabley caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: left;">I couldn’t get the embedding options to work (eg theme etc).</td>
</tr>
<tr class="even">
<td style="text-align: left;">No emoji reactions.</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
</div>
</div>
</figure>
</div>
</section>
<section id="utterances" class="level2">
<h2 class="anchored" data-anchor-id="utterances">Utterances</h2>
<p>Utterances uses GitHub Issues to store comment data. If you are deploying your site from GitHub and are not making good use of the repo issues, this could be a viable option. It’s important to note that your users will need a GitHub account and to authenticate in order to post comments on your site. Please consult the <a href="https://utteranc.es/">Utterances documentation</a> for more information. In order to read and write issues on your repo, you do need to give the <a href="https://github.com/apps/utterances">Utterances GitHub app</a> permission to read and write repo metadata. Without doing this, the Utterances UI will appear on your site but will not function correctly. Code for A minimal Quarto site with Utterances is <a href="https://github.com/r-leyshon/quarto-utterances">available on GitHub</a>. The site can be viewed below.</p>
<div style="text-align:center;padding-left:10%;">
<p><iframe class="iframey" src="https://r-leyshon.github.io/quarto-utterances/" style="overflow:hidden;margin:0;padding:0;width:80%"></iframe></p>
</div>
<hr>
<div id="tbl-panel" class="quarto-layout-panel anchored">
<figure class="quarto-float quarto-float-tbl figure">
<figcaption class="quarto-float-caption-top quarto-float-caption quarto-float-tbl" id="tbl-panel-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Table&nbsp;2: Utterances Observations
</figcaption>
<div aria-describedby="tbl-panel-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div class="quarto-layout-row">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-panel" style="flex-basis: 50.0%;justify-content: center;">
<div id="tbl-first" class="striped hover tabley quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-first-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Pros
</figcaption>
<div aria-describedby="tbl-first-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table class="table-striped table-hover tabley caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: left;">Either give it global access to all repos or selected repos.</td>
</tr>
<tr class="even">
<td style="text-align: left;">User must sign into GitHub and authorise the Utterances app.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Markdown compatibility as per usual GitHub issues.</td>
</tr>
<tr class="even">
<td style="text-align: left;">First comment on a page creates an issue, subsequent comments are added to the same issue.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Reactions to comments with emojis.</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-panel" style="flex-basis: 50.0%;justify-content: center;">
<div id="tbl-second" class="striped hover tabley quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-second-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Cons
</figcaption>
<div aria-describedby="tbl-second-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table class="table-striped table-hover tabley caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: left;">Every comment creates a GitHub issue on the repo – 1 per page of your site. Maybe a bit noisy for some.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Marking issues as closed does not remove them from the site.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Deleting the issue from the repo will remove the comment but may also mess up the mapping of comments to pages. May cause issues moderating.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Couldn’t get the labels feature to work, despite creating the issue label of the same name, as the documentation suggested.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Cannot respond to a comment with text.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Revisions are possible though are not achievable through the website UI. By navigating to the repo issues, you can edit any comment that was authored by your account.</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
</div>
</div>
</figure>
</div>
</section>
<section id="giscus" class="level2">
<h2 class="anchored" data-anchor-id="giscus">Giscus</h2>
<p>Giscus emulates the Utterances functionality, but by using the new(ish) GitHub Discussions feature as the comment database, instead of the repo issues. The <a href="https://giscus.app/">Giscus documentation</a> currently states the widget is in development, as well as the GitHub API that it relies on, so expect some changes with this solution. The Giscus documentation also provides a really useful tool that checks whether Giscus has been configured correctly on the target repo. This same tool was very useful in configuring the YAML metadata for my site correctly.</p>
<p>In the same way as Utterances, you will need to give the <a href="https://github.com/apps/giscus">Giscus GitHub app</a> permissions to read and write repo metadata. Users will also require a GitHub account in order to authenticate. Code for A minimal quarto site with Giscus is <a href="https://github.com/r-leyshon/quarto-giscus">available on GitHub</a>. The <a href="https://r-leyshon.github.io/quarto-giscus/">site can be viewed here</a> (the Giscus interface will not render correctly within an iframe).</p>
<hr>
<div id="tbl-panel" class="quarto-layout-panel anchored">
<figure class="quarto-float quarto-float-tbl figure">
<figcaption class="quarto-float-caption-top quarto-float-caption quarto-float-tbl" id="tbl-panel-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Table&nbsp;3: Giscus Observations
</figcaption>
<div aria-describedby="tbl-panel-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div class="quarto-layout-row">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-panel" style="flex-basis: 50.0%;justify-content: center;">
<div id="tbl-first" class="striped hover tabley quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-first-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Pros
</figcaption>
<div aria-describedby="tbl-first-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table class="table-striped table-hover tabley caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: left;">Activate discussions on your repo or give it global access to all repos.</td>
</tr>
<tr class="even">
<td style="text-align: left;">User must sign into GitHub and authorise the Giscus app.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Allows emoji reactions to articles or comments on articles.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Allows text responses to specific comments.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Markdown compatibility as per usual GitHub issues.</td>
</tr>
<tr class="even">
<td style="text-align: left;">First comment on a page creates a GitHub Discussion, subsequent comments are added to the same Discussion.</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-panel" style="flex-basis: 50.0%;justify-content: center;">
<div id="tbl-second" class="striped hover tabley quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-second-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Cons
</figcaption>
<div aria-describedby="tbl-second-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table class="table-striped table-hover tabley caption-top table">
<colgroup>
<col style="width: 100%">
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: left;">Like Utterances, Giscus revisions are possible only through the GitHub UI. By navigating to the GitHub repo discussions tab, you can edit any post authored by your account.</td>
</tr>
<tr class="even">
<td style="text-align: left;">Changing configuration (e.g., mapping: pathname to mapping: URL) will cause old Giscus comments to be lost from your site. Though they are still available in the repo under the discussions tab.</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
</div>
</div>
</figure>
</div>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>Overall, I felt that Hypothesis offered the greatest functionality. However, it also presents a new service for your audience to subscribe to. If like me, your target audience consists mainly of programmers, then it is likely they will already have access to GitHub accounts and therefore one of the GitHub-specific solutions may present less of a barrier to your audience engaging with your content.</p>
<p>Comparing GitHub Utterances with Giscus, I considered the greater functionality and look of the Giscus UI to be just a little better than Utterances. Therefore I opted for this solution for this site, though I’ll keep it under review incase of breaking changes.</p>
<p>It’s great that the Quarto development team have implemented support for several commenting widgets. Although there are a great variety of alternative solutions available and implementing any of these solutions should be possible as quarto document metadata is fully customisable.</p>
<p>If you spot an error with this article, or have a suggested improvement then feel free to leave a comment (GitHub login required) or <a href="https://github.com/r-leyshon/blogging/issues">raise an issue on GitHub</a>.</p>
<p id="fin">
<i>fin!</i>
</p>


</section>

 ]]></description>
  <category>Explanation</category>
  <category>Quarto</category>
  <category>Comments</category>
  <category>Hypothesis</category>
  <category>GitHub</category>
  <category>Utterances</category>
  <category>Giscus</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/17-quarto-comments.html</guid>
  <pubDate>Sun, 18 Aug 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/17-quarto-comments/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>4 Books That Shaped my Civil Service Career</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/book-reviews/05-career-reads.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/book-reviews/05-career-reads/intro-img.jpg" alt="A man ascends a staircase of books" style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>4 years into my career in the UK civil service, I am about to move to a new department and am taking the time to consider successes, challenges and lessons learned along the way. It’s been an incredible journey. I’ve been lucky to have worked with some of the most talented people and on some fantastic projects.</p>
<p>One of my most valued learning opportunities is in the form of a great book recommendation, especially when someone has taken the time to get to know the kinds of books that I would benefit from reading.</p>
<p>As I anticipate starting in my new role, I’d like to take the opportunity to pay some of these thoughtful recommendations forward. If you’re starting out in the public sector, or considering it, then you may find some of these texts to be helpful.</p>
<p>Some of the books will be useful to a broad audience of civil servants, though others may be more relevant to those who program as part of their job.</p>
</section>
<section id="the-recommendations" class="level2">
<h2 class="anchored" data-anchor-id="the-recommendations">The Recommendations</h2>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Product Placement
</div>
</div>
<div class="callout-body-container callout-body">
<p>This article is not sponsored by anyone. All of the books here have been read and integrated into my working practice to some extent.</p>
</div>
</div>
<p>I will work through a recommendation for each year of service in roughly chronological order - no ranking implied. Two of the recommendations are open source and free to anyone with internet access. The books appeal to distinct audiences, so I’ll try to set out who I think will find the content useful.</p>
<section id="habits-of-highly-effective-people" class="level3">
<h3 class="anchored" data-anchor-id="habits-of-highly-effective-people">1. 7 Habits of Highly Effective People</h3>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/book-reviews/05-career-reads/book_7-habits_600x600.webp" class="center img-fluid figure-img"></p>
<figcaption>7 Habits of Highly Effective People book cover. Image © Franklin Covey.</figcaption>
</figure>
</div>
<p>The Seven Habits of Highly Effective People by Stephen R. Covey <span class="citation" data-cites="Covey">[1]</span>.</p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-2-contents" aria-controls="callout-2" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Who Is It For? (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-2" class="callout-2-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>Just about anyone. But particularly, anyone whose job affords them the responsibility of directing their own time. Though there is also a lot to be applied to your everyday life, too.</p>
</div>
</div>
</div>
<p>I have a bit of a long history with this book. I was initially recommended it when working as a high school science teacher as part of a training event. For some curious reason I found myself resistant to picking it up. I think I felt too busy to dedicate myself to much reading in general at that point in my career and in retrospect, that was a mistake. This book has lots of really useful, actionable guidance for doing more of what matters most.</p>
<p>Starting from a point of personal effectiveness will set you up for success in the civil service, particularly as you develop proficiency in this skill, you may be expected to extend this effectiveness to others through the management of people and projects.</p>
<p>The highlight of the book for me is the famous Covey Quadrant:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/book-reviews/05-career-reads/7_habits_decision-making_matrix.webp" class="center img-fluid figure-img" style="width:40.0%"></p>
<figcaption>Covey quadrant. Image licensed under Creative Commons.</figcaption>
</figure>
</div>
<p>This was adapted from the Eisenhower Matrix, named after the former president who had devised the rubric. Covey’s guidance was to spend time at the start of the day (or even better, at the end of the previous day) triaging your tasks against the dimensions of urgency and importance. And to be mindful that the everyday pressures may push you towards quadrant 1, while the most effective use of your time is generally within quadrant 2.</p>
<p>This approach seemed revelatory to me at the time of reading and would have certainly helped during the early stages in my education career. I readily applied this approach from the outset of my civil service career as a financial analyst and then later as a data science lecturer. As I progressed to managing a faculty of data science lecturers, I had spotted a team in another government department using an interactive team Covey Quadrant. This was a great opportunity to trial the approach with the faculty, who were extremely busy delivering technical training across public sector bodies while also needing to iterate and improve on the service offering.</p>
<p>This tool was exceptionally useful in helping manage demand. Conversations about feasible turnaround and how incoming priorities compared to pre-existing work were well-informed by a quick weekly update to the dashboard’s data (nothing more than updating a couple of columns in an Excel spreadsheet). This tool became my indispensible buddy during that period of high demand and I would thoroughly recommend putting something similar together if you find your team need help managing competing demands.</p>
</section>
<section id="agile-dsdm-project-framework" class="level3">
<h3 class="anchored" data-anchor-id="agile-dsdm-project-framework">2. Agile DSDM Project Framework</h3>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/book-reviews/05-career-reads/agile-pm-square.webp" class="center img-fluid figure-img" style="width:50.0%"></p>
<figcaption>Agile PM logo. Image © Agile Business Consortium Limited.</figcaption>
</figure>
</div>
<p>The Dynamic Systems Development Method <span class="citation" data-cites="dsdm">[2]</span>, an Agile project management framework, published by the Agile Business Consortium Ltd.&nbsp;The handbook is available to read for free on their website.</p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-3-contents" aria-controls="callout-3" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Who Is It For? (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-3" class="callout-3-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>Anyone who needs to design, build or ship products, particularly software. So any of the following (not an exhaustive list):</p>
<ul>
<li>Data Scientist</li>
<li>Data Engineer</li>
<li>Data Analyst</li>
<li>Software Developer</li>
<li>Agile Coach</li>
<li>Project Manager</li>
<li>Scrum Master</li>
<li>Product Owner</li>
<li>Development Team Lead</li>
<li>Software Developer</li>
<li>QA/Test Manager</li>
<li>Change Manager</li>
<li>IT Manager</li>
<li>Program Manager</li>
<li>Business Architect</li>
<li>Business Analyst</li>
<li>System Architect</li>
<li>Technical Architect</li>
<li>Solution Architect</li>
<li>UX/UI Designer</li>
<li>User Researcher</li>
<li>Interaction Designer</li>
<li>Graphic Designer</li>
<li>Portfolio Manager</li>
<li>Digital Marketing Specialist</li>
<li>Security Architect</li>
</ul>
</div>
</div>
</div>
<p>A few years back, Agile seemed to be the in-vogue project management (PM) system, especially for digital products. It always vocal critics, but recently I sense it has crested a wave.</p>
<p>In my opinion, that’s a shame. The core principles of Agile I consider to be worth the effort:</p>
<ul>
<li>Customer-centric development</li>
<li>Iterative delivery</li>
<li>Flat development structure - succeed or fail together</li>
</ul>
<p>A whole industry of consultation has grown around the popularity of Agile DSDM. Finding objective guidance on its implementation can be tricky to find, particularly if you wish to compare and contrast it with other PM frameworks.</p>
<blockquote class="blockquote">
<p>There’s no <strong>right</strong> way to do Agile. It’s whatever you can make work with the resources at your disposal.</p>
</blockquote>
<p>In some guidance, the core Agile principles become a bit obscured by process. An organisation’s interpretation of how Agile works often boils down to a vague understanding of role definitions and sprint ceremonies. If enough of your ‘Agile champions’ leave, you may find yourself with all the process and none of the benefits. I once heard that “Agile simply produces rubbish quickly”. I’d say that if an organisation is careless about its practice, that’s a definite possibility.</p>
<p>The reason I found Agile DSDM so influential, is that I benefited from working with people who really championed it. Colleagues who had grasped the pros and cons and were keen to help the rest of the team navigate the implementation of the Agile philosophy. I had benefited from taking this experience with me to other organisations that had not discovered Agile, and enjoying the benefits of these simple rules.</p>
<div class="quarto-figure quarto-figure-center" style="display:block;margin-left:auto;margin-right:auto;width:70%;">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/book-reviews/05-career-reads/3b-projectvariables-trad.webp" class="img-fluid figure-img"></p>
<figcaption>Project demands payoff. Image © Agile Business Consortium Limited.</figcaption>
</figure>
</div>
<p>I recall one of my directors stating something to the effect of,</p>
<p>“You can have something built quickly and with quality, but have to pay for it. Or it can be built with quality and cheaply, but you’ll have to wait for it. Or it could be built quickly and on a shoestring budget, but it’ll be rubbish.”</p>
<p>I’d say that’s a good interpretation of the Agile priorities. It was very motivating to hear this unsolicited endorsement of Agile principles from the leadership in my division. It felt that adoption of Agile had been firmly embedded and supported throughout all tiers within the hierarchy of my new role. This widespread adoption had contrasted with previous organisations that I had worked for, where Agile had been a new and unproved innovation.</p>
<p>The best Agile teams and practitioners are brutally exact with timescales. The Agile process involves phases of development punctuated by mini evaluation cycles. Feedback from the customers is then used to inform priorities in subsequent development phases. This frequent feedback to and from the customer keeps them highly engaged and giving them greater input into the quality of the final product. When managed well, Agile really works and your customers tell others that it works, too!</p>
</section>
<section id="data-visualisation-a-handbook-for-data-driven-design" class="level3">
<h3 class="anchored" data-anchor-id="data-visualisation-a-handbook-for-data-driven-design">3. Data Visualisation: A Handbook for Data Driven Design</h3>
<p>Data Visualisation: A Handbook for Data Driven Design by Andy Kirk <span class="citation" data-cites="kirk-viz">[3]</span>.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/book-reviews/05-career-reads/81G2OggHvvL.jpg" class="center img-fluid figure-img"></p>
<figcaption>Data Visualisation book cover. Image © Data Viz Excellence, Everywhere Ltd.</figcaption>
</figure>
</div>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-4-contents" aria-controls="callout-4" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Who Is It For? (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-4" class="callout-4-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<ul>
<li>Researchers</li>
<li>Data Analysts</li>
<li>Data Scientists</li>
<li>Data Journalists</li>
<li>Software Developers</li>
<li>Business Intelligence Analysts</li>
<li>Statisticians</li>
<li>Graphic Designers</li>
<li>Communications Professionals</li>
<li>UX/UI Designers</li>
<li>Marketing Analysts</li>
<li>Educators and Trainers</li>
</ul>
</div>
</div>
</div>
<p>I had been recommended this book during my time as a financial analyst working in a multidisciplinary team, some years before joining the civil service. A very talented graphic designer had recommended the book and lent it to me for a period. Some years later, I picked up a copy of the second edition.</p>
<p>This is a fantastic gift for anyone interested in data visualisation. Analysts can become a bit dismissive of data visualisation, considering it nothing more than learning your organisation’s house style. In fact, this ‘done in a day’ attitude can be quite prevalent and quality assurance of charts can be extremely opinionated, placing increasingly narrow guardrails around the kinds of visualisation an organisation would consider publishing. 5 minutes of flicking through this book is enough to blow any conservatism in this space out of the water.</p>
<p>This was the first book that I read where the ‘authoritative view from above’ that can so often be implicit in many visualisations was challenged. Representing uncertainty, discussing the creative process and intended outcomes of a visualisation effort, this book steps the reader from ideation to completion. The author also collates a rich, evocative compendium of fantastic data visualisations, inviting the reader to learn, question and wonder about the various topics they present.</p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Tip
</div>
</div>
<div class="callout-body-container callout-body">
<p>If you’re reading on mobile, the interactive below may look better in landscape.</p>
</div>
</div>
<div style="text-align:center;padding-left:10%;">
<p><iframe style="overflow:hidden;border:0;margin:0;padding:0;width:90%" src="https://pudding.cool/2018/10/city_3d"></iframe></p>
</div>
<p>Straddling the intersection of art, design and analytics, this book provokes analysts to challenge tautologies and to think creatively.</p>
</section>
<section id="ons-duck-book" class="level3">
<h3 class="anchored" data-anchor-id="ons-duck-book">4. ONS Duck Book</h3>
<p><img src="https://best-practice-and-impact.github.io/qa-of-code-guidance/_static/duck_book_logo.svg" class="center" alt="Quality Assurance of Code for Analysis and Research - logo" width="50%"></p>
<p>The Duck Book <span class="citation" data-cites="quac">[4]</span> is the warmly regarded pseudonym for Quality Assurance of Code for Analysis &amp; Research (QuAC!), published by the <a href="https://dataingovernment.blog.gov.uk/2021/03/24/software-engineering-in-analysis/">UK Statistics Authority</a>.</p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-6-contents" aria-controls="callout-6" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Who Is It For? (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-6" class="callout-6-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>Anyone undertaking analysis in Government, here’s a non-exhaustive list:</p>
<ul>
<li>Economist</li>
<li>Operational Research Analyst</li>
<li>Social Researcher</li>
<li>Statistician</li>
<li>Data Scientist</li>
<li>Data Analyst</li>
<li>Data Engineer</li>
<li>Digital, Data and Technology Analyst</li>
<li>Policy Analyst</li>
<li>Evaluation Specialist</li>
<li>Behavioural Scientist</li>
<li>Demographer</li>
<li>Geospatial Analyst</li>
<li>Intelligence Analyst</li>
<li>Risk Analyst</li>
<li>Performance Analyst</li>
<li>Financial Analyst</li>
<li>Business Intelligence Analyst</li>
<li>Market Analyst</li>
<li>Health Analyst</li>
<li>Education Analyst</li>
<li>Environmental Analyst</li>
</ul>
</div>
</div>
</div>
<p>The duck book is a fantastic resource that continues to evolve. It has changed considerably since I first read it in 2020 and now has information relevant to managers of development teams, too.</p>
<p>This book was incredibly useful to me when starting to develop code for the civil service. I had come from smaller development teams in wider public sector organisations. Constraints in those organisations meant that lone-wolf development was commonplace. No code review and too much trust placed on individual analysts to judge the proportionate risk management for multiple projects.</p>
<p>Joining the civil service meant greater collaboration on analytical products. This book became my bible for 12 months while I got to grips with git, GitHub &amp; best practice for collaborative development. It does a great job of giving a high level overview of all the important aspects and quality standards that a government analyst needs to have awareness of, without being overwhelming. You can read it quickly in an hour, yet its guidance is something that I have continually referred. Any developer new to the civil service would benefit from taking the time with this document.</p>
<p>I couldn’t discuss the Duck book without giving a special mention to the curated xkcd comic strips that the team have included at relevant points. It’s rare to get a sense of humour in a civil service publication and very much appreciated.</p>
<div class="quarto-figure quarto-figure-center" style="display:block;margin-left:auto;margin-right:auto;width:30%;">
<figure class="figure">
<p><img src="https://thedatasavvycorner.netlify.app/www/book-reviews/05-career-reads/xkcd-documents.png" class="img-fluid figure-img"></p>
<figcaption>Comic demonstrating poor file naming. Image © XKCD.</figcaption>
</figure>
</div>
</section>
</section>
<section id="special-mentions" class="level2">
<h2 class="anchored" data-anchor-id="special-mentions">Special Mentions</h2>
<p>These books are also great but perhaps have a more specialist readership or I am still in the process of reading and assimilating their concepts:</p>
<ul>
<li>The Turing Way <span class="citation" data-cites="turing-way">[5]</span> for championing open source and reproducible data science.</li>
<li>AQUA book <span class="citation" data-cites="aqua">[6]</span> by Government Analysis Function &amp; H.M. Treasury, guidance on producing quality analysis for government.</li>
<li>Deep Work by Cal Newport: <span class="citation" data-cites="DeepWork">[7]</span> Be more intentional with how you allocate your working day.</li>
<li>Complexity Economics and Sustainable Development: A Computational Framework for Policy Priority Inference by Omar Guerrero and Gonzalo Castañeda <span class="citation" data-cites="ppi">[8]</span>.</li>
</ul>
</section>
<section id="in-summary" class="level2">
<h2 class="anchored" data-anchor-id="in-summary">In Summary</h2>
<p>In this article, I shared four books that influenced my journey in the UK civil service. These books provide a well-rounded toolkit for analysts, offering guidance on personal effectiveness, project management, data visualisation, and code quality assurance. They have been instrumental in shaping my career, and I hope they offer similar value to others embarking on a similar path.</p>



</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body" data-entry-spacing="0">
<div id="ref-Covey" class="csl-entry">
<div class="csl-left-margin">[1] </div><div class="csl-right-inline">Stephen R. Covey, <em><span class="nocase">The Seven Habits of Highly Effective People</span></em>. <span>Free Press</span>, 2004.</div>
</div>
<div id="ref-dsdm" class="csl-entry">
<div class="csl-left-margin">[2] </div><div class="csl-right-inline">Agile Business Consortium Limited, <span>“<span>The DSDM Agile Project Framework</span>.”</span> <a href="{https://www.agilebusiness.org/dsdm-project-framework.html}">{https://www.agilebusiness.org/dsdm-project-framework.html}</a></div>
</div>
<div id="ref-kirk-viz" class="csl-entry">
<div class="csl-left-margin">[3] </div><div class="csl-right-inline">Andy Kirk, <em><span>Data Visualisation</span>: <span class="nocase">A handbook for data driven design</span></em>. <span>SAGE Publications Ltd</span>, 2019.</div>
</div>
<div id="ref-quac" class="csl-entry">
<div class="csl-left-margin">[4] </div><div class="csl-right-inline">Office for National Statistics, <span>“<span class="nocase">Quality Assurance of Code for Analysis and Research</span>.”</span> <a href="{https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html}">{https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html}</a></div>
</div>
<div id="ref-turing-way" class="csl-entry">
<div class="csl-left-margin">[5] </div><div class="csl-right-inline"><span>“<span>The Turing Way</span>.”</span> <a href="{https://book.the-turing-way.org/}">{https://book.the-turing-way.org/}</a></div>
</div>
<div id="ref-aqua" class="csl-entry">
<div class="csl-left-margin">[6] </div><div class="csl-right-inline"><span>“<span class="nocase">The Aqua Book: guidance on producing quality analysis</span>.”</span> <a href="https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government">https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government</a></div>
</div>
<div id="ref-DeepWork" class="csl-entry">
<div class="csl-left-margin">[7] </div><div class="csl-right-inline">Cal Newport, <em><span>Deep Work</span></em>. PIATKUS, 1997.</div>
</div>
<div id="ref-ppi" class="csl-entry">
<div class="csl-left-margin">[8] </div><div class="csl-right-inline">Omar Guerrero and Gonzalo Castañeda, <em><span class="nocase">Complexity Economics and Sustainable Development</span>: <span class="nocase">A Computational Framework for Policy Priority Inference</span></em>. <span>Cambridge University Press</span>, 2024.</div>
</div>
</div></section></div> ]]></description>
  <category>Non-fiction</category>
  <category>Career</category>
  <category>Productivity</category>
  <category>Programming</category>
  <category>Analyst</category>
  <category>Public Sector</category>
  <category>Civil Service</category>
  <guid>https://thedatasavvycorner.netlify.app/book-reviews/05-career-reads.html</guid>
  <pubDate>Sun, 28 Jul 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/book-reviews/05-career-reads/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Custom Marks With Pytest in Plain English</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/16-pytest-marks.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/16-pytest-marks/intro-img.jpg" alt="Futuristic workbench quality controlled ampules." style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<blockquote class="blockquote">
<p>“The only flake I want is that of a croissant.” Mariel Feldman.</p>
</blockquote>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p><code>pytest</code> is a testing package for the python framework. It is broadly used to quality assure code logic. This article discusses custom marks, use cases and patterns for selecting and deselecting marked tests. This blog is the fifth and final in a series of blogs called <a href="../blogs/index.html#category=pytest-in-plain-english">pytest in plain English</a>, favouring accessible language and simple examples to explain the more intricate features of the <code>pytest</code> package.</p>
<p>For a wealth of documentation, guides and how-tos, please consult the <a href="https://docs.pytest.org/en/8.0.x/" target="_blank"><code>pytest</code> documentation</a>.</p>
<section id="what-are-pytest-custom-marks" class="level3">
<h3 class="anchored" data-anchor-id="what-are-pytest-custom-marks">What are <code>pytest</code> Custom Marks?</h3>
<p>Marks are a way to conditionally run specific tests. There are a few marks that come with the <code>pytest</code> package. To view these, run <code>pytest --markers</code> from the command line. This will print a list of the pre-registered marks available within the <code>pytest</code> package. However, it is extremely easy to register your own markers, allowing greater control over which tests get executed.</p>
<p>This article will cover:</p>
<ul>
<li>Reasons for marking tests</li>
<li>Registering marks with <code>pytest</code></li>
<li>Marking tests</li>
<li>Including or excluding markers from the command line</li>
</ul>
<div class="callout callout-style-simple callout-none no-icon callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-1-contents" aria-controls="callout-1" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon no-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">None</span>A Note on the Purpose (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-1" class="callout-1-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>This article intends to discuss clearly. It doesn’t aim to be clever or impressive. Its aim is to extend understanding without overwhelming the reader. The code may not always be optimal, favouring a simplistic approach wherever possible.</p>
</div>
</div>
</div>
</section>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Programmers with a working knowledge of python and some familiarity with <code>pytest</code> and packaging. The type of programmer who has wondered about how to follow best practice in testing python code.</p>
</section>
<section id="what-youll-need" class="level3">
<h3 class="anchored" data-anchor-id="what-youll-need">What You’ll Need:</h3>
<ul class="task-list">
<li><label><input type="checkbox">Preferred python environment manager (eg <code>conda</code>)</label></li>
<li><label><input type="checkbox"><code>pip install pytest==8.1.1</code></label></li>
<li><label><input type="checkbox">Git</label></li>
<li><label><input type="checkbox">GitHub account</label></li>
<li><label><input type="checkbox">Command line access</label></li>
</ul>
</section>
<section id="preparation" class="level3">
<h3 class="anchored" data-anchor-id="preparation">Preparation</h3>
<p>This blog is accompanied by code in <a href="https://github.com/r-leyshon/pytest-fiddly-examples">this repository</a>. The main branch provides a template with the minimum structure and requirements expected to run a <code>pytest</code> suite. The repo branches contain the code used in the examples of the following sections.</p>
<p>Feel free to fork or clone the repo and checkout to the example branches as needed.</p>
<p>The example code that accompanies this article is available in the <a href="https://github.com/r-leyshon/pytest-fiddly-examples/tree/marks">marks branch</a> of the repo.</p>
</section>
</section>
<section id="overview" class="level2">
<h2 class="anchored" data-anchor-id="overview">Overview</h2>
<p>Occasionally, we write tests that are a bit distinct to the rest of our test suite. They could be integration tests, calling on elements of our code from multiple modules. They could be end to end tests, executing a pipeline from start to finish. Or they could be a flaky or brittle sort of test, a test that is prone to failure on specific operating systems, architectures or when external dependencies do not provide reliable inputs.</p>
<p>There are multiple ways to handle these kinds of tests, including mocking, as discussed in <a href="../blogs/15-pytest-mocking.html">my previous blog</a>. Mocking can often take a bit of time, and developers don’t always have that precious commodity. So instead, they may mark the test, ensuring that it doesn’t get run on continuous integration (CI) checks. This may involve flagging any flaky test as “technical debt” to be investigated and fixed later.</p>
<p>In fact, there are a number of reasons that we may want to selectively run elements of a test suite. Here is a selection of scenarios that could benefit from marking.</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-2-contents" aria-controls="callout-2" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Flaky Tests: Common Causes
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-2" class="callout-2-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<div class="table-responsive-md">
<table class="table-light table-hover caption-top table">
<colgroup>
<col style="width: 22%">
<col style="width: 22%">
<col style="width: 55%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Category</strong></th>
<th><strong>Cause</strong></th>
<th><strong>Explanation</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>External Dependencies</td>
<td>Network</td>
<td>Network latency, outages, or Domain Name System (DNS) issues.</td>
</tr>
<tr class="even">
<td></td>
<td>Web APIs</td>
<td>Breaking changes, rate limits, or outages.</td>
</tr>
<tr class="odd">
<td></td>
<td>Databases</td>
<td>Concurrency issues, data changes, or connection problems.</td>
</tr>
<tr class="even">
<td></td>
<td>Timeouts</td>
<td>Hardcoded or too-short timeouts cause failures.</td>
</tr>
<tr class="odd">
<td>Environment Dependencies</td>
<td>Environment Variables</td>
<td>Incorrectly set environment variables.</td>
</tr>
<tr class="even">
<td></td>
<td>File System</td>
<td>File locks, permissions, or missing files.</td>
</tr>
<tr class="odd">
<td></td>
<td>Resource Limits</td>
<td>Insufficient CPU, memory, or disk space.</td>
</tr>
<tr class="even">
<td>State Dependencies</td>
<td>Shared State</td>
<td>Interference between tests sharing state.</td>
</tr>
<tr class="odd">
<td></td>
<td>Order Dependency</td>
<td>Tests relying on execution order.</td>
</tr>
<tr class="even">
<td>Test Data</td>
<td>Random Data</td>
<td>Different results on each run due to random data and seed not set.</td>
</tr>
<tr class="odd">
<td>Concurrency Issues</td>
<td>Parallel Execution</td>
<td>Tests not designed for parallel execution.</td>
</tr>
<tr class="even">
<td></td>
<td>Locks</td>
<td>Deadlocks or timeouts involving locks or semaphores.</td>
</tr>
<tr class="odd">
<td></td>
<td>Race Conditions</td>
<td>Tests depend on the order of execution of threads or processes.</td>
</tr>
<tr class="even">
<td></td>
<td>Async Operations</td>
<td>Improperly awaited asynchronous code.</td>
</tr>
<tr class="odd">
<td>Hardware and System Issues</td>
<td>Differences in Hardware</td>
<td>Variations in performance across hardware or operating systems.</td>
</tr>
<tr class="even">
<td></td>
<td>System Load</td>
<td>Failures under high system load due to resource contention.</td>
</tr>
<tr class="odd">
<td>Non-deterministic Inputs</td>
<td>Time</td>
<td>Variations in current time affecting test results.</td>
</tr>
<tr class="even">
<td></td>
<td>User Input</td>
<td>Non-deterministic user input causing flaky behaviour.</td>
</tr>
<tr class="odd">
<td></td>
<td>Filepaths</td>
<td>CI runner filepaths may be hard to predict.</td>
</tr>
<tr class="even">
<td>Test Implementation Issues</td>
<td>Assertions</td>
<td>Incorrect or overly strict assertions.</td>
</tr>
<tr class="odd">
<td></td>
<td>Setup and Teardown</td>
<td>Inconsistent state due to improper setup or teardown.</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<p>In the case of integration tests, one approach may be to group them all together and have them execute within a dedicated CI workflow. This is common practice as developers may want to stay alert to problems with external resources that their code depends upon, while not failing ‘core’ CI checks about changes to the source code. If your code relies on a web API; for instance; you’re probably less concerned about temporary outages in that service. However, a breaking change to that service would require our source code to be adapted. Once more, life is a compromise.</p>
<blockquote class="blockquote">
<p>“<em>Le mieux est l’ennemi du bien.</em>” (The best is the enemy of the good), Voltaire</p>
</blockquote>
</section>
<section id="custom-marks-in-pytest" class="level2">
<h2 class="anchored" data-anchor-id="custom-marks-in-pytest">Custom Marks in <code>pytest</code></h2>
<p>Marking allows us to have greater control over which of our tests are executed when we invoke <code>pytest</code>. Marking is conveniently implemented in the following way (presuming you have already written your source and test code):</p>
<ol type="1">
<li>Register a custom marker</li>
<li>Assign the new marker name to the target test</li>
<li>Invoke <code>pytest</code> with the <code>-m</code> (MARKEXPR) flag.</li>
</ol>
<p>This section uses code available in the <a href="https://github.com/r-leyshon/pytest-fiddly-examples/tree/marks">marks branch</a> of the GitHub repository.</p>
<section id="define-the-source-code" class="level3">
<h3 class="anchored" data-anchor-id="define-the-source-code">Define the Source Code</h3>
<p>I have a motley crew of functions to consider. A sort of homage to Sergio Leone’s ‘The Good, The Bad &amp; the Ugly’, although I’ll let you figure out which is which.</p>
<img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/16-pytest-marks/cowboy-bots.jpg" alt="The Flaky, The Slow &amp; The Needy" style="display:block;margin-left:auto;margin-right:auto;width:400px;">

<section id="the-flaky-function" class="level4">
<h4 class="anchored" data-anchor-id="the-flaky-function">The Flaky Function</h4>
<p>Here we define a function that will fail half the time. What a terrible test to have. The root of this unpredictable behaviour should be diagnosed as a priority as a matter of sanity.</p>
<div id="7fa4ce31" class="cell" data-execution_count="1">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> random</span>
<span id="cb1-2"></span>
<span id="cb1-3"></span>
<span id="cb1-4"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> croissant():</span>
<span id="cb1-5">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""A very flaky function."""</span></span>
<span id="cb1-6">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">round</span>(random.uniform(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>:</span>
<span id="cb1-7">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span></span>
<span id="cb1-8">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb1-9">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">raise</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">Exception</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Flaky test detected!"</span>)</span></code></pre></div></div>
</div>
</section>
<section id="the-slow-function" class="level4">
<h4 class="anchored" data-anchor-id="the-slow-function">The Slow Function</h4>
<p>This function is going to be pretty slow. Slow test suites throttle our productivity. Once it finishes waiting for a specified number of seconds, it will return a string.</p>
<div id="d039d681" class="cell" data-execution_count="2">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb2-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> time</span>
<span id="cb2-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> typing <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Union</span>
<span id="cb2-3"></span>
<span id="cb2-4"></span>
<span id="cb2-5"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> take_a_nap(how_many_seconds:Union[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">float</span>]) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="cb2-6">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Mimic a costly function by just doing nothing for a specified time."""</span></span>
<span id="cb2-7">    time.sleep(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">float</span>(how_many_seconds))</span>
<span id="cb2-8">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rise and shine!"</span></span></code></pre></div></div>
</div>
</section>
<section id="the-needy-function" class="level4">
<h4 class="anchored" data-anchor-id="the-needy-function">The Needy Function</h4>
<p>Finally, the needy function will have an external dependency on a website. This test will simply check whether we get a HTTP status code of 200 (ok) when we request any URL.</p>
<div id="3d6f4c38" class="cell" data-execution_count="3">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb3-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb3-2"></span>
<span id="cb3-3"></span>
<span id="cb3-4"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> check_site_available(url:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, timeout:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">bool</span>:</span>
<span id="cb3-5">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Checks if a site is available."""</span></span>
<span id="cb3-6">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="cb3-7">        response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.get(url, timeout<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>timeout)</span>
<span id="cb3-8">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> response.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span></span>
<span id="cb3-9">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> requests.RequestException:</span>
<span id="cb3-10">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span></span></code></pre></div></div>
</div>
</section>
<section id="the-wrapper" class="level4">
<h4 class="anchored" data-anchor-id="the-wrapper">The Wrapper</h4>
<p>Finally, I’ll introduce a wrapper that will act as an opportunity for an integration test. This is a bit awkward, as none of the above functions are particularly related to each other.</p>
<p>This function will execute the <code>check_site_available()</code> and <code>take_a_nap()</code> together. A pretty goofy example, I admit. Based on the status of the url request, a string will be returned.</p>
<div id="a9aafac1" class="cell" data-execution_count="4">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> time</span>
<span id="cb4-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> typing <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Union</span>
<span id="cb4-3"></span>
<span id="cb4-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb4-5"></span>
<span id="cb4-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> goofy_wrapper(url:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, timeout:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="cb4-7">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Check a site is available, pause for no good reason before summarising</span></span>
<span id="cb4-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    outcome with a string."""</span></span>
<span id="cb4-9">    msg <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Napping for </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>timeout<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> seconds.</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb4-10">    msg <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> msg <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> take_a_nap(timeout)</span>
<span id="cb4-11">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> check_site_available(url):</span>
<span id="cb4-12">        msg <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> msg <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Your site is up!"</span></span>
<span id="cb4-13">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb4-14">        msg <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> msg <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Your site is down!"</span></span>
<span id="cb4-15"></span>
<span id="cb4-16">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> msg</span></code></pre></div></div>
</div>
</section>
</section>
<section id="lets-get-testing" class="level3">
<h3 class="anchored" data-anchor-id="lets-get-testing">Let’s Get Testing</h3>
<p>Initially, I will define a test that does nothing other than pass. This will be a placeholder, unmarked test.</p>
<div id="c2f9ae06" class="cell" data-execution_count="5">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb5-1"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_nothing():</span>
<span id="cb5-2">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">pass</span></span></code></pre></div></div>
</div>
<p>Next, I import <code>croissant()</code> and assert that it returns <code>True</code>. As you may recall from above, <code>croissant()</code> will do so ~50 % of the time.</p>
<div id="092b973f" class="cell" data-execution_count="6">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb6-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.do_something <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> (</span>
<span id="cb6-2">    croissant,</span>
<span id="cb6-3">    )</span>
<span id="cb6-4"></span>
<span id="cb6-5"></span>
<span id="cb6-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_nothing():</span>
<span id="cb6-7">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">pass</span></span>
<span id="cb6-8"></span>
<span id="cb6-9"></span>
<span id="cb6-10"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_croissant():</span>
<span id="cb6-11">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> croissant()</span></code></pre></div></div>
</div>
<p>Now running <code>pytest -v</code> will print the test results, reporting test outcomes for each test separately (<code>-v</code> means verbose).</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb7-1">...<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v</span></span>
<span id="cb7-2">============================= test session starts =============================</span>
<span id="cb7-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb7-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb7-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb7-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb7-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb7-8">collected 2 items                                                             </span>
<span id="cb7-9">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_nothing</span> PASSED                          <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 50<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb7-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>croissant PASSED                        <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb7-11">============================== 2 passed in 0.05s ==============================</span></code></pre></div></div>
<p>But note that half the time, I will also get the following output:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb8-1">...<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v</span></span>
<span id="cb8-2">============================= test session starts =============================</span>
<span id="cb8-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb8-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb8-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb8-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb8-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb8-8">collected 2 items                         </span>
<span id="cb8-9"></span>
<span id="cb8-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_nothing</span> PASSED                          <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 50<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb8-11">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>croissant FAILED                        <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb8-12"></span>
<span id="cb8-13">================================== FAILURES ==================================</span>
<span id="cb8-14">_______________________________ test_croissant ________________________________</span>
<span id="cb8-15">    @pytest.mark.flaky</span>
<span id="cb8-16">    def test_croissant<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">()</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="cb8-17">&gt;       assert croissant<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">()</span></span>
<span id="cb8-18">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:1</span>7<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> </span>
<span id="cb8-19">_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _</span>
<span id="cb8-20">    def croissant<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">()</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="cb8-21">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"""A very flaky function."""</span></span>
<span id="cb8-22">        if round<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">(</span>random.uniform<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">(</span>0, 1<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">))</span> == 1<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="cb8-23">            return True</span>
<span id="cb8-24">        else<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span></span>
<span id="cb8-25">&gt;           raise Exception<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Flaky test detected!"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">)</span></span>
<span id="cb8-26">E           Exception<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Flaky test detected!</span>
<span id="cb8-27"></span>
<span id="cb8-28">src/example_pkg/do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:1</span>3<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Exception</span>
<span id="cb8-29">============================short test summary info ===========================</span>
<span id="cb8-30">FAILED ...<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>croissant - Exception<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> Flaky test detected!</span>
<span id="cb8-31">========================= 1 failed, 1 passed in 0.07s =========================</span></code></pre></div></div>
<p>To prevent this flaky test from failing the test suite, we can choose to mark it as flaky, and optionally skip it when invoking <code>pytest</code>. To go about that, we first need to register a new marker. To do that, let’s update out project’s <code>pyproject.toml</code> to include additional options for a <code>flaky</code> mark:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb9-1"># `pytest` configurations</span>
<span id="cb9-2"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[tool.pytest.ini_options]</span></span>
<span id="cb9-3">markers = <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span></span>
<span id="cb9-4">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"flaky: tests that can randomly fail through no change to the code"</span>,</span>
<span id="cb9-5"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
<p>Note that when registering a marker in this way, text after the colon is an optional mark description. Saving the document and running <code>pytest --markers</code> should show that a new custom marker is available to our project:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb10-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest --markers</span></span>
<span id="cb10-2">@pytest.mark.flaky<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> tests that can randomly fail through no change to the code</span>
<span id="cb10-3">...</span></code></pre></div></div>
<p>Now that we have confirmed our marker is available for use, we can use it to mark <code>test_croissant()</code> as flaky:</p>
<div id="a44e76c0" class="cell" data-execution_count="7">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb11-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb11-2"></span>
<span id="cb11-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.do_something <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> (</span>
<span id="cb11-4">    croissant,</span>
<span id="cb11-5">    )</span>
<span id="cb11-6"></span>
<span id="cb11-7"></span>
<span id="cb11-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_nothing():</span>
<span id="cb11-9">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">pass</span></span>
<span id="cb11-10"></span>
<span id="cb11-11"></span>
<span id="cb11-12"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.flaky</span></span>
<span id="cb11-13"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_croissant():</span>
<span id="cb11-14">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> croissant()</span></code></pre></div></div>
</div>
<p>Note that we need to import <code>pytest</code> to our test module in order to use the <code>pytest.mark.&lt;MARK_NAME&gt;</code> decorator.</p>
<section id="selecting-a-single-mark" class="level4">
<h4 class="anchored" data-anchor-id="selecting-a-single-mark">Selecting a Single Mark</h4>
<p>Now that we have registered and marked a test as <code>flaky</code>, we can adapt our <code>pytest</code> call to execute tests with that mark only. The pattern we will use is:</p>
<blockquote class="blockquote">
<p><code>pytest -v -m "&lt;INSERT_MARK_NAME&gt;"</code></p>
</blockquote>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb12-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v -m "flaky"</span></span>
<span id="cb12-2">============================= test session starts =============================</span>
<span id="cb12-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb12-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb12-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb12-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb12-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb12-8">collected 2 items / 1 deselected / 1 selected                                 </span>
<span id="cb12-9"></span>
<span id="cb12-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>croissant PASSED                        <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb12-11"></span>
<span id="cb12-12">======================= 1 passed, 1 deselected in 0.05s =======================</span></code></pre></div></div>
<p>Now we see that <code>test_croissant()</code> was executed, while the unmarked <code>test_nothing()</code> was not.</p>
</section>
<section id="deselecting-a-single-mark" class="level4">
<h4 class="anchored" data-anchor-id="deselecting-a-single-mark">Deselecting a Single Mark</h4>
<p>More useful than selectively running a flaky test is to deselect it. In this way, it cannot fail our test suite. This is achieved with the following pattern:</p>
<blockquote class="blockquote">
<p><code>pytest -v -m "not &lt;INSERT_MARK_NAME&gt;"</code></p>
</blockquote>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb13-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v -m "not flaky"</span></span>
<span id="cb13-2">============================= test session starts =============================</span>
<span id="cb13-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb13-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb13-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb13-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb13-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb13-8">collected 2 items / 1 deselected / 1 selected                                  </span>
<span id="cb13-9"></span>
<span id="cb13-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_nothing</span> PASSED                          <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb13-11"></span>
<span id="cb13-12">======================= 1 passed, 1 deselected in 0.05s =======================</span></code></pre></div></div>
<p>Note that this time, <code>test_flaky()</code> was not executed.</p>
</section>
<section id="selecting-multiple-marks" class="level4">
<h4 class="anchored" data-anchor-id="selecting-multiple-marks">Selecting Multiple Marks</h4>
<p>In this section, we will introduce another, differently marked test to illustrate the syntax for running multiple marks. For this example, we’ll test <code>take_a_nap()</code>:</p>
<div id="7c174745" class="cell" data-execution_count="8">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb14-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb14-2"></span>
<span id="cb14-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.do_something <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> (</span>
<span id="cb14-4">    croissant,</span>
<span id="cb14-5">    take_a_nap,</span>
<span id="cb14-6">    )</span>
<span id="cb14-7"></span>
<span id="cb14-8"></span>
<span id="cb14-9"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_nothing():</span>
<span id="cb14-10">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">pass</span></span>
<span id="cb14-11"></span>
<span id="cb14-12"></span>
<span id="cb14-13"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.flaky</span></span>
<span id="cb14-14"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_croissant():</span>
<span id="cb14-15">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> croissant()</span>
<span id="cb14-16"></span>
<span id="cb14-17"></span>
<span id="cb14-18"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.slow</span></span>
<span id="cb14-19"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_take_a_nap():</span>
<span id="cb14-20">    out <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> take_a_nap(how_many_seconds<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span>
<span id="cb14-21">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">isinstance</span>(out, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>), <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"a string was not returned: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span>(out)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb14-22">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> out <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rise and shine!"</span>, <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"unexpected string pattern: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>out<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span></code></pre></div></div>
</div>
<p>Our new test just makes some simple assertions about the string <code>take_a_nap()</code> returns after snoozing. But notice what happens when running <code>pytest -v</code> now:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb15-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v</span></span>
<span id="cb15-2">============================= test session starts =============================</span>
<span id="cb15-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb15-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb15-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb15-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb15-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb15-8">collected 3 items                                                             </span>
<span id="cb15-9"></span>
<span id="cb15-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_nothing</span> PASSED                          <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 33<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb15-11">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>croissant PASSED                        <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 66<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb15-12">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_take_</span>a_nap PASSED                       <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb15-13"></span>
<span id="cb15-14">============================== 3 passed in 3.07s ==============================</span></code></pre></div></div>
<p>The test suite now takes in excess of 3 seconds to execute, as the test specified for <code>take_a_nap()</code> to sleep for that period. Let’s update our <code>pyproject.toml</code> and register a new mark:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb16" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb16-1"># `pytest` configurations</span>
<span id="cb16-2"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[tool.pytest.ini_options]</span></span>
<span id="cb16-3">markers = <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span></span>
<span id="cb16-4">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"flaky: tests that can randomly fail through no change to the code"</span>,</span>
<span id="cb16-5">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"slow: marks tests as slow (deselect with '-m \"</span>not slow\<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"')"</span>,</span>
<span id="cb16-6"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
<p>Note that the nested speech marks within the description of the <code>slow</code> mark were escaped. <code>pytest</code> would have complained that the toml file was not valid unless we ensured it was valid <a href="https://toml.io/en/">toml syntax</a>.</p>
<p>In order to run tests marked with either <code>flaky</code> or <code>slow</code>, we can use <code>or</code>:</p>
<blockquote class="blockquote">
<p><code>pytest -v -m "&lt;INSERT_MARK_1&gt; or &lt;INSERT_MARK_2&gt;"</code></p>
</blockquote>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb17" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb17-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v -m "flaky or slow"</span></span>
<span id="cb17-2">============================= test session starts =============================</span>
<span id="cb17-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb17-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb17-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb17-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb17-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb17-8">collected 3 items / 1 deselected / 2 selected                                 </span>
<span id="cb17-9"></span>
<span id="cb17-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>croissant PASSED                        <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 50<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb17-11">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_take_</span>a_nap PASSED                       <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb17-12"></span>
<span id="cb17-13">======================= 2 passed, 1 deselected in 3.06s =======================</span></code></pre></div></div>
<p>Note that anything not marked with <code>flaky</code> or <code>slow</code> (eg <code>test_nothing()</code>) was not run. Also, <code>test_croissant()</code> failed 3 times in a row while I tried to get a passing run. I didn’t want the flaky exception to carry on presenting itself. While I may be sprinkling glitter, I do not want to misrepresent how frustrating flaky tests can be!</p>
<p><img src="https://thedatasavvycorner.netlify.app/www/16-pytest-marks/glitter-optimized.gif" alt="Glitter animation" style="display:block;margin-left:auto;margin-right:auto;width:320px;"></p>
</section>
<section id="complex-selection-rules" class="level4">
<h4 class="anchored" data-anchor-id="complex-selection-rules">Complex Selection Rules</h4>
<p>By adding an additional mark, we can illustrate more complex selection and deselection rules for invoking <code>pytest</code>. Let’s write an integration test that checks whether the domain for this blog site can be reached.</p>
<div id="42070b82" class="cell" data-execution_count="9">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb18" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb18-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb18-2"></span>
<span id="cb18-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.do_something <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> (</span>
<span id="cb18-4">    croissant,</span>
<span id="cb18-5">    take_a_nap,</span>
<span id="cb18-6">    check_site_available,</span>
<span id="cb18-7">    )</span>
<span id="cb18-8"></span>
<span id="cb18-9"></span>
<span id="cb18-10"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_nothing():</span>
<span id="cb18-11">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">pass</span></span>
<span id="cb18-12"></span>
<span id="cb18-13"></span>
<span id="cb18-14"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.flaky</span></span>
<span id="cb18-15"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_croissant():</span>
<span id="cb18-16">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> croissant()</span>
<span id="cb18-17"></span>
<span id="cb18-18"></span>
<span id="cb18-19"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.slow</span></span>
<span id="cb18-20"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_take_a_nap():</span>
<span id="cb18-21">    out <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> take_a_nap(how_many_seconds<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span>
<span id="cb18-22">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">isinstance</span>(out, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>), <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"a string was not returned: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span>(out)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb18-23">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> out <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rise and shine!"</span>, <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"unexpected string pattern: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>out<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb18-24"></span>
<span id="cb18-25"></span>
<span id="cb18-26"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.integration</span></span>
<span id="cb18-27"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_check_site_available():</span>
<span id="cb18-28">    url <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://thedatasavvycorner.com/"</span></span>
<span id="cb18-29">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> check_site_available(url), <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"site </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>url<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> is down..."</span></span></code></pre></div></div>
</div>
<p>Now updating our <code>pyproject.toml</code> like so:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb19" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb19-1"># `pytest` configurations</span>
<span id="cb19-2"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[tool.pytest.ini_options]</span></span>
<span id="cb19-3">markers = <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span></span>
<span id="cb19-4">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"flaky: tests that can randomly fail through no change to the code"</span>,</span>
<span id="cb19-5">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"slow: marks tests as slow (deselect with '-m \"</span>not slow\<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"')"</span>,</span>
<span id="cb19-6">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"integration: tests that require external resources"</span>,</span>
<span id="cb19-7"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
<p>Now we can combine <code>and</code> and <code>not</code> statements when calling <code>pytest</code> to execute just the tests we need to. In the below, I choose to run the <code>slow</code> and <code>integration</code> tests while excluding that pesky <code>flaky</code> test.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb20" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb20-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v -m "slow or integration and not flaky"</span></span>
<span id="cb20-2">============================= test session starts =============================</span>
<span id="cb20-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb20-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb20-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb20-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb20-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb20-8">collected 4 items / 2 deselected / 2 selected                                 </span>
<span id="cb20-9"></span>
<span id="cb20-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_take_</span>a_nap PASSED                       <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 50<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb20-11">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>check_site_available PASSED             <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb20-12"></span>
<span id="cb20-13">======================= 2 passed, 2 deselected in 3.29s =======================</span></code></pre></div></div>
<p>Note that both <code>test_nothing()</code> (unmarked) and <code>test_croissant()</code> (deselected) were not run.</p>
</section>
<section id="marks-and-test-classes" class="level4">
<h4 class="anchored" data-anchor-id="marks-and-test-classes">Marks and Test Classes</h4>
<p>Note that so far, we have applied marks to test functions only. But we can also apply marks to an entire test class, or even target specific test modules. For this section, I will introduce the wrapper function introduced earlier and use a test class to group its tests together. I will mark those tests with 2 new marks, <code>classy</code> and <code>subclassy</code>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb21" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb21-1"># `pytest` configurations</span>
<span id="cb21-2"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[tool.pytest.ini_options]</span></span>
<span id="cb21-3">markers = <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span></span>
<span id="cb21-4">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"flaky: tests that can randomly fail through no change to the code"</span>,</span>
<span id="cb21-5">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"slow: marks tests as slow (deselect with '-m \"</span>not slow\<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"')"</span>,</span>
<span id="cb21-6">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"integration: tests that require external resources"</span>,</span>
<span id="cb21-7">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"classy: tests arranged in a class"</span>,</span>
<span id="cb21-8">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"subclassy: test methods"</span>,</span>
<span id="cb21-9"><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
<p>Updating our test module to include <code>test_goofy_wrapper()</code>:</p>
<div id="3cd8b08b" class="cell" data-execution_count="10">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb22" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb22-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb22-2"></span>
<span id="cb22-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.do_something <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> (</span>
<span id="cb22-4">    croissant,</span>
<span id="cb22-5">    take_a_nap,</span>
<span id="cb22-6">    check_site_available,</span>
<span id="cb22-7">    goofy_wrapper</span>
<span id="cb22-8">    )</span>
<span id="cb22-9"></span>
<span id="cb22-10"></span>
<span id="cb22-11"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_nothing():</span>
<span id="cb22-12">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">pass</span></span>
<span id="cb22-13"></span>
<span id="cb22-14"></span>
<span id="cb22-15"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.flaky</span></span>
<span id="cb22-16"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_croissant():</span>
<span id="cb22-17">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> croissant()</span>
<span id="cb22-18"></span>
<span id="cb22-19"></span>
<span id="cb22-20"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.slow</span></span>
<span id="cb22-21"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_take_a_nap():</span>
<span id="cb22-22">    out <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> take_a_nap(how_many_seconds<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span>
<span id="cb22-23">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">isinstance</span>(out, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>), <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"a string was not returned: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span>(out)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb22-24">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> out <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rise and shine!"</span>, <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"unexpected string pattern: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>out<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb22-25"></span>
<span id="cb22-26"></span>
<span id="cb22-27"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.integration</span></span>
<span id="cb22-28"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_check_site_available():</span>
<span id="cb22-29">    url <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://thedatasavvycorner.com/"</span></span>
<span id="cb22-30">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> check_site_available(url), <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"site </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>url<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> is down..."</span></span>
<span id="cb22-31"></span>
<span id="cb22-32"></span>
<span id="cb22-33"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.classy</span></span>
<span id="cb22-34"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">class</span> TestGoofyWrapper:</span>
<span id="cb22-35">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.subclassy</span></span>
<span id="cb22-36">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_goofy_wrapper_url_exists(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>):</span>
<span id="cb22-37">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> goofy_wrapper(</span>
<span id="cb22-38">            <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://thedatasavvycorner.com/"</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span></span>
<span id="cb22-39">            ).endswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Your site is up!"</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"The site wasn't up."</span></span>
<span id="cb22-40">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.subclassy</span></span>
<span id="cb22-41">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_goofy_wrapper_url_does_not_exist(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>):</span>
<span id="cb22-42">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> goofy_wrapper(</span>
<span id="cb22-43">            <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://thegoofycorner.com/"</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span></span>
<span id="cb22-44">            ).endswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Your site is down!"</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"The site wasn't down."</span></span></code></pre></div></div>
</div>
<p>Note that targeting either the <code>classy</code> or <code>subclassy</code> mark results in the same output - all tests within this test class are executed:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb23" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb23-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v -m "classy"</span></span>
<span id="cb23-2">============================= test session starts =============================</span>
<span id="cb23-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb23-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb23-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb23-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb23-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb23-8">collected 6 items / 4 deselected / 2 selected                                 </span>
<span id="cb23-9"></span>
<span id="cb23-10">TestGoofyWrapper<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>goofy_wrapper_url_exists PASSED                   <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 50<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb23-11">TestGoofyWrapper<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>goofy_wrapper_url_does_not_exist PASSED           <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb23-12"></span>
<span id="cb23-13">======================= 2 passed, 4 deselected in 2.30s =======================</span></code></pre></div></div>
<p>Nobody created the domain <code>https://thegoofycorner.com/</code> yet, such a shame.</p>
</section>
<section id="tests-with-multiple-marks" class="level4">
<h4 class="anchored" data-anchor-id="tests-with-multiple-marks">Tests with Multiple Marks</h4>
<p>Note that we can use multiple marks with any test or test class. Let’s update <code>TestGoofyWrapper</code> to be marked as <code>integration</code> &amp; <code>slow</code>:</p>
<div id="d88ecb6c" class="cell" data-execution_count="11">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb24" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb24-1"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.slow</span></span>
<span id="cb24-2"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.integration</span></span>
<span id="cb24-3"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.classy</span></span>
<span id="cb24-4"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">class</span> TestGoofyWrapper:</span>
<span id="cb24-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.subclassy</span></span>
<span id="cb24-6">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_goofy_wrapper_url_exists(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>):</span>
<span id="cb24-7">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> goofy_wrapper(</span>
<span id="cb24-8">            <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://thedatasavvycorner.com/"</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span></span>
<span id="cb24-9">            ).endswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Your site is up!"</span>),<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"The site wasn't up."</span></span>
<span id="cb24-10">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.mark.subclassy</span></span>
<span id="cb24-11">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_goofy_wrapper_url_does_not_exist(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>):</span>
<span id="cb24-12">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> goofy_wrapper(</span>
<span id="cb24-13">            <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://thegoofycorner.com/"</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span></span>
<span id="cb24-14">            ).endswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Your site is down!"</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"The site wasn't down."</span></span></code></pre></div></div>
</div>
<p>This test class can now be exclusively targeted by specifying multiple marks with <code>and</code>:</p>
<blockquote class="blockquote">
<p><code>pytest -v -m "&lt;INSERT_MARK_1&gt; and &lt;INSERT_MARK2&gt;... and &lt;INSERT_MARK_N&gt;"</code></p>
</blockquote>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb25" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb25-1"></span>
<span id="cb25-2">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v -m "integration and slow"   </span></span>
<span id="cb25-3">============================= test session starts =============================</span>
<span id="cb25-4">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...</span>
<span id="cb25-5">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb25-6">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb25-7">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb25-8">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb25-9">collected 6 items / 4 deselected / 2 selected                                 </span>
<span id="cb25-10"></span>
<span id="cb25-11">TestGoofyWrapper<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>goofy_wrapper_url_exists PASSED                   <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span> 50<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb25-12">TestGoofyWrapper<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_</span>goofy_wrapper_url_does_not_exist PASSED           <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb25-13"></span>
<span id="cb25-14">======================= 2 passed, 4 deselected in 2.30s =======================</span></code></pre></div></div>
<p>Note that even though there are other tests marked with <code>integration</code> and <code>slow</code> separately, they are excluded on the basis that <code>and</code> expects them to be marked with both.</p>
</section>
<section id="deselecting-all-marks" class="level4">
<h4 class="anchored" data-anchor-id="deselecting-all-marks">Deselecting All Marks</h4>
<p>Now that we have introduced multiple custom markers to our test suite, what if we want to exclude all of these marked tests, just running the ‘core’ test suite? Unfortunately, there is not a way to specify ‘unmarked’ tests. There is an old <code>pytest</code> plugin called <a href="https://pypi.org/project/pytest-unmarked/"><code>pytest-unmarked</code></a> that allowed this functionality. Unfortunately, this plugin is not being actively maintained and is not compatible with <code>pytest</code> v8.0.0+. You could introduce a ‘standard’ or ‘core’ marker, but you’d need to remember to mark every unmarked test within your test suite with it.</p>
<p>Alternatively, what we can do is exclude each of the marks that have been registered. There are 2 patterns for achieving this:</p>
<blockquote class="blockquote">
<ol type="1">
<li><code>pytest -v -m "not &lt;INSERT_MARK_1&gt; ... or not &lt;INSERT_MARK_N&gt;"</code></li>
<li><code>pytest -v -m "not (&lt;INSERT_MARK_1&gt; ... or &lt;INSERT_MARK_N&gt;)"</code></li>
</ol>
</blockquote>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb26" style="background: #f1f3f5;"><pre class="sourceCode abc code-with-copy"><code class="sourceCode abc"><span id="cb26-1">... <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">% pytest -v -m "not (flaky or slow or integration)"</span></span>
<span id="cb26-2">============================= test session starts =============================</span>
<span id="cb26-3">platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- ...</span>
<span id="cb26-4">cachedir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> .pytest_cache</span>
<span id="cb26-5">rootdir<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> /...</span>
<span id="cb26-6">configfile<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> pyproject.toml</span>
<span id="cb26-7">testpaths<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span> ./tests</span>
<span id="cb26-8">collected 6 items / 5 deselected / 1 selected                                 </span>
<span id="cb26-9"></span>
<span id="cb26-10">tests/test_do_something.py<span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">::test_nothing</span> PASSED                          <span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">[1</span>00<span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%]</span></span>
<span id="cb26-11"></span>
<span id="cb26-12">======================= 1 passed, 5 deselected in 0.05s =======================</span></code></pre></div></div>
<p>Note that using <code>or</code> has greedily excluded any test marked with at least one of the specified marks.</p>
</section>
</section>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>Registering marks with <code>pytest</code> is very easy and is useful for controlling which tests are executed. We have illustrated:</p>
<ul>
<li>registering marks</li>
<li>marking tests and test classes</li>
<li>the use of the <code>pytest -m</code> flag</li>
<li>selection of multiple marks</li>
<li>deselection of multiple marks</li>
</ul>
<p>Overall, this feature of <code>pytest</code> is simple and intuitive. There are more options for marking tests. I recommend reading the <a href="https://docs.pytest.org/en/7.1.x/example/markers.html"><code>pytest</code> custom markers</a> examples for more information.</p>
<p>As mentioned earlier, this is the final in the <a href="../blogs/index.html#category=pytest-in-plain-english">pytest in plain English</a> series. I will be taking a break from blogging about testing for a while. But colleagues have asked about articles on property-based testing and some of the more useful <code>pytest</code> plug-ins. I plan to cover these topics at a later date.</p>
<p>If you spot an error with this article, or have a suggested improvement then feel free to <a href="https://github.com/r-leyshon/blogging/issues">raise an issue on GitHub</a>.</p>
<p>Happy testing!</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>To past and present colleagues who have helped to discuss pros and cons, establishing practice and firming-up some opinions. Particularly:</p>
<ul>
<li>Ethan</li>
<li>Sergio</li>
</ul>
<p id="fin">
<i>fin!</i>
</p>


</section>

 ]]></description>
  <category>Explanation</category>
  <category>pytest</category>
  <category>Unit tests</category>
  <category>marks</category>
  <category>custom marks</category>
  <category>markers</category>
  <category>pytest-in-plain-english</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/16-pytest-marks.html</guid>
  <pubDate>Mon, 22 Jul 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/16-pytest-marks/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Mocking With Pytest in Plain English</title>
  <dc:creator>Rich Leyshon</dc:creator>
  <link>https://thedatasavvycorner.netlify.app/blogs/15-pytest-mocking.html</link>
  <description><![CDATA[ 





<p><img class="shaded_box" src="https://thedatasavvycorner.netlify.app/www/15-pytest-mocking/intro-img.jpg" alt="A futuristic marionette." style="display:block;margin-left:auto;margin-right:auto;width:40%;border:none;"></p>
<blockquote class="blockquote">
<p>“A day without laughter is a day wasted.” Charlie Chaplin</p>
</blockquote>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p><code>pytest</code> is a testing package for the python framework. It is broadly used to quality assure code logic. This article discusses the dark art of mocking, why you should do it and the nuts and bolts of implementing mocked tests. This blog is the fourth in a series of blogs called <a href="../blogs/index.html#category=pytest-in-plain-english">pytest in plain English</a>, favouring accessible language and simple examples to explain the more intricate features of the <code>pytest</code> package.</p>
<p>For a wealth of documentation, guides and how-tos, please consult the <a href="https://docs.pytest.org/en/8.0.x/" target="_blank"><code>pytest</code> documentation</a>.</p>
<section id="what-does-mocking-mean" class="level3">
<h3 class="anchored" data-anchor-id="what-does-mocking-mean">What does Mocking Mean?</h3>
<p>Code often has external dependencies:</p>
<ul>
<li>Web APIs (as in this article)</li>
<li>Websites (if scraping / crawling)</li>
<li>External code (importing packages)</li>
<li>Data feeds and databases</li>
<li>Environment variables</li>
</ul>
<p>As developers cannot control the behaviour of those dependencies, they would not write tests dependent upon them. In order to test their source code that depends on these services, developers need to replace the properties of these services when the test suite runs. Injecting replacement values into the code at runtime is generally referred to as mocking. Mocking these values means that developers can feed dependable results to their code and make reliable assertions about the code’s behaviour, without changes in the ‘outside world’ affecting outcomes in the system under test.</p>
<p>Developers who write unit tests may also mock their own code. The “unit” in the term “unit test” implies complete isolation from external dependencies. Mocking is an indispensible tool in achieving that isolation within a test suite. It ensures that code can be efficiently verified in any order, without dependencies on other elements in your codebase. However, mocking also adds to code complexity, increasing cognitive load and generally making things harder to debug.</p>
<div class="callout callout-style-simple callout-none no-icon callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-1-contents" aria-controls="callout-1" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon no-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">None</span>A Note on the Purpose (Click to expand)
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-1" class="callout-1-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>This article intends to discuss clearly. It doesn’t aim to be clever or impressive. Its aim is to extend understanding without overwhelming the reader. The code may not always be optimal, favouring a simplistic approach wherever possible.</p>
</div>
</div>
</div>
</section>
<section id="intended-audience" class="level3">
<h3 class="anchored" data-anchor-id="intended-audience">Intended Audience</h3>
<p>Programmers with a working knowledge of python, HTTP requests and some familiarity with <code>pytest</code> and packaging. The type of programmer who has wondered about how to follow best practice in testing python code.</p>
</section>
<section id="what-youll-need" class="level3">
<h3 class="anchored" data-anchor-id="what-youll-need">What You’ll Need:</h3>
<ul class="task-list">
<li><label><input type="checkbox">Preferred python environment manager (eg <code>conda</code>)</label></li>
<li><label><input type="checkbox"><code>pip install pytest==8.1.1 requests mockito</code></label></li>
<li><label><input type="checkbox">Git</label></li>
<li><label><input type="checkbox">GitHub account</label></li>
<li><label><input type="checkbox">Command line access</label></li>
</ul>
</section>
<section id="preparation" class="level3">
<h3 class="anchored" data-anchor-id="preparation">Preparation</h3>
<p>This blog is accompanied by code in <a href="https://github.com/r-leyshon/pytest-fiddly-examples">this repository</a>. The main branch provides a template with the minimum structure and requirements expected to run a <code>pytest</code> suite. The repo branches contain the code used in the examples of the following sections.</p>
<p>Feel free to fork or clone the repo and checkout to the example branches as needed.</p>
<p>The example code that accompanies this article is available in the <a href="https://github.com/r-leyshon/pytest-fiddly-examples/tree/mocking">mocking branch</a> of the repo.</p>
</section>
</section>
<section id="overview" class="level2">
<h2 class="anchored" data-anchor-id="overview">Overview</h2>
<p>Mocking is one of the trickier elements of testing. It’s a bit niche and is often perceived to be too hacky to be worth the effort. The options for mocking in python are numerous and this adds to the complexity of many example implementations you will find online.</p>
<p>There is also a compromise in simplicity versus flexibility. Some of the options available are quite involved and can be adapted to the nichest of cases, but may not be the best option for those new to mocking. With this in mind, I present 3 alternative methods for mocking python source code. So if you’ll forgive me, this is the first of the <a href="../blogs/index.html#category=pytest-in-plain-english"><code>pytest</code> in plain English</a> series where I introduce alternative testing practices from beyond the <code>pytest</code> package.</p>
<ol type="1">
<li><a href="https://docs.pytest.org/en/stable/how-to/monkeypatch.html"><strong>monkeypatch</strong></a>: The <code>pytest</code> fixture designed for mocking. The origin of the fixture’s name is debated but potentially arose from the term ‘guerrilla patch’ which may have been misinterpreted as ‘gorilla patch’. This is the concept of modifying source code at runtime, which probably sounds a bit like ‘monkeying with the code’.</li>
<li><a href="https://docs.python.org/3/library/unittest.mock.html"><strong>MagicMock</strong></a>: This is the mocking object provided by python3’s builtin <code>unittest</code> package.</li>
<li><a href="https://mockito-python.readthedocs.io/en/latest/walk-through.html"><strong>mockito</strong></a>: This package is based upon the popular Java framework of the same name. Despite having a user-friendly syntax, <code>mockito</code> is robust and secure.</li>
</ol>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-2-contents" aria-controls="callout-2" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>A note on the language
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-2" class="callout-2-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>Mocking has a bunch of synonyms &amp; related language which can be a bit off-putting. All of the below terms are associated with mocking. Some may be preferred to the communities of specific programming frameworks over others.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 28%">
<col style="width: 28%">
<col style="width: 42%">
</colgroup>
<thead>
<tr class="header">
<th>Term</th>
<th>Brief Meaning</th>
<th>Frameworks/Libraries</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Mocking</td>
<td>Creating objects that simulate the behaviour of real objects for testing</td>
<td>Mockito (Java), unittest.mock (Python), Jest (JavaScript), Moq (.NET)</td>
</tr>
<tr class="even">
<td>Spying</td>
<td>Observing and recording method calls on real objects</td>
<td>Mockito (Java), Sinon (JavaScript), unittest.mock (Python), RSpec (Ruby)</td>
</tr>
<tr class="odd">
<td>Stubbing</td>
<td>Replacing methods with predefined behaviours or return values</td>
<td>Sinon (JavaScript), RSpec (Ruby), PHPUnit (PHP), unittest.mock (Python)</td>
</tr>
<tr class="even">
<td>Patching</td>
<td>Temporarily modifying or replacing parts of code for testing</td>
<td>unittest.mock (Python), pytest-mock (Python), PowerMock (Java)</td>
</tr>
<tr class="odd">
<td>Faking</td>
<td>Creating simplified implementations of complex dependencies</td>
<td>Faker (multiple languages), Factory Boy (Python), FactoryGirl (Ruby)</td>
</tr>
<tr class="even">
<td>Dummy Objects</td>
<td>Placeholder objects passed around but never actually used</td>
<td>Can be created in any testing framework</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</section>
<section id="mocking-in-python" class="level2">
<h2 class="anchored" data-anchor-id="mocking-in-python">Mocking in Python</h2>
<p>This section will walk through some code that uses HTTP requests to an external service and how we can go about testing the code’s behaviour without relying on that service being available. Feel free to clone the repository and check out to the <a href="https://github.com/r-leyshon/pytest-fiddly-examples/tree/mocking">example code</a> branch to run the examples.</p>
<p>The purpose of the code is to retrieve jokes from <a href="https://icanhazdadjoke.com/" class="uri">https://icanhazdadjoke.com/</a> like so:</p>
<div id="2c2f1bc9" class="cell" data-execution_count="2">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> _ <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">range</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>):</span>
<span id="cb1-2">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>))</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>The shovel was a ground-breaking invention.
My wife is on a tropical fruit diet, the house is full of stuff. It is enough to make a mango crazy.
Where do cats write notes?
Scratch Paper!</code></pre>
</div>
</div>
<div class="callout callout-style-default callout-caution callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Caution
</div>
</div>
<div class="callout-body-container callout-body">
<p>The jokes are provided by <a href="https://icanhazdadjoke.com/" class="uri">https://icanhazdadjoke.com/</a> and are not curated by me. In my testing of the service I have found the jokes to be harmless fun, but I cannot guarantee that. If an offensive joke is returned, this is unintentional but <a href="https://github.com/r-leyshon/blogging/issues">let me know about it</a> and I will generate new jokes.</p>
</div>
</div>
<section id="define-the-source-code" class="level3">
<h3 class="anchored" data-anchor-id="define-the-source-code">Define the Source Code</h3>
<p>The function <code>get_joke()</code> uses 2 internals:</p>
<ol type="1">
<li><code>_query_endpoint()</code> Used to construct the HTTP request with required headers and user agent.</li>
<li><code>_handle_response()</code> Used to catch HTTP errors, or to pull the text out of the various response formats.</li>
</ol>
<div id="1aaf97f4" class="cell" data-execution_count="3">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Retrieve dad jokes available."""</span></span>
<span id="cb3-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb3-3"></span>
<span id="cb3-4"></span>
<span id="cb3-5"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _query_endpoint(</span>
<span id="cb3-6">    endp:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, usr_agent:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, f:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>,</span>
<span id="cb3-7">    ) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> requests.models.Response:</span>
<span id="cb3-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Utility for formatting query string &amp; requesting endpoint."""</span></span>
<span id="cb3-9">    HEADERS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="cb3-10">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"User-Agent"</span>: usr_agent,</span>
<span id="cb3-11">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Accept"</span>: f,</span>
<span id="cb3-12">        }</span>
<span id="cb3-13">    resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.get(endp, headers<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>HEADERS)</span>
<span id="cb3-14">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> resp</span></code></pre></div></div>
</div>
<p>Keeping separate, the part of the codebase that you wish to target for mocking is often the simplest way to go about things. The target for our mocking will be the command that integrates with the external service, so <code>requests.get()</code> here.</p>
<p>The use of <code>requests.get()</code> in the code above depends on a few things:</p>
<ol type="1">
<li>An endpoint string.</li>
<li>A dictionary with string values for the keys “User-Agent” and “Accept”.</li>
</ol>
<p>We’ll need to consider those dependencies when mocking. Once we return a response from the external service, we need a utility to handle the various statuses of that response:</p>
<div id="0542379a" class="cell" data-execution_count="4">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Retrieve dad jokes available."""</span></span>
<span id="cb4-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb4-3"></span>
<span id="cb4-4"></span>
<span id="cb4-5"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _query_endpoint(</span>
<span id="cb4-6">    endp:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, usr_agent:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, f:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>,</span>
<span id="cb4-7">    ) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> requests.models.Response:</span>
<span id="cb4-8">    ...</span>
<span id="cb4-9"></span>
<span id="cb4-10"></span>
<span id="cb4-11"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _handle_response(r: requests.models.Response) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="cb4-12">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Utility for handling reponse object &amp; returning text content.</span></span>
<span id="cb4-13"></span>
<span id="cb4-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Parameters</span></span>
<span id="cb4-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    ----------</span></span>
<span id="cb4-16"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    r : requests.models.Response</span></span>
<span id="cb4-17"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Response returned from webAPI endpoint.</span></span>
<span id="cb4-18"></span>
<span id="cb4-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Raises</span></span>
<span id="cb4-20"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    ------</span></span>
<span id="cb4-21"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    NotImplementedError</span></span>
<span id="cb4-22"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Requested format `f` was not either 'text/plain' or 'application/json'. </span></span>
<span id="cb4-23"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    requests.HTTPError</span></span>
<span id="cb4-24"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        HTTP error was encountered.</span></span>
<span id="cb4-25"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="cb4-26">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> r.ok:</span>
<span id="cb4-27">        c_type <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> r.headers[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>]</span>
<span id="cb4-28">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> c_type <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>:</span>
<span id="cb4-29">            content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> r.json()</span>
<span id="cb4-30">            content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> content[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"joke"</span>]</span>
<span id="cb4-31">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">elif</span> c_type <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>:</span>
<span id="cb4-32">            content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> r.text</span>
<span id="cb4-33">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb4-34">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">raise</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">NotImplementedError</span>(</span>
<span id="cb4-35">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"This client accepts 'application/json' or 'text/plain' format"</span></span>
<span id="cb4-36">                )</span>
<span id="cb4-37">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb4-38">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">raise</span> requests.HTTPError(</span>
<span id="cb4-39">            <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>r<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span>status_code<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>r<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span>reason<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb4-40">        )</span>
<span id="cb4-41">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> content</span></code></pre></div></div>
</div>
<p>Once <code>_query_endpoint()</code> gets us a response, we can feed it into <code>_handle_response()</code>, where different logic is executed depending on the response’s properties. Specifically, any response we want to mock would need the following:</p>
<ol type="1">
<li>headers, containing a dictionary eg: <code>{"content_type": "plain/text"}</code></li>
<li>A <code>json()</code> method.</li>
<li><code>text</code>, <code>status_code</code> and <code>reason</code> attributes.</li>
</ol>
<p>Finally, the above functions get wrapped in the <code>get_joke()</code> function below:</p>
<div id="8a30c979" class="cell" data-execution_count="5">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Retrieve dad jokes available."""</span></span>
<span id="cb5-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb5-3"></span>
<span id="cb5-4"></span>
<span id="cb5-5"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _query_endpoint(</span>
<span id="cb5-6">    endp:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, usr_agent:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, f:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>,</span>
<span id="cb5-7">    ) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> requests.models.Response:</span>
<span id="cb5-8">    ...</span>
<span id="cb5-9"></span>
<span id="cb5-10"></span>
<span id="cb5-11"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _handle_response(r: requests.models.Response) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="cb5-12">    ...</span>
<span id="cb5-13"></span>
<span id="cb5-14"></span>
<span id="cb5-15"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> get_joke(</span>
<span id="cb5-16">    endp:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://icanhazdadjoke.com/"</span>,</span>
<span id="cb5-17">    usr_agent:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"datasavvycorner.com (https://github.com/r-leyshon/pytest-fiddly-examples)"</span>, </span>
<span id="cb5-18">    f:<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>,</span>
<span id="cb5-19">) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>:</span>
<span id="cb5-20">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Request a joke from icanhazdadjoke.com.</span></span>
<span id="cb5-21"></span>
<span id="cb5-22"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Ask for a joke in either plain text or JSON format. Return the joke text.</span></span>
<span id="cb5-23"></span>
<span id="cb5-24"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Parameters</span></span>
<span id="cb5-25"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    ----------</span></span>
<span id="cb5-26"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    endp : str, optional</span></span>
<span id="cb5-27"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Endpoint to query, by default "https://icanhazdadjoke.com/"</span></span>
<span id="cb5-28"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    usr_agent : str, optional</span></span>
<span id="cb5-29"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        User agent value, by default</span></span>
<span id="cb5-30"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        "datasavvycorner.com (https://github.com/r-leyshon/pytest-fiddly-examples)"</span></span>
<span id="cb5-31"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    f : str, optional</span></span>
<span id="cb5-32"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Format to request eg "application.json", by default "text/plain"</span></span>
<span id="cb5-33"></span>
<span id="cb5-34"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Returns</span></span>
<span id="cb5-35"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    -------</span></span>
<span id="cb5-36"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    str</span></span>
<span id="cb5-37"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        Joke text.</span></span>
<span id="cb5-38"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="cb5-39">    r <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> _query_endpoint(endp<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>endp, usr_agent<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>usr_agent, f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>f)</span>
<span id="cb5-40">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> _handle_response(r)</span></code></pre></div></div>
</div>
</section>
<section id="lets-get-testing" class="level3">
<h3 class="anchored" data-anchor-id="lets-get-testing">Let’s Get Testing</h3>
<p>The behaviour in <code>get_joke()</code> is summarised in the flowchart below:</p>
<p><img style="display:block;margin-left:auto;margin-right:auto;width:500px;" src="https://thedatasavvycorner.netlify.app/www/15-pytest-mocking/01-control-flow.jpg" alt="get_joke() logic flow"></p>
<p>There are 4 outcomes to check, coloured red and green in the process chart above.</p>
<ol type="1">
<li><code>get_joke()</code> successfully returns joke text when the user asked for json format.</li>
<li><code>get_joke()</code> successfully returns joke text when the user asked for plain text.</li>
<li><code>get_joke()</code> raises <code>NotImplementedError</code> if any other valid format is asked for. Note that the API also accepts HTML and image formats, though parsing the joke text out of those is more involved and beyond the scope of this blog.</li>
<li><code>get_joke()</code> raises a <code>HTTPError</code> if the response from the API was not ok.</li>
</ol>
<p>Note that the event that we wish to target for mocking is highlighted in blue - we don’t want our tests to execute any real requests.</p>
<p>The strategy for testing this function without making requests to the web API is composed of 4 similar steps, regardless of the package used to implement the mocking.</p>
<p><img style="display:block;margin-left:auto;margin-right:auto;width:800px;" src="https://thedatasavvycorner.netlify.app/www/15-pytest-mocking/02-mock-steps.jpg" alt="4 step mocking strategy"></p>
<ol type="1">
<li><strong>Mock:</strong> Define the object or property that you wish to use as a replacement. This could be a static value or something a bit more involved, like a mock class that can return dynamic values depending upon the values it receives.</li>
<li><strong>Patch:</strong> Replace part of the source code with our mock value.</li>
<li><strong>Use:</strong> Use the source code to return a value.</li>
<li><strong>Assert:</strong> Check the returned value is what you expect.</li>
</ol>
<p>In the examples that follow, I will label the equivalent steps for the various mocking implementations.</p>
</section>
<section id="the-ultimate-joke" class="level3">
<h3 class="anchored" data-anchor-id="the-ultimate-joke">The “Ultimate Joke”</h3>
<p>What hard-coded text shall I use for my expected joke? I’ll <a href="../blogs/11-fiddly-bits-of-pytest.html">create a fixture</a> that will serve up this joke text to all of the test modules used below. I’m only going to define it once and then refer to it throughout several examples below. So it needs to be a pretty memorable, awesome joke.</p>
<div id="bc4bcde8" class="cell" data-execution_count="6">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb6-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb6-2"></span>
<span id="cb6-3"></span>
<span id="cb6-4"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.fixture</span>(scope<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"session"</span>)</span>
<span id="cb6-5"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> ULTI_JOKE():</span>
<span id="cb6-6">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> (<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Doc, I can't stop singing 'The Green, Green Grass of Home.' That "</span></span>
<span id="cb6-7">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sounds like Tom Jones Syndrome. Is it common? Well, It's Not Unusual."</span>)</span></code></pre></div></div>
</div>
<p>Being a Welshman, I may be a bit biased. But that’s a pretty memorable dad joke in my opinion. This joke will be available to every test within my test suite when I execute <code>pytest</code> from the command line. The assertions that we will use when using <code>get_joke()</code> will expect this string to be returned. If some other joke is returned, then we have not mocked correctly and an HTTP request was sent to the API.</p>
</section>
<section id="mocking-everything" class="level3">
<h3 class="anchored" data-anchor-id="mocking-everything">Mocking Everything</h3>
<p>I’ll start with an example of how to mock <code>get_joke()</code> completely. This is an intentionally bad idea. In doing this, the test won’t actually be executing any of the code, just returning a hard-coded value for the joke text. All this does is prove that the mocking works as expected and has nothing to do with the logic in our source code.</p>
<p>So why am I doing it? Hopefully I can illustrate the most basic implementation of mocking in this way. I’m not having to think about how I can mock a response object with all the required properties. I just need to provide some hard coded text.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-1-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-1" aria-controls="tabset-1-1" aria-selected="true" href="">monkeypatch</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-2" aria-controls="tabset-1-2" aria-selected="false" href="">MagicMock</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-3" aria-controls="tabset-1-3" aria-selected="false" href="">mockito</a></li></ul>
<div class="tab-content">
<div id="tabset-1-1" class="tab-pane active" aria-labelledby="tabset-1-1-tab">
<div id="fef776d2" class="cell" data-execution_count="7">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb7-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> example_pkg.only_joking</span>
<span id="cb7-2"></span>
<span id="cb7-3"></span>
<span id="cb7-4"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_monkeypatched_entirely(monkeypatch, ULTI_JOKE):</span>
<span id="cb7-5">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Completely replace the entire get_joke return value.</span></span>
<span id="cb7-6"></span>
<span id="cb7-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Not a good idea for testing as none of our source code will be tested. But</span></span>
<span id="cb7-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    this demonstrates how to entirely scrub a function and replace with any</span></span>
<span id="cb7-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    placeholder value at pytest runtime."""</span></span>
<span id="cb7-10">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1</span></span>
<span id="cb7-11">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_joke():</span>
<span id="cb7-12">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Return the joke text.</span></span>
<span id="cb7-13"></span>
<span id="cb7-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        monkeypatch.setattr expects the value argument to be callable. In plain</span></span>
<span id="cb7-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        English, a function or class."""</span></span>
<span id="cb7-16">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> ULTI_JOKE</span>
<span id="cb7-17">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2</span></span>
<span id="cb7-18">    monkeypatch.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">setattr</span>(</span>
<span id="cb7-19">        target<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>example_pkg.only_joking,</span>
<span id="cb7-20">        name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"get_joke"</span>,</span>
<span id="cb7-21">        value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>_mock_joke</span>
<span id="cb7-22">        )</span>
<span id="cb7-23">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3 &amp; 4</span></span>
<span id="cb7-24">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Use the module's namespace to correspond with the monkeypatch</span></span>
<span id="cb7-25">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> example_pkg.only_joking.get_joke() <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE </span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li>Step 2 requires the hard coded text to be returned from a callable, like a function or class. So we define <code>_mock_joke</code> to serve the text in the required format.</li>
</ul></li>
<li><strong>Step 2</strong>
<ul>
<li><code>monkeypatch.setattr()</code> is able to take the module namespace that we imported as the target. This must be the namespace where the function (or variable etc) is defined.</li>
</ul></li>
<li><strong>Step 3</strong>
<ul>
<li>When invoking the function, be sure to reference the function in the same way as it was monkeypatched.</li>
<li>Aliases can also be used if preferable (eg <code>import example_pkg.only_joking as jk</code>). Be sure to update your reference to <code>get_joke()</code> in step 2 and 3 to match your import statement.</li>
</ul></li>
</ul>
</div>
</div>
</div>
<div id="tabset-1-2" class="tab-pane" aria-labelledby="tabset-1-2-tab">
<div id="6e9777dd" class="cell" data-execution_count="8">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb8-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest.mock <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> MagicMock, patch</span>
<span id="cb8-2"></span>
<span id="cb8-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> example_pkg.only_joking</span>
<span id="cb8-4"></span>
<span id="cb8-5"></span>
<span id="cb8-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_magicmocked_entirely(ULTI_JOKE):</span>
<span id="cb8-7">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Completely replace the entire get_joke return value.</span></span>
<span id="cb8-8"></span>
<span id="cb8-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Not a good idea for testing as none of our source code will be tested. But</span></span>
<span id="cb8-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    this demonstrates how to entirely scrub a function and replace with any</span></span>
<span id="cb8-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    placeholder value at pytest runtime."""</span></span>
<span id="cb8-12">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1</span></span>
<span id="cb8-13">    _mock_joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> MagicMock(return_value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>ULTI_JOKE)</span>
<span id="cb8-14">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2</span></span>
<span id="cb8-15">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> patch(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"example_pkg.only_joking.get_joke"</span>, _mock_joke):</span>
<span id="cb8-16">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3</span></span>
<span id="cb8-17">        joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> example_pkg.only_joking.get_joke()</span>
<span id="cb8-18">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4</span></span>
<span id="cb8-19">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE</span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li><code>MagicMock()</code> allows us to return static values as mock objects.</li>
</ul></li>
<li><strong>Step 3</strong>
<ul>
<li>When you use <code>get_joke()</code>, be sure to call reference the namespace in the same way as to your patch in step 2.</li>
</ul></li>
</ul>
</div>
</div>
</div>
<div id="tabset-1-3" class="tab-pane" aria-labelledby="tabset-1-3-tab">
<div id="1b3eff53" class="cell" data-execution_count="9">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb9-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mockito <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> when, unstub</span>
<span id="cb9-2"></span>
<span id="cb9-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> example_pkg.only_joking</span>
<span id="cb9-4"></span>
<span id="cb9-5"></span>
<span id="cb9-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_mockitoed_entirely(ULTI_JOKE):</span>
<span id="cb9-7">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Completely replace the entire get_joke return value.</span></span>
<span id="cb9-8"></span>
<span id="cb9-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Not a good idea for testing as none of our source code will be tested. But</span></span>
<span id="cb9-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    this demonstrates how to entirely scrub a function and replace with any</span></span>
<span id="cb9-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    placeholder value at pytest runtime."""</span></span>
<span id="cb9-12">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1 &amp; 2</span></span>
<span id="cb9-13">    when(example_pkg.only_joking).get_joke().thenReturn(ULTI_JOKE)</span>
<span id="cb9-14">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3</span></span>
<span id="cb9-15">    joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> example_pkg.only_joking.get_joke()</span>
<span id="cb9-16">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4</span></span>
<span id="cb9-17">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE</span>
<span id="cb9-18">    unstub()</span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1 &amp; 2</strong>
<ul>
<li><code>mockito</code>’s intuitive <code>when(...).thenReturn(...)</code> pattern allows you to reference any object within the imported namespace. Like with <code>MagicMock</code>, the static string <code>ULTI_JOKE</code> can be referenced.</li>
</ul></li>
<li><strong>Step 3</strong>
<ul>
<li>When you use <code>get_joke()</code>, be sure to call reference the namespace in the same way as to your patch in step 2.</li>
</ul></li>
<li><strong>unstub</strong>
<ul>
<li>This step explicitly ‘unpatches’ <code>get_joke()</code>. If you did not <code>unstub()</code>, the patch to <code>get_joke()</code> would persist through the rest of your tests.</li>
<li><code>mockito</code> allows you to implicitly <code>unstub()</code> by using the context manager <code>with</code>.</li>
</ul></li>
</ul>
</div>
</div>
</div>
</div>
</div>
</section>
<section id="monkeypatch-without-oop" class="level3">
<h3 class="anchored" data-anchor-id="monkeypatch-without-oop"><code>monkeypatch()</code> without OOP</h3>
<p>Something I’ve noticed about the <code>pytest</code> documentation for <code>monkeypatch</code>, is that it gets straight into mocking with Object Oriented Programming (OOP). While this may be a bit more convenient, it is certainly not a requirement of using <code>monkeypatch</code> and definitely adds to the cognitive load for new users. This first example will mock the value of <code>requests.get</code> without using classes.</p>
<div id="42a47a8c" class="cell" data-execution_count="10">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb10-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb10-2"></span>
<span id="cb10-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb10-4"></span>
<span id="cb10-5"></span>
<span id="cb10-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_monkeypatched_no_OOP(monkeypatch, ULTI_JOKE):</span>
<span id="cb10-7">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock the response object</span></span>
<span id="cb10-8">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_response(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb10-9">        resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.models.Response()</span>
<span id="cb10-10">        resp.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span id="cb10-11">        resp._content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ULTI_JOKE.encode(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"UTF8"</span>)</span>
<span id="cb10-12">        resp.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>}</span>
<span id="cb10-13">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> resp</span>
<span id="cb10-14">    </span>
<span id="cb10-15">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch requests.get</span></span>
<span id="cb10-16">    monkeypatch.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">setattr</span>(requests, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"get"</span>, _mock_response)</span>
<span id="cb10-17">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3: Use requests.get</span></span>
<span id="cb10-18">    joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_joke()</span>
<span id="cb10-19">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4: Assert</span></span>
<span id="cb10-20">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE, <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Expected:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>ULTI_JOKE<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">Found:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>joke<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'"</span></span>
<span id="cb10-21">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># will also work for json format</span></span>
<span id="cb10-22">    joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>)</span>
<span id="cb10-23">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE, <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Expected:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>ULTI_JOKE<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">Found:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>joke<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'"</span></span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li>The return value of <code>requests.get()</code> will be a response object. We need to mock this object with the methods and attributes required by the <code>_handle_response()</code> function.</li>
<li>We need to encode the static joke text as bytes to format the data. Response objects encode data as bytes for interoperatability and optimisation purposes.</li>
</ul></li>
<li><strong>step4</strong>
<ul>
<li>As we have set an appropriate value for the mocked response’s <code>_content</code> attribute, the mocked joke will be returned for both JSON and plain text formats - very convenient!</li>
</ul></li>
</ul>
</div>
</div>
</section>
<section id="condition-1-test-json" class="level3">
<h3 class="anchored" data-anchor-id="condition-1-test-json">Condition 1: Test JSON</h3>
<p>In this example, we demonstrate the same functionality as above, but the <code>monkeypatch</code> example will use an object-oriented design pattern. This approach more closely follows that of the <code>pytest</code> documentation. As before, <code>MagicMock</code> and <code>mockito</code> examples will be included.</p>
<p>The purpose of this test is to test the outcome of <code>get_joke()</code> when the user specifies a json format.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-2-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-2-1" aria-controls="tabset-2-1" aria-selected="true" href="">monkeypatch</a></li><li class="nav-item"><a class="nav-link" id="tabset-2-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-2-2" aria-controls="tabset-2-2" aria-selected="false" href="">MagicMock</a></li><li class="nav-item"><a class="nav-link" id="tabset-2-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-2-3" aria-controls="tabset-2-3" aria-selected="false" href="">mockito</a></li></ul>
<div class="tab-content">
<div id="tabset-2-1" class="tab-pane active" aria-labelledby="tabset-2-1-tab">
<div id="monkey-fixture" class="cell" data-execution_count="11">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb11-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb11-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb11-3"></span>
<span id="cb11-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb11-5"></span>
<span id="cb11-6"></span>
<span id="cb11-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.fixture</span></span>
<span id="cb11-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_response(ULTI_JOKE):</span>
<span id="cb11-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Return a class instance that will mock all the properties of a response</span></span>
<span id="cb11-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    object that get_joke needs to work.</span></span>
<span id="cb11-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="cb11-12">    HEADERS_MAP <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="cb11-13">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>: {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>},</span>
<span id="cb11-14">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>: {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>},</span>
<span id="cb11-15">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/html"</span>: {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/html"</span>},</span>
<span id="cb11-16">    }</span>
<span id="cb11-17"></span>
<span id="cb11-18">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">class</span> MockResponse:</span>
<span id="cb11-19">        <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">__init__</span>(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>, f, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb11-20">            <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.ok <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span></span>
<span id="cb11-21">            <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.f <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> f</span>
<span id="cb11-22">            <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> HEADERS_MAP[f] <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># header corresponds to format that</span></span>
<span id="cb11-23">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># the user requested</span></span>
<span id="cb11-24">            <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.text <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ULTI_JOKE </span>
<span id="cb11-25"></span>
<span id="cb11-26">        <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> json(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>):</span>
<span id="cb11-27">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.f <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>:</span>
<span id="cb11-28">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"joke"</span>: ULTI_JOKE}</span>
<span id="cb11-29">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span></span>
<span id="cb11-30"></span>
<span id="cb11-31">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> MockResponse</span>
<span id="cb11-32"></span>
<span id="cb11-33"></span>
<span id="cb11-34"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_json_monkeypatched(monkeypatch, _mock_response, ULTI_JOKE):</span>
<span id="cb11-35">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for JSON joke.</span></span>
<span id="cb11-36"></span>
<span id="cb11-37"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Test get_joke using the mock class fixture. This approach is the</span></span>
<span id="cb11-38"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    implementation suggested in the pytest docs.</span></span>
<span id="cb11-39"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span>
<span id="cb11-40">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb11-41">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_get_good_resp(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb11-42">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Return fixtures with the correct header.</span></span>
<span id="cb11-43"></span>
<span id="cb11-44"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        If the test uses "text/plain" format, we need to return a MockResponse</span></span>
<span id="cb11-45"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        class instance with headers attribute equal to</span></span>
<span id="cb11-46"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        {"Content-Type": "text/plain"}, likewise for JSON.</span></span>
<span id="cb11-47"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        """</span></span>
<span id="cb11-48">        f <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> kwargs[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"headers"</span>][<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Accept"</span>]</span>
<span id="cb11-49">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> _mock_response(f)</span>
<span id="cb11-50">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Step 2: Patch</span></span>
<span id="cb11-51">    monkeypatch.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">setattr</span>(requests, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"get"</span>, _mock_get_good_resp)</span>
<span id="cb11-52">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Step 3: Use</span></span>
<span id="cb11-53">    j_json <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>)</span>
<span id="cb11-54">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Step 4: Assert</span></span>
<span id="cb11-55">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> j_json <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE, <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Expected:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>ULTI_JOKE<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">Found:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>j_json<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'"</span></span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li>We define a mocked class instance with the necessary properties expected by <code>_handle_response()</code>.</li>
<li>The mocked response is served to our test as a <code>pytest</code> fixture.</li>
<li>Within the test, we need another function, which will be able to take the arguments passed to <code>requests.get()</code>. This will allow our class instance to retrieve the appropriate header from the <code>HEADERS_MAP</code> dictionary.</li>
</ul></li>
</ul>
<p>As you may appreciate, this does not appear to be the most straight forward implementation, but it will allow us to test when the user asks for JSON, plain text or HTML formats. In the above test, we assert against JSON format only.</p>
</div>
</div>
</div>
<div id="tabset-2-2" class="tab-pane" aria-labelledby="tabset-2-2-tab">
<div id="54f7b8fa" class="cell" data-execution_count="12">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb12-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest.mock <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> MagicMock, patch</span>
<span id="cb12-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb12-3"></span>
<span id="cb12-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb12-5"></span>
<span id="cb12-6"></span>
<span id="cb12-7"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_json_magicmocked(ULTI_JOKE):</span>
<span id="cb12-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for JSON joke."""</span></span>
<span id="cb12-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb12-10">    mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> MagicMock(spec<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>requests.models.Response)</span>
<span id="cb12-11">    mock_response.ok <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span></span>
<span id="cb12-12">    mock_response.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>}</span>
<span id="cb12-13">    mock_response.json.return_value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"joke"</span>: ULTI_JOKE}</span>
<span id="cb12-14">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb12-15">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> patch(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"requests.get"</span>, return_value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>mock_response):</span>
<span id="cb12-16">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3: Use</span></span>
<span id="cb12-17">        joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>)</span>
<span id="cb12-18">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4: Assert</span></span>
<span id="cb12-19">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE</span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li><code>MagicMock()</code> can return a mock object with a specification designed to mock response objects. Super useful.</li>
<li>Our static joke content can be served directly to <code>MagicMock</code> without the need for an intermediate class.</li>
<li>In comparison to the <code>monkeypatch</code> approach, this appears to be more straight forward and maintainable.</li>
</ul></li>
</ul>
</div>
</div>
</div>
<div id="tabset-2-3" class="tab-pane" aria-labelledby="tabset-2-3-tab">
<div id="37b639ff" class="cell" data-execution_count="13">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb13-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mockito <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> when, unstub</span>
<span id="cb13-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb13-3"></span>
<span id="cb13-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> example_pkg.only_joking</span>
<span id="cb13-5"></span>
<span id="cb13-6"></span>
<span id="cb13-7"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_json_mockitoed(ULTI_JOKE):</span>
<span id="cb13-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for JSON joke."""</span></span>
<span id="cb13-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb13-10">    _mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.models.Response()</span>
<span id="cb13-11">    _mock_response.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span id="cb13-12">    _mock_response._content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">b'{"joke": "'</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> ULTI_JOKE.encode(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"utf-8"</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">b'"}'</span></span>
<span id="cb13-13">    _mock_response.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>}</span>
<span id="cb13-14">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb13-15">    when(requests).get(...).thenReturn(_mock_response)</span>
<span id="cb13-16">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3: Use</span></span>
<span id="cb13-17">    joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> example_pkg.only_joking.get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"application/json"</span>)</span>
<span id="cb13-18">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4: Assert</span></span>
<span id="cb13-19">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE</span>
<span id="cb13-20">    unstub()</span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li>In order to encode the expected joke for JSON format, we need a dictionary encoded within a bytestring. This bit is a little tricky.</li>
<li>Alternatively, define the expected dictionary and use the <code>json</code> package. <code>json.dumps(dict).encode("UTF8")</code> will format the content dictionary in the required way.</li>
</ul></li>
<li><strong>Step 2</strong>
<ul>
<li><code>mockito</code>’s <code>when()</code> approach will allow you to access the methods of the object that is being patched, in this case <code>requests</code>.</li>
<li><code>mockito</code> allows you to pass the <code>...</code> argument to a patched method, to indicate that whatever arguments were passed to <code>get()</code>, return the specified mock value.</li>
<li>Being able to specify values passed in place of <code>...</code> will allow you to set different return values depending on argument values received by <code>get()</code>.</li>
</ul></li>
</ul>
</div>
</div>
</div>
</div>
</div>
</section>
<section id="condition-2-test-plain-text" class="level3">
<h3 class="anchored" data-anchor-id="condition-2-test-plain-text">Condition 2: Test Plain Text</h3>
<p>The purpose of this test is to check the outcome when the user specifies a plain/text format while using <code>get_joke()</code>.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-3-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-3-1" aria-controls="tabset-3-1" aria-selected="true" href="">monkeypatch</a></li><li class="nav-item"><a class="nav-link" id="tabset-3-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-3-2" aria-controls="tabset-3-2" aria-selected="false" href="">MagicMock</a></li><li class="nav-item"><a class="nav-link" id="tabset-3-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-3-3" aria-controls="tabset-3-3" aria-selected="false" href="">mockito</a></li></ul>
<div class="tab-content">
<div id="tabset-3-1" class="tab-pane active" aria-labelledby="tabset-3-1-tab">
<div id="c20c7dd0" class="cell" data-execution_count="14">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb14-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb14-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb14-3"></span>
<span id="cb14-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb14-5"></span>
<span id="cb14-6"></span>
<span id="cb14-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.fixture</span></span>
<span id="cb14-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_response(ULTI_JOKE):</span>
<span id="cb14-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""The same fixture as was used for testing JSON format"""</span></span>
<span id="cb14-10">    ...</span>
<span id="cb14-11"></span>
<span id="cb14-12"></span>
<span id="cb14-13"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_text_monkeypatched(monkeypatch, _mock_response, ULTI_JOKE):</span>
<span id="cb14-14">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for plain text joke."""</span></span>
<span id="cb14-15">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb14-16">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_get_good_resp(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb14-17">        f <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> kwargs[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"headers"</span>][<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Accept"</span>]</span>
<span id="cb14-18">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> _mock_response(f)</span>
<span id="cb14-19">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb14-20">    monkeypatch.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">setattr</span>(requests, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"get"</span>, _mock_get_good_resp)</span>
<span id="cb14-21">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3: Use</span></span>
<span id="cb14-22">    j_txt <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>)</span>
<span id="cb14-23">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4: Assert</span></span>
<span id="cb14-24">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> j_txt <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE, <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Expected:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>ULTI_JOKE<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">Found:</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>j_txt<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">'"</span></span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li>We can use the same mock class as for testing Condition 1, due to the content of the <code>HEADERS_MAP</code> dictionary.</li>
</ul></li>
</ul>
</div>
</div>
</div>
<div id="tabset-3-2" class="tab-pane" aria-labelledby="tabset-3-2-tab">
<div id="290503f5" class="cell" data-execution_count="15">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb15-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest.mock <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> MagicMock, patch</span>
<span id="cb15-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb15-3"></span>
<span id="cb15-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb15-5"></span>
<span id="cb15-6"></span>
<span id="cb15-7"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_text_magicmocked(ULTI_JOKE):</span>
<span id="cb15-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for plain text joke."""</span></span>
<span id="cb15-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb15-10">    mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> MagicMock(spec<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>requests.models.Response)</span>
<span id="cb15-11">    mock_response.ok <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span></span>
<span id="cb15-12">    mock_response.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>}</span>
<span id="cb15-13">    mock_response.text <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ULTI_JOKE</span>
<span id="cb15-14">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb15-15">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> patch(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"requests.get"</span>, return_value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>mock_response):</span>
<span id="cb15-16">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3: Use</span></span>
<span id="cb15-17">        joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>)</span>
<span id="cb15-18">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4: Assert</span></span>
<span id="cb15-19">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE</span></code></pre></div></div>
</div>
</div>
<div id="tabset-3-3" class="tab-pane" aria-labelledby="tabset-3-3-tab">
<div id="00b8b1a9" class="cell" data-execution_count="16">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb16" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb16-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mockito <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> when, unstub</span>
<span id="cb16-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb16-3"></span>
<span id="cb16-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> example_pkg.only_joking</span>
<span id="cb16-5"></span>
<span id="cb16-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_text_mockitoed(ULTI_JOKE):</span>
<span id="cb16-7">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for plain text joke."""</span></span>
<span id="cb16-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb16-9">    mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.models.Response()</span>
<span id="cb16-10">    mock_response.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span id="cb16-11">    mock_response._content <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ULTI_JOKE.encode(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"utf-8"</span>)</span>
<span id="cb16-12">    mock_response.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>}</span>
<span id="cb16-13">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb16-14">    when(requests).get(...).thenReturn(mock_response)</span>
<span id="cb16-15">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3: Use</span></span>
<span id="cb16-16">    joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> example_pkg.only_joking.get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/plain"</span>)</span>
<span id="cb16-17">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 4: Assert</span></span>
<span id="cb16-18">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">assert</span> joke <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> ULTI_JOKE</span>
<span id="cb16-19">    unstub()</span></code></pre></div></div>
</div>
</div>
</div>
</div>
</section>
<section id="condition-3-test-not-implemented" class="level3">
<h3 class="anchored" data-anchor-id="condition-3-test-not-implemented">Condition 3: Test Not Implemented</h3>
<p>This test will check the outcome of what happens when the user asks for a format other than text or JSON format. As the webAPI also offers image or HTML formats, a response 200 (ok) would be returned from the service. But I was too busy (lazy) to extract the text from those formats.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-4-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-4-1" aria-controls="tabset-4-1" aria-selected="true" href="">monkeypatch</a></li><li class="nav-item"><a class="nav-link" id="tabset-4-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-4-2" aria-controls="tabset-4-2" aria-selected="false" href="">MagicMock</a></li><li class="nav-item"><a class="nav-link" id="tabset-4-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-4-3" aria-controls="tabset-4-3" aria-selected="false" href="">mockito</a></li></ul>
<div class="tab-content">
<div id="tabset-4-1" class="tab-pane active" aria-labelledby="tabset-4-1-tab">
<div id="5a12263d" class="cell" data-execution_count="17">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb17" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb17-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb17-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb17-3"></span>
<span id="cb17-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb17-5"></span>
<span id="cb17-6"></span>
<span id="cb17-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.fixture</span></span>
<span id="cb17-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_response(ULTI_JOKE):</span>
<span id="cb17-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""The same fixture as was used for testing JSON format"""</span></span>
<span id="cb17-10">    ...</span>
<span id="cb17-11"></span>
<span id="cb17-12"></span>
<span id="cb17-13"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_not_implemented_monkeypatched(</span>
<span id="cb17-14">    monkeypatch, _mock_response):</span>
<span id="cb17-15">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for HTML response."""</span></span>
<span id="cb17-16">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  step 1: Mock</span></span>
<span id="cb17-17">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_get_good_resp(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb17-18">        f <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> kwargs[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"headers"</span>][<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Accept"</span>]</span>
<span id="cb17-19">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> _mock_response(f)</span>
<span id="cb17-20">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb17-21">    monkeypatch.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">setattr</span>(requests, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"get"</span>, _mock_get_good_resp)</span>
<span id="cb17-22">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3 &amp; 4 Use (try to but exception is raised) &amp; Assert</span></span>
<span id="cb17-23">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> pytest.raises(</span>
<span id="cb17-24">        <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">NotImplementedError</span>,</span>
<span id="cb17-25">        match<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"This client accepts 'application/json' or 'text/plain' format"</span>):</span>
<span id="cb17-26">        get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/html"</span>)</span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li>We can use the same mock class as for testing Condition 1, due to the content of the <code>HEADERS_MAP</code> dictionary.</li>
</ul></li>
<li><strong>Step 4</strong>
<ul>
<li>We use a context manager (<code>with pytest.raises</code>) which catches the raised exception and stops it from terminating our <code>pytest</code> session.</li>
<li>The asserted <code>match</code> argument can take a regular expression, so that wildcard patterns can be used. This allows matching of part of the exception message.</li>
</ul></li>
</ul>
</div>
</div>
</div>
<div id="tabset-4-2" class="tab-pane" aria-labelledby="tabset-4-2-tab">
<div id="263a074b" class="cell" data-execution_count="18">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb18" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb18-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb18-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb18-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest.mock <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> MagicMock, patch</span>
<span id="cb18-4"></span>
<span id="cb18-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb18-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test__handle_response_not_implemented_magicmocked():</span>
<span id="cb18-7">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for HTML response."""</span></span>
<span id="cb18-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb18-9">    mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> MagicMock(spec<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>requests.models.Response)</span>
<span id="cb18-10">    mock_response.ok <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span></span>
<span id="cb18-11">    mock_response.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/html"</span>}</span>
<span id="cb18-12">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  step 2: Patch</span></span>
<span id="cb18-13">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> patch(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"requests.get"</span>, return_value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>mock_response):</span>
<span id="cb18-14">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3 &amp; 4 Use (try to but exception is raised) &amp; Assert</span></span>
<span id="cb18-15">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> pytest.raises(</span>
<span id="cb18-16">            <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">NotImplementedError</span>,</span>
<span id="cb18-17">            match<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"client accepts 'application/json' or 'text/plain' format"</span>):</span>
<span id="cb18-18">            get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/html"</span>)</span></code></pre></div></div>
</div>
</div>
<div id="tabset-4-3" class="tab-pane" aria-labelledby="tabset-4-3-tab">
<div id="316a4922" class="cell" data-execution_count="19">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb19" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb19-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mockito <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> when, unstub</span>
<span id="cb19-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb19-3"></span>
<span id="cb19-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> example_pkg.only_joking</span>
<span id="cb19-5"></span>
<span id="cb19-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_not_implemented_mockitoed():</span>
<span id="cb19-7">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test behaviour when user asked for HTML response."""</span></span>
<span id="cb19-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb19-9">    mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.models.Response()</span>
<span id="cb19-10">    mock_response.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span id="cb19-11">    mock_response.headers <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Content-Type"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/html"</span>}</span>
<span id="cb19-12">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb19-13">    when(</span>
<span id="cb19-14">        example_pkg.only_joking</span>
<span id="cb19-15">        )._query_endpoint(...).thenReturn(mock_response)</span>
<span id="cb19-16">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3 &amp; 4 Use (try to but exception is raised) &amp; Assert</span></span>
<span id="cb19-17">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> pytest.raises(</span>
<span id="cb19-18">        <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">NotImplementedError</span>,</span>
<span id="cb19-19">        match<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"This client accepts 'application/json' or 'text/plain' format"</span>):</span>
<span id="cb19-20">        example_pkg.only_joking.get_joke(f<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text/html"</span>)</span>
<span id="cb19-21">    unstub()</span></code></pre></div></div>
</div>
</div>
</div>
</div>
</section>
<section id="condition-4-test-bad-response" class="level3">
<h3 class="anchored" data-anchor-id="condition-4-test-bad-response">Condition 4: Test Bad Response</h3>
<p>In this test, we simulate a bad response from the webAPI, which could arise for a number of reasons:</p>
<ul>
<li>The api is unavailable.</li>
<li>The request asked for a resource that is not available.</li>
<li>Too many requests were made in a short period.</li>
</ul>
<p>These conditions are those that we have the least control over and therefore have the greatest need for mocking.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-5-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-5-1" aria-controls="tabset-5-1" aria-selected="true" href="">monkeypatch</a></li><li class="nav-item"><a class="nav-link" id="tabset-5-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-5-2" aria-controls="tabset-5-2" aria-selected="false" href="">MagicMock</a></li><li class="nav-item"><a class="nav-link" id="tabset-5-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-5-3" aria-controls="tabset-5-3" aria-selected="false" href="">mockito</a></li></ul>
<div class="tab-content">
<div id="tabset-5-1" class="tab-pane active" aria-labelledby="tabset-5-1-tab">
<div id="346a48c3" class="cell" data-execution_count="20">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb20" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb20-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb20-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb20-3"></span>
<span id="cb20-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke, _handle_response</span>
<span id="cb20-5"></span>
<span id="cb20-6"></span>
<span id="cb20-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@pytest.fixture</span></span>
<span id="cb20-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_bad_response():</span>
<span id="cb20-9">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">class</span> MockBadResponse:</span>
<span id="cb20-10">        <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">__init__</span>(<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb20-11">            <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.ok <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span></span>
<span id="cb20-12">            <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">404</span></span>
<span id="cb20-13">            <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">self</span>.reason <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Not Found"</span></span>
<span id="cb20-14">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> MockBadResponse</span>
<span id="cb20-15"></span>
<span id="cb20-16"></span>
<span id="cb20-17"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_http_error_monkeypatched(</span>
<span id="cb20-18">    monkeypatch, _mock_bad_response):</span>
<span id="cb20-19">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test bad HTTP response."""</span></span>
<span id="cb20-20">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  step 1: Mock</span></span>
<span id="cb20-21">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _mock_get_bad_response(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>args, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>kwargs):</span>
<span id="cb20-22">        f <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> kwargs[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"headers"</span>][<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Accept"</span>]</span>
<span id="cb20-23">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> _mock_bad_response(f)</span>
<span id="cb20-24">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  step 2: Patch</span></span>
<span id="cb20-25">    monkeypatch.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">setattr</span>(requests, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"get"</span>, _mock_get_bad_response)</span>
<span id="cb20-26">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3 &amp; 4 Use (try to but exception is raised) &amp; Assert</span></span>
<span id="cb20-27">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> pytest.raises(requests.HTTPError, match<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"404: Not Found"</span>):</span>
<span id="cb20-28">        get_joke()</span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Notes
</div>
</div>
<div class="callout-body-container callout-body">
<ul>
<li><strong>Step 1</strong>
<ul>
<li>This time we need to define a new fixture that returns a bad response.</li>
<li>Alternatively, we could have implemented a single fixture for all of our tests that dynamically served a good or bad response dependent upon arguments passed to <code>get_joke()</code>, for example different string values passed as the endpoint.</li>
<li>In a more thorough implementation of <code>get_joke()</code>, you may wish to retry the request for certain HTTP error status codes. The ability to provide mocked objects that reliably serve those statuses allow you to deterministically validate your code’s behaviour.</li>
</ul></li>
</ul>
</div>
</div>
</div>
<div id="tabset-5-2" class="tab-pane" aria-labelledby="tabset-5-2-tab">
<div id="2bfb9deb" class="cell" data-execution_count="21">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb21" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb21-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pytest</span>
<span id="cb21-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> unittest.mock <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> MagicMock, patch</span>
<span id="cb21-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb21-4"></span>
<span id="cb21-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> example_pkg.only_joking <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> get_joke</span>
<span id="cb21-6"></span>
<span id="cb21-7"></span>
<span id="cb21-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_http_error_magicmocked():</span>
<span id="cb21-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test bad HTTP response."""</span></span>
<span id="cb21-10">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb21-11">    _mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> MagicMock(spec<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>requests.models.Response)</span>
<span id="cb21-12">    _mock_response.ok <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">False</span></span>
<span id="cb21-13">    _mock_response.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">404</span></span>
<span id="cb21-14">    _mock_response.reason <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Not Found"</span></span>
<span id="cb21-15">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb21-16">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> patch(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"requests.get"</span>, return_value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>_mock_response):</span>
<span id="cb21-17">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3 &amp; 4 Use (try to but exception is raised) &amp; Assert</span></span>
<span id="cb21-18">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> pytest.raises(requests.HTTPError, match<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"404: Not Found"</span>):</span>
<span id="cb21-19">            get_joke()</span></code></pre></div></div>
</div>
</div>
<div id="tabset-5-3" class="tab-pane" aria-labelledby="tabset-5-3-tab">
<div id="1a4f2dce" class="cell" data-execution_count="22">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb22" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb22-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mockito <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> when, unstub</span>
<span id="cb22-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb22-3"></span>
<span id="cb22-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> example_pkg.only_joking</span>
<span id="cb22-5"></span>
<span id="cb22-6"></span>
<span id="cb22-7"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> test_get_joke_http_error_mockitoed():</span>
<span id="cb22-8">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Test bad HTTP response."""</span></span>
<span id="cb22-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1: Mock</span></span>
<span id="cb22-10">    _mock_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.models.Response()</span>
<span id="cb22-11">    _mock_response.status_code <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">404</span></span>
<span id="cb22-12">    _mock_response.reason <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Not Found"</span></span>
<span id="cb22-13">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2: Patch</span></span>
<span id="cb22-14">    when(example_pkg.only_joking)._query_endpoint(...).thenReturn(</span>
<span id="cb22-15">        _mock_response)</span>
<span id="cb22-16">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 3 &amp; 4 Use (try to but exception is raised) &amp; Assert</span></span>
<span id="cb22-17">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> pytest.raises(requests.HTTPError, match<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"404: Not Found"</span>):</span>
<span id="cb22-18">        example_pkg.only_joking.get_joke()</span>
<span id="cb22-19">    unstub()</span></code></pre></div></div>
</div>
</div>
</div>
</div>
</section>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>We have thoroughly tested our code using approaches that mock the behaviour of an external webAPI. We have also seen how to implement those tests with 3 different packages.</p>
<p>I hope that this has provided you with enough introductory material to begin mocking tests if you have not done so before. If you find that your specific use case for mocking is quite nuanced and fiddly (it’s likely to be that way), then the alternative implementations presented here can help you to understand how to solve your specific mocking dilemma.</p>
<p>One final quote for those developers having their patience tested by errors attempting to implement mocking:</p>
<blockquote class="blockquote">
<p>“He who laughs last, laughs loudest.”</p>
</blockquote>
<p>…or she for that matter: Don’t give up!</p>
<p>If you spot an error with this article, or have a suggested improvement then feel free to <a href="https://github.com/r-leyshon/blogging/issues">raise an issue on GitHub</a>.</p>
<p>Happy testing!</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>To past and present colleagues who have helped to discuss pros and cons, establishing practice and firming-up some opinions. Special thanks to Edward for bringing <code>mockito</code> to my attention.</p>
<p>The diagrams used in this article were produced with the excellent <a href="https://excalidraw.com/">Excalidraw</a>.</p>
<p id="fin">
<i>fin!</i>
</p>


</section>

 ]]></description>
  <category>Explanation</category>
  <category>pytest</category>
  <category>Unit tests</category>
  <category>mocking</category>
  <category>pytest-in-plain-english</category>
  <category>mockito</category>
  <category>MagicMock</category>
  <category>monkeypatch</category>
  <guid>https://thedatasavvycorner.netlify.app/blogs/15-pytest-mocking.html</guid>
  <pubDate>Sun, 14 Jul 2024 00:00:00 GMT</pubDate>
  <media:content url="https://thedatasavvycorner.netlify.app/./www/15-pytest-mocking/intro-img.jpg" medium="image" type="image/jpeg"/>
</item>
</channel>
</rss>
