Jekyll2024-03-25T11:16:03+00:00https://hansschnedlitz.com/feed.xmlHans SchnedlitzHans Schnedlitz is a Ruby on Rails Engineer from Vienna.Hans SchnedlitzPorting a Ruby Gem to the browser with ruby.wasm2024-03-25T08:54:00+00:002024-03-25T08:54:00+00:00https://hansschnedlitz.com/2024/03/25/porting-a-ruby-gem-to-the-browser-with-ruby-wasm<p>When I built a small Ruby command-line tool - <a href="https://github.com/hschne/tints-n-shades/">Tints ‘N Shades</a> - I wondered: What does it take to run this library in the browser? Can it be done? Should it be done? It is pretty cool to make Ruby libraries available to <a href="https://ruby-next.github.io/">play around with in the browser</a>, after all.</p>
<p>The only way to find out is to try. I did, and now I can happily answer those questions. But first, a bit of context.</p>
<h2 id="ruby--web-assembly">Ruby & Web Assembly</h2>
<p>The way to run Ruby code in the browser is through Web Assembly (WASM), which is <a href="https://caniuse.com/wasm">well-supported</a>. The theory is straightforward: Take some Ruby code and compile it to a WASM module. Ship that module to the browser - to any browser - and it just works. Easy, right?</p>
<p>Of course, the reality is nowhere as simple as that. A lot of amazing work has been done in recent years, with <a href="https://www.ruby-lang.org/en/news/2022/12/25/ruby-3-2-0-released/"> Ruby 3.2 </a> adding Web Assembly/WASI support and libraries such as <a href="https://github.com/ruby/ruby.wasm">ruby.wasm</a> simplifying the process of compiling Ruby to WASM.</p>
<p>If you want to give it a quick try, ruby.wasm provides various Ruby runtimes as WASM modules, so all it takes to run Ruby code in the browser is this:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"><!-- index.html --></span>
<span class="nt"><html></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"https://cdn.jsdelivr.net/npm/@ruby/3.3-wasm-wasi@2.5.0/dist/browser.script.iife.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/ruby"</span><span class="nt">></span>
<span class="nx">require</span> <span class="dl">"</span><span class="s2">js</span><span class="dl">"</span>
<span class="nx">JS</span><span class="p">.</span><span class="nb">global</span><span class="p">[:</span><span class="nb">document</span><span class="p">].</span><span class="nx">write</span> <span class="dl">"</span><span class="s2">Hello, world!</span><span class="dl">"</span>
<span class="nt"></script></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>That works for standard functions - but we want something else. We want to use our library code in the browser, so the basic Ruby runtime is insufficient. We have to incorporate our library into a WASM module. Luckily, ruby.wasm is also a toolchain that helps with that.</p>
<p>So what do we want to run in the browser? Let’s use Tints N’ Shades as an example. It generates tints and shades for a given color in various formats, similarily to <a href="https://www.tints.dev/brand/2522FC">Tints.dev</a> or other web based tools. It has basically no dependencies - apart from <a href="https://github.com/rails/thor">Thor</a>. The library code exposes a simple interface so it’s easy to call from JavaScript.</p>
<p>Let’s try it.</p>
<h2 id="compiling-to-web-assembly">Compiling to Web Assembly</h2>
<p>The simplest way to compile an existing Ruby library to WASM is by utilizing ruby.wasm’s built in <a href="https://github.com/ruby/ruby.wasm/pull/358"> bundler support </a>. To create and run our custom WASM module in the browser, we first need to add the <code class="language-plaintext highlighter-rouge">ruby_wasm</code> and <code class="language-plaintext highlighter-rouge">js</code> gems.</p>
<p>Because we are building on top of an existing gem, there already is a Gemfile. I ended up creating a separate one (<code class="language-plaintext highlighter-rouge">Gemfile-web</code>) and prefixing all bundler-related commands with <code class="language-plaintext highlighter-rouge">BUNDLE_GEMFILE=Gemfile-web</code>. It works, okay?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BUNDLE_GEMFILE=Gemfile-web bundle add ruby_wasm js
</code></pre></div></div>
<p>While we’re at it, let’s modify the Gemfile to use our own gem as a dependency. It should now look something like this.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gem "js", "~> 2.5"
gem "ruby_wasm", "~> 2.5"
gem "tints-n-shades", path: "."
</code></pre></div></div>
<p>That’s all there is to it, really. Let’s compile a WASM module using the following command. This will take a while.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BUNDLE_GEMFILE=Gemfile-web bundle install
BUNDLE_GEMFILE=Gemfile-web bundle exec rbwasm build -o ruby-web.wasm
</code></pre></div></div>
<h2 id="running-in-the-browser">Running in the Browser</h2>
<p>There are several different ways to use your module in the browser. The simplest one - as it requires no additional build steps - is to follow the instructions over <a href="https://developer.mozilla.org/en-US/docs/WebAssembly/Loading_and_running">at MDN</a>. Let’s create a new JavaScript file and run some exemplary Ruby code to verify that everything works.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">DefaultRubyVM</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">https://cdn.jsdelivr.net/npm/@ruby/3.3-wasm-wasi@2.5.0/dist/browser/+esm</span><span class="dl">"</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">response</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">fetch</span><span class="p">(</span><span class="dl">"</span><span class="s2">./ruby-web.wasm</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">module</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">compileStreaming</span><span class="p">(</span><span class="nx">response</span><span class="p">);</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">vm</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">DefaultRubyVM</span><span class="p">(</span><span class="nx">module</span><span class="p">);</span>
<span class="nx">vm</span><span class="p">.</span><span class="nb">eval</span><span class="p">(</span><span class="s2">`
require "/bundle/setup"
require "js"
require "tns"
JS.global[:document].write "Version: #{TNS::VERSION}"
`</span><span class="p">);</span>
</code></pre></div></div>
<p>Note that the <code class="language-plaintext highlighter-rouge">js</code> gem is only required to interact with JavaScript from within your Ruby code. You will likely want to return primitives from your Ruby code to your JS runtime, and you can do that by simply returning values from <code class="language-plaintext highlighter-rouge">vm.eval</code>.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">vm</span><span class="p">.</span><span class="nb">eval</span><span class="p">(</span><span class="s2">`
require "/bundle/setup"
require "tns"
TNS::VERSION
`</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">Version:</span><span class="dl">"</span><span class="p">,</span> <span class="nx">result</span><span class="p">.</span><span class="nx">toString</span><span class="p">());</span>
</code></pre></div></div>
<p>Create a minimal <code class="language-plaintext highlighter-rouge">index.html</code> to load the JavaScript, open it in your browser, and you’ll see the
ibrary version printed to the console 👌</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><!doctype html></span>
<span class="nt"><html</span> <span class="na">lang=</span><span class="s">"en"</span><span class="nt">></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"UTF-8"</span> <span class="nt">/></span>
<span class="nt"><meta</span> <span class="na">name=</span><span class="s">"viewport"</span> <span class="na">content=</span><span class="s">"width=device-width, initial-scale=1"</span> <span class="nt">/></span>
<span class="nt"><title></span>Ruby WASM<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></body></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"module"</span> <span class="na">src=</span><span class="s">"index.js"</span><span class="nt">></script></span>
<span class="nt"></html></span>
</code></pre></div></div>
<h1 id="outlook">Outlook</h1>
<p>Compared to a couple of years ago, compiling Ruby code to WASM has become incredibly simple. There is no overly complex setup required. However, some practical issues remain. For one, the size of the WASM module is significant — 52MB in our case.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls</span> <span class="nt">-lah</span>
5.7K Mar 25 09:50 index.html
1.6K Mar 25 09:49 index.js
52M Mar 18 13:36 ruby-web.wasm
</code></pre></div></div>
<p>When compressed, that file size goes down to something like 15MB, but still - that’s not great.</p>
<p>Because Ruby is an interpreted language, running Ruby as a WASM module requires that that module incorporate an entire Ruby VM. And that thing ain’t small. We can do some things to reduce the module size, for example by using a minimal build profile with <code class="language-plaintext highlighter-rouge">rbwasm build</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BUNDLE_GEMFILE=Gemfile-web bundle exec rbwasm build --build-profile minimal -o ruby-web.wasm
</code></pre></div></div>
<p>There are some downsides, though. Using the <code class="language-plaintext highlighter-rouge">minimal</code> profile means that several standard modules (e.g. <code class="language-plaintext highlighter-rouge">js</code>, <code class="language-plaintext highlighter-rouge">yaml</code>, <code class="language-plaintext highlighter-rouge">stdio</code>) are excluded. Depending on your specific use case, that may lead to breakage. I found that this shaves around 10MB from the final module, which also isn’t that significant in the grand scheme of things.</p>
<p>So, is it practical to use Ruby WASM modules rather than plain old JavaScript? Given the file size, probably not. Is it interesting and fun, though? Absolutely!</p>
<p>I am excited and curious to learn how the Ruby WASM story continues. You can see Tints ‘N Shades running in the browser <a href="https://www.hansschnedlitz.com/tints-n-shades/">here</a>.</p>Hans SchnedlitzWhen I built a small Ruby command-line tool - Tints ‘N Shades - I wondered: What does it take to run this library in the browser? Can it be done? Should it be done? It is pretty cool to make Ruby libraries available to play around with in the browser, after all.Using Jekyll with Esbuild2024-02-09T09:00:00+00:002024-02-09T09:00:00+00:00https://hansschnedlitz.com/2024/02/09/using-jekyll-with-esbuild<p>I know. What an unholy union. Why would anyone do this? Why would <em>anyone</em> want that?</p>
<p>Well, first for science. Obviously.</p>
<p>Second, believe it or not, there are <em>actual</em> good reasons for combining Esbuild with Jekyll. I like Jekyll. It’s mature, has many plugins, and a vibrant ecosystem. Also, it’s built on Ruby, and Ruby is fantastic. Most importantly, it’s simple.</p>
<p>But.</p>
<p>Sometimes, you want that extra bit of visual oomph. Sometimes, you want to do weird or cool things, and a bundler might be necessary under those circumstances. Esbuild is modern and efficient. If you’re working with Ruby on Rails, you might already be used to it. Coincidentally, it’s also simple. Relatively speaking.</p>
<p>If you look at it like that, maybe Jekyll and Esbuild are meant to work together after all?</p>
<p class="notice--info">Why not Hugo? Or Bridgetown? Or any other static site generator that’s not Jekyll? Look, I don’t know what else to say. For science. I like Jekyll. It works for me 🙃</p>
<h2 id="what-well-do">What we’ll do</h2>
<p>The idea is this. We want to use Esbuild for bundling JS and CSS and let Jekyll take care of the rest. We’ll also add some plugins to Esbuild (<a href="https://github.com/postcss/autoprefixer">autoprefixer</a>, <a href="https://www.npmjs.com/package/esbuild-sass-plugin">build-sass-plugin</a>, <a href="https://postcss.org/">postcss</a>). First, so we can keep using the SCSS that Jekyll already provides, second for demonstration purposes.</p>
<p>We’ll also make sure we end up with a pleasant developer experience. Jekyll and Esbuild might not want to play nice, but we’ll make them 😈</p>
<h2 id="getting-set-up">Getting set up</h2>
<p class="notice--info">Heads up! This guide assumes you have Ruby, Bundler, and Node set up on your machine. Also, Jekyll should already be installed. If that’s not the case, take care of that before you continue.</p>
<p>We’ll forego any plugins and start with an empty Jekyll scaffold to keep things simple.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jekyll new jekyll-esbuild --blank && cd jekyll-esbuild
</code></pre></div></div>
<p>Let’s create an empty Javascript file for demonstration purposes. Let’s also install Esbuild and the plugins we are going to use.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir assets/javascript && touch assets/javascript/main.js
npm i -D --save-exact esbuild
npm i -D autoprefixer esbuild-sass-plugin postcss
</code></pre></div></div>
<p>When we run <code class="language-plaintext highlighter-rouge">jekyll serve --watch</code> we should see <em>something</em> on <a href="http://localhost:4000">localhost:4000</a>.</p>
<p><img src="https://hansschnedlitz.com/assets/images/posts/2024-jekyll-esbuild/jekyll.webp" alt="A Jekyll site" class="align-center" /></p>
<h2 id="bundling-assets-with-esbuild">Bundling Assets with Esbuild</h2>
<p>To use Esbuild with our CSS plugins, using the command line won’t do. We’ll have to create a small build script.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// scripts/build.mjs</span>
<span class="k">import</span> <span class="nx">esbuild</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">esbuild</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">sassPlugin</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">esbuild-sass-plugin</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">postcss</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">postcss</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">autoprefixer</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">autoprefixer</span><span class="dl">"</span><span class="p">;</span>
<span class="k">await</span> <span class="nx">esbuild</span><span class="p">.</span><span class="nx">build</span><span class="p">({</span>
<span class="na">entryPoints</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">assets/css/main.scss</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">assets/javascript/main.js</span><span class="dl">"</span><span class="p">],</span>
<span class="na">outdir</span><span class="p">:</span> <span class="dl">"</span><span class="s2">_site/assets</span><span class="dl">"</span><span class="p">,</span>
<span class="na">bundle</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">plugins</span><span class="p">:</span> <span class="p">[</span>
<span class="nx">sassPlugin</span><span class="p">({</span>
<span class="k">async</span> <span class="nx">transform</span><span class="p">(</span><span class="nx">source</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">css</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">postcss</span><span class="p">([</span><span class="nx">autoprefixer</span><span class="p">]).</span><span class="nx">process</span><span class="p">(</span><span class="nx">source</span><span class="p">,</span> <span class="p">{</span>
<span class="na">from</span><span class="p">:</span> <span class="kc">undefined</span><span class="p">,</span>
<span class="p">});</span>
<span class="k">return</span> <span class="nx">css</span><span class="p">;</span>
<span class="p">},</span>
<span class="na">loadPaths</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">_sass</span><span class="dl">"</span><span class="p">],</span>
<span class="p">}),</span>
<span class="p">],</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Once we update our <code class="language-plaintext highlighter-rouge">package.json,</code> we can build our assets.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"devDependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"build"</span><span class="p">:</span><span class="w"> </span><span class="s2">"node scripts/build.mjs"</span><span class="p">,</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>If we were to run <code class="language-plaintext highlighter-rouge">npm run build</code> now, we’d get an error. Our <code class="language-plaintext highlighter-rouge">assets/main.scss</code> still contains YML markup that we need to remove.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--- <<< DELETE ME
--- <<<
@import "base";
</code></pre></div></div>
<p>After that, we should be able to build assets without any issues.</p>
<h3 id="configuring-jekyll">Configuring Jekyll</h3>
<p>Now, if you keep a close eye on the <code class="language-plaintext highlighter-rouge">_site</code> folder and make some changes to any watched files, you’ll notice that your built files will be changed. Makes sense because, as things stand, Jekyll still feels responsible for assets, thus overwriting or deleting the files created by Esbuild.</p>
<p>Let’s fix that. The simplest way to tell Jekyll to not worry about CSS and JS assets is to exclude the respective folders by updating <code class="language-plaintext highlighter-rouge">_config.yml</code>.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">exclude</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">_sass</span> <span class="c1"># Let build handle CSS</span>
<span class="pi">-</span> <span class="s">assets/css</span> <span class="c1"># Let build handle CSS</span>
<span class="pi">-</span> <span class="s">assets/javascript</span> <span class="c1"># Let esuild handle JS</span>
<span class="pi">-</span> <span class="s">scripts</span>
<span class="pi">-</span> <span class="s">package.json</span>
<span class="pi">-</span> <span class="s">package-lock.json</span>
<span class="na">keep_files</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">assets/css</span> <span class="c1"># Let build handle CSS</span>
<span class="pi">-</span> <span class="s">assets/javascript</span> <span class="c1"># Let esuild handle JS</span>
</code></pre></div></div>
<p>Notice that we also told Jekyll to not delete CSS and JS assets when rebuilding. We also excluded some additional files that our Jekyll site doesn’t need. See <a href="https://jekyllrb.com/docs/configuration/options/">configuration options</a> if you need more details.</p>
<p class="notice--warning">This won’t do if you’re using a theme. By excluding asset folders, theme styles won’t be processed appropriately. There are ways to fix that, but it’s a bit much for this blog post.</p>
<p>Let’s bundle and serve again to make sure everything is still working.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build
jekyll serve --watch
</code></pre></div></div>
<p>Changing anything (e.g. CSS) will no longer overwrite files created by Esbuild. Success! Unfortunately, our changes won’t be reflected on your site. Let’s change that by adding watch mode to Esbuild.</p>
<h3 id="improving-developer-experience">Improving Developer Experience</h3>
<p>Until the beginning of 2023, adding watch mode was a simple matter of adding <code class="language-plaintext highlighter-rouge">watch: true</code> to the arguments of Esbuild. Now, it’s a bit different. I opted to change the script behavior based on input arguments, but you can also create a separate script.</p>
<p>The updated <code class="language-plaintext highlighter-rouge">scripts/build.mjs</code> looks something like this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">args</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">argv</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">watch</span> <span class="o">=</span> <span class="nx">args</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="dl">"</span><span class="s2">--watch</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">context</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">esbuild</span><span class="p">.</span><span class="nx">context</span><span class="p">({</span>
<span class="c1">// ...</span>
<span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">watch</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">watch</span><span class="p">();</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">Watching!</span><span class="dl">"</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">rebuild</span><span class="p">();</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">dispose</span><span class="p">();</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">Build done!</span><span class="dl">"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After updating <code class="language-plaintext highlighter-rouge">package.json</code> once again, we can start watching our file changes by running <code class="language-plaintext highlighter-rouge">npm run watch</code>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"devDependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"watch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"node scripts/build.mjs --watch"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>We’re already using Ruby anyway, so we might as well use <a href="https://github.com/ddollar/foreman">Foreman</a> to run everything we need with a simple command. Note that I also added <a href="https://browsersync.io/">browser-sync</a> for live reloading to the <code class="language-plaintext highlighter-rouge">Procfile</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jekyll: jekyll serve --watch
build: npm run watch
browser: browser-sync start --proxy localhost:4000 --files "**/*"
</code></pre></div></div>
<p>Run <code class="language-plaintext highlighter-rouge">foreman start</code> and your site should reload whenever you change your CSS, JS, or content. Mission accomplished!</p>Hans SchnedlitzI know. What an unholy union. Why would anyone do this? Why would anyone want that?Continuous Deployment with GitHub Actions and Kamal2024-01-07T15:00:00+00:002024-01-07T15:00:00+00:00https://hansschnedlitz.com/2024/01/07/continuous-deployment-with-github-actions-and-kamal<p><a href="https://kamal-deploy.org/">Kamal</a> is a wonderfully simple way to deploy your applications anywhere. It will also be <a href="https://github.com/rails/rails/issues/50441">included by default in Rails 8</a>. Kamal is trivial, but I don’t recommend using it on your development machine.</p>
<p>From experience working on an oldish laptop, I can tell you that building Docker images locally is not fun. Also, why would you, when GitHub Actions are for free!</p>
<p>In this post, I’ll show you how to build a simple CI pipeline with Kamal. We’ll create an application image and deploy it on every push. We’ll also add some simple image caching to speed up the workflow.</p>
<h3 id="the-complete-workflow">The Complete Workflow</h3>
<p>This is what we’ll end up with at the end of this post.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Deploy</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">push</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">main</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">Deploy</span><span class="pi">:</span>
<span class="na">if</span><span class="pi">:</span> <span class="s">${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">DOCKER_BUILDKIT</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">production</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout code</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v3</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Docker Buildx</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">buildx</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/setup-buildx-action@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Login to Docker Hub</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/login-action@v3</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">username</span><span class="pi">:</span> <span class="s">hschne</span>
<span class="na">password</span><span class="pi">:</span> <span class="s">${{ secrets.DOCKER_REGISTRY_KEY }}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set Tag</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">tag</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo "tag=$(git rev-parse "$GITHUB_SHA")" >> $GITHUB_OUTPUT</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build image</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/build-push-action@v5</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">.</span>
<span class="na">builder</span><span class="pi">:</span> <span class="s">${{ steps.buildx.outputs.name }}</span>
<span class="na">push</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">labels</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"service=anonymous-location"</span>
<span class="na">tags</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"hschne/anonymous-location:latest"</span>
<span class="s">"hschne/anonymous-location:${{ steps.tag.outputs.tag }}"</span>
<span class="na">cache-from</span><span class="pi">:</span> <span class="s">type=gha</span>
<span class="na">cache-to</span><span class="pi">:</span> <span class="s">type=gha,mode=max</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">webfactory/ssh-agent@v0.7.0</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ssh-private-key</span><span class="pi">:</span> <span class="s">${{ secrets.SSH_PRIVATE_KEY }}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy command</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">bundle exec kamal deploy --skip-push</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">RAILS_MASTER_KEY</span><span class="pi">:</span> <span class="s">${{ secrets.RAILS_MASTER_KEY }}</span>
<span class="na">KAMAL_REGISTRY_PASSWORD</span><span class="pi">:</span> <span class="s">${{ secrets.DOCKER_REGISTRY_KEY }}</span>
</code></pre></div></div>
<p>That’s something. Well, nobody ever said GitHub actions are succinct. Let’s look at what’s going on here step by step.</p>
<h2 id="step-by-step">Step By Step</h2>
<p>The workflow above is based on <a href="https://dev.to/haukot/how-to-cache-mrsk-deployments-in-ci-52h9">the one in this post</a>. Unfortunately, that post is so old that it still refers to Kamal as Mrsk. I had to make some adjustments to how the image built and deployed.</p>
<p class="notice--info">I won’t go into details on GitHub Actions specifics. If you’re new to GitHub Actions or unfamiliar with one particular piece of syntax, I recommend you check out <a href="https://docs.github.com/en/actions">the documentation</a>.</p>
<p>To build and deploy our Rails application, we need to provide GitHub actions with three <a href="https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions">secrets</a>:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">RAILS_MASTER_KEY</code>: The master key to your Rails application credentials.</li>
<li><code class="language-plaintext highlighter-rouge">DOCKER_REGISTRY_KEY</code>: The API token for pushing and pulling from your container registry.</li>
<li><code class="language-plaintext highlighter-rouge">SSH_PRVIATE_KEY</code>: The SSH key for accessing the server where your app is deployed.</li>
</ul>
<p>We’ll also need to set some environment variables. We are building a production image. We’ll also need to instruct the Docker build step to use Docker Buildkit, as that is one of the requirements of Kamal.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">env</span><span class="pi">:</span>
<span class="na">DOCKER_BUILDKIT</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">production</span>
</code></pre></div></div>
<p>Our deployment workflow needs to do a couple of things. First, we need to check out the application source code. Next, we’ll log into Docker Hub to push our image.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout code</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v3</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Docker Buildx</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">buildx</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/setup-buildx-action@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Login to Docker Hub</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/login-action@v3</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">username</span><span class="pi">:</span> <span class="s">hschne</span>
<span class="na">password</span><span class="pi">:</span> <span class="s">${{ secrets.DOCKER_REGISTRY_KEY }}</span>
</code></pre></div></div>
<p>Kamal uses the git hash of the latest commit to determine which image to deploy, so image tags must match git commit hashes. We define this tag with a separate workflow step.</p>
<p>We use the <a href="https://github.com/docker/build-push-action">docker/build-push-action</a> to build the application image. In addition to setting the correct tag, the image build step must also provide a label matching your service name. Because the image should be pushed to your container registry, we set <code class="language-plaintext highlighter-rouge">push: true</code>, and because we want ludicrous build speed we instruct the build step to utilize the GitHub Actions cache.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set Tag</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">tag</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo "tag=$(git rev-parse "$GITHUB_SHA")" >> $GITHUB_OUTPUT</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build image</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/build-push-action@v5</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">.</span>
<span class="na">builder</span><span class="pi">:</span> <span class="s">${{ steps.buildx.outputs.name }}</span>
<span class="na">push</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">labels</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"service=service-name"</span>
<span class="na">tags</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"user/image-name:latest"</span>
<span class="s">"user/image-name:${{ steps.tag.outputs.tag }}"</span>
<span class="na">cache-from</span><span class="pi">:</span> <span class="s">type=gha</span>
<span class="na">cache-to</span><span class="pi">:</span> <span class="s">type=gha,mode=max</span>
</code></pre></div></div>
<p>Once the image has been built and pushed, you only need to trigger the deployment using Kamal. We use the <a href="https://github.com/webfactory/ssh-agent">webfactory/ssh-agent</a> to establish a connection to our production server. After installing the required Ruby dependencies, it’s only a matter of running Kamal. As the image is already built and pushed, we use the <code class="language-plaintext highlighter-rouge">--skip-push</code> flag.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">webfactory/ssh-agent@v0.7.0</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ssh-private-key</span><span class="pi">:</span> <span class="s">${{ secrets.SSH_PRIVATE_KEY }}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy command</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">bundle exec kamal deploy --skip-push</span>
</code></pre></div></div>
<p>And that’s it! If you’ve enjoyed this post or have any other tips on how to use Kamal together with GitHub Actions, let me know! 🙂</p>Hans SchnedlitzKamal is a wonderfully simple way to deploy your applications anywhere. It will also be included by default in Rails 8. Kamal is trivial, but I don’t recommend using it on your development machine.The Simplest Static Site Generator2023-11-21T12:04:00+00:002023-11-21T12:04:00+00:00https://hansschnedlitz.com/2023/11/21/the-simplest-static-site-generator<p>Sometimes, you want to build a simple HTML page and populate it with some data. You may have some JSON lying around and want to make a simple website to visualize its contents. Or perhaps you want to show off <a href="https://www.hansschnedlitz.com/bookshelf/">how many books you read</a> in the last few years.</p>
<p>In any case, we’re talking about a small set of data and a <em>single</em> HTML page here. Would you use a static site generator for that? Which one? Jekyll? Hugo, Gatsby? Isn’t that overkill for what we’re trying to do?</p>
<p>I was pondering questions like this when I came across <a href="https://pandoc.org/MANUAL.html#templates">Pandoc Templates</a>. Turns out there is a way to generate static sites that is much simpler than most other options out there.</p>
<p class="notice--info">Alright, this might not be the absolutely simplest way to generate a static site. You got me. I bet someone, somewhere builds their static sites using a one-line Perl script. Jokes aside, let me know if you find an even simpler approach to building static sites.</p>
<h2 id="enter-the-template">Enter The Template</h2>
<p>A Pandoc template is not much to look at. It’s just an HTML file filled with some special syntax sprinkled in. Syntax that allows you to write <a href="https://pandoc.org/MANUAL.html#conditionals">conditionals</a> or <a href="https://pandoc.org/MANUAL.html#for-loops">loops</a> as well as do some other stuff. Let’s say we want to list authors and their respective books to simplify things. To do so, let’s create <code class="language-plaintext highlighter-rouge">template.html</code>:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html</span> <span class="na">lang=</span><span class="s">"en"</span><span class="nt">></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"utf-8"</span> <span class="nt">/></span>
<span class="nt"><meta</span> <span class="na">name=</span><span class="s">"date"</span> <span class="na">content=</span><span class="s">"$date-meta$"</span> <span class="nt">/></span>
<span class="nt"><title></span>$title$<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><section></span>
$for(authors)$
<span class="nt"><h2</span> <span class="na">class=</span><span class="s">"author"</span><span class="nt">></span>$authors.name$<span class="nt"></h2></span>
<span class="nt"><ul></span>
$for(authors.books)$
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"book"</span><span class="nt">></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"$authors.books.link$"</span><span class="nt">></span>$authors.books.name$<span class="nt"></a></span>
<span class="nt"></li></span>
$endfor$
<span class="nt"></ul></span>
$endfor$
<span class="nt"></section></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>Now that we have a template, we must populate it with some data. I’ve found the most uncomplicated way to do so is to create a markdown file that contains the data as <a href="https://pandoc.org/chunkedhtml-demo/8.10-metadata-blocks.html#extension-yaml_metadata_block">metadata</a>. We keep our data in <code class="language-plaintext highlighter-rouge">authors.md</code>.</p>
<div class="language-md highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="na">title</span><span class="pi">:</span> <span class="s">Authors and their Books</span>
<span class="na">authors</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">James S.A. Corey</span>
<span class="na">books</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Leviathan Wakes</span>
<span class="na">link</span><span class="pi">:</span> <span class="s">https://www.goodreads.com/book/show/8855321-leviathan-wakes</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Leviathan Falls</span>
<span class="na">link</span><span class="pi">:</span> <span class="s">https://www.goodreads.com/book/show/28335699-leviathan-falls</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Margaret Atwood</span>
<span class="na">books</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">The Handmaid's Tale</span>
<span class="na">link</span><span class="pi">:</span> <span class="s">https://www.goodreads.com/book/show/38447.The_Handmaid_s_Tale</span>
<span class="nn">---</span>
</code></pre></div></div>
<p class="notice--info">I found Markdown documents with embedded metadata easy enough to work with. If you prefer specific JSON or YML files, you can use the <code class="language-plaintext highlighter-rouge">--metadata-file</code> flag to pass them to your template. See also the other <a href="https://pandoc.org/MANUAL.html#reader-options">reader options</a>.</p>
<p>Then, run <a href="https://pandoc.org/">Pandoc</a> to generate your static site. And that’s it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pandoc <span class="nt">--standalone</span> <span class="nt">--template</span> template.html authors.md <span class="nt">-o</span> index.html
</code></pre></div></div>
<p>I’ll take Pandoc over other options every day of the week for this particular use case. What about you?</p>
<h2 id="i-dont-want-to-install-pandoc">I Don’t Want to Install Pandoc!</h2>
<p>I know, right? I don’t want to either. Let’s use Docker instead. Put this into your <code class="language-plaintext highlighter-rouge">.aliases</code> and use <code class="language-plaintext highlighter-rouge">pandock</code> rather than <code class="language-plaintext highlighter-rouge">pandoc</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">alias </span><span class="nv">pandock</span><span class="o">=</span><span class="s1">'docker run --rm -v "$(pwd):/data" -u $(id -u):$(id -g) pandoc/latex'</span>
</code></pre></div></div>Hans SchnedlitzSometimes, you want to build a simple HTML page and populate it with some data. You may have some JSON lying around and want to make a simple website to visualize its contents. Or perhaps you want to show off how many books you read in the last few years.(Ab)Using Single Table Inheritance to Refactor Fat Models2021-07-24T17:00:00+00:002021-07-24T17:00:00+00:00https://hansschnedlitz.com/2021/07/24/ab-using-single-table-inheritance-to-simplify-your-legacy-models<p>How to deal with a model that tries to do too much? Consider something like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="n">validates</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">presence: </span><span class="kp">true</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'green'</span><span class="p">]</span> <span class="p">},</span> <span class="ss">if: </span><span class="o">-></span> <span class="p">{</span> <span class="nb">name</span> <span class="o">==</span> <span class="s1">'AVOCADO'</span><span class="p">}</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'yellow'</span><span class="p">]</span> <span class="p">},</span> <span class="ss">if: </span><span class="o">-></span> <span class="p">{</span> <span class="nb">name</span> <span class="o">==</span> <span class="s1">'POTATO'</span><span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>It makes sense to split this up into three classes: <code class="language-plaintext highlighter-rouge">Vegetable</code>, <code class="language-plaintext highlighter-rouge">Avocado</code> and <code class="language-plaintext highlighter-rouge">Potato</code>. Alright, this particular class isn’t so bad, but imagine <code class="language-plaintext highlighter-rouge">Vegetable</code> being hundreds of lines long and containing dozens of validations like that. Yikes.</p>
<figure>
<img src="https://hansschnedlitz.com/assets/images/posts/2021-07-25/potato.jpg" alt="A big Potato" class="align-center" />
<figcaption>Quite the enormous potato!</figcaption>
</figure>
<h2 id="cutting-up-our-fat-model">Cutting Up our Fat Model</h2>
<p>Rails gives us a straightforward way to refactor <code class="language-plaintext highlighter-rouge">Vegetable</code>: <a href="https://api.rubyonrails.org/classes/ActiveRecord/Inheritance.html">Single Table Inheritance</a>. We can create some submodels to split up <code class="language-plaintext highlighter-rouge">Vegetable</code> and improve our code’s cohesion.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ApplicationRecord</span>
<span class="n">validates</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">presence: </span><span class="kp">true</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Avocado</span> <span class="o"><</span> <span class="no">Vegetable</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'green'</span><span class="p">]</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Potato</span> <span class="o"><</span> <span class="no">Vegetable</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'yellow'</span><span class="p">]</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>If you create a new model from scratch, this approach will just work™. But if you are working with existing code, as in our case, things tend not to be so simple. Rails makes two assumptions when you use single table inheritance:</p>
<ul>
<li>The subtype of your model is designated by a column <code class="language-plaintext highlighter-rouge">type</code>.</li>
<li>The <code class="language-plaintext highlighter-rouge">type</code> column contains the literal name of your subtypes, e.g. <code class="language-plaintext highlighter-rouge">Avocado</code>, <code class="language-plaintext highlighter-rouge">Potato</code>.</li>
</ul>
<p>Our models don’t adhere to these requirements. Our database looks like this:</p>
<table>
<thead>
<tr>
<th>id</th>
<th>name</th>
<th>color</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>VEGETABLE</td>
<td><code class="language-plaintext highlighter-rouge">nil</code></td>
</tr>
<tr>
<td>2</td>
<td>AVOCADO</td>
<td>green</td>
</tr>
<tr>
<td>3</td>
<td>POTATO</td>
<td>yellow</td>
</tr>
</tbody>
</table>
<p>The value that distinguishes the types of vegetables lives in the <code class="language-plaintext highlighter-rouge">name</code> column rather than the <code class="language-plaintext highlighter-rouge">type</code> column. Also, the names themselves are uppercase versions of our subclass-names: <code class="language-plaintext highlighter-rouge">AVOCADO</code> rather than <code class="language-plaintext highlighter-rouge">Avocado</code> and so on. To solve these issues you <em>could</em> migrate your data - and if you can, you definitely should! But sometimes that just isn’t an option.</p>
<p>Luckily, there are ways to shoehorn single table inheritance into models like these.</p>
<h2 id="adapting-single-table-inheritance">Adapting Single Table Inheritance</h2>
<p>After splitting up your models, you may try to run and run a query to get all potatoes:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Potato</span><span class="p">.</span><span class="nf">all</span>
</code></pre></div></div>
<p>Surprise. Instead of returning only a single record, all vegetables are returned. This is not at all what we wanted! We need to tell Rails about our non-standard inheritance column <code class="language-plaintext highlighter-rouge">name</code>. To do so, we can update the parent model <code class="language-plaintext highlighter-rouge">Vegetable</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ApplicationRecord</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">inheritance_column</span><span class="o">=</span><span class="s1">'name'</span>
<span class="o">...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Unfortunately, doing so will not only make our queries <em>still</em> return nonsense - <code class="language-plaintext highlighter-rouge">Potato.all</code> now returns no records at all - but also break a bunch of other things. Even creating new vegetables now fails:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Raises ActiveRecord::SubclassNotFound (The single-table inheritance mechanism failed to locate the subclass: 'POTATO'...</span>
<span class="no">Vegetable</span><span class="p">.</span><span class="nf">create</span><span class="p">(</span><span class="ss">name: </span><span class="s1">'POTATO'</span><span class="p">)</span>
</code></pre></div></div>
<p>Rails expects the inheritance column to contain the class name of the specific sub-type, <code class="language-plaintext highlighter-rouge">Potato</code> rather than <code class="language-plaintext highlighter-rouge">POTATO</code>. Under the hood, it executes <code class="language-plaintext highlighter-rouge">POTATO.constantize</code>, which of course doesn’t work. We have to change how Rails locates the types used to instantiate STI records. But how?</p>
<p>Enter <code class="language-plaintext highlighter-rouge">sti_class_for</code>. By overwriting this method, we can customize which types are used for instantiation:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">inheritance_column</span> <span class="o">=</span> <span class="s2">"name"</span>
<span class="k">class</span> <span class="o"><<</span> <span class="nb">self</span>
<span class="k">def</span> <span class="nf">sti_class_for</span><span class="p">(</span><span class="n">type_name</span><span class="p">)</span>
<span class="k">super</span><span class="p">(</span><span class="n">type_name</span><span class="p">.</span><span class="nf">dowcase</span><span class="p">.</span><span class="nf">camelize</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p class="notice--warning"><strong>Warning</strong>: <code class="language-plaintext highlighter-rouge">sti_class_for</code> was added in Rails 6.1. If you are stuck with an earlier version of Rails, you can use <code class="language-plaintext highlighter-rouge">find_sti_class</code> instead. It does pretty much the same thing but is <em>private</em>. You can still overwrite it all the same of course, just be careful.</p>
<p>That fixes querying for vegetables. However, querying our subclasses and creating new sub-records <em>still</em> does not work like we want it to:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Potato</span><span class="p">.</span><span class="nf">all</span>
<span class="o">=></span> <span class="c1">#<ActiveRecord::Relation []></span>
<span class="no">Potato</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">color: </span><span class="s1">'yellow'</span><span class="p">)</span>
<span class="o">=></span> <span class="c1">#<Potato id: 6, name: "Potato", color: "yellow",</span>
</code></pre></div></div>
<p>Although we overwrite <code class="language-plaintext highlighter-rouge">sti_class_for</code>, Rails uses the wrong <code class="language-plaintext highlighter-rouge">name</code> values. We have to ask ourselves: How does Rails know which values to put into the inheritance column when instantiating child records? It uses <code class="language-plaintext highlighter-rouge">sti_name</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">sti_name</span>
<span class="n">store_full_sti_class</span> <span class="p">?</span> <span class="nb">name</span> <span class="p">:</span> <span class="nb">name</span><span class="p">.</span><span class="nf">demodulize</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You probably know where this is going. Let’s overwrite <code class="language-plaintext highlighter-rouge">sti_name</code> as well:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">inheritance_column</span> <span class="o">=</span> <span class="s2">"name"</span>
<span class="k">class</span> <span class="o"><<</span> <span class="nb">self</span>
<span class="k">def</span> <span class="nf">sti_class_for</span><span class="p">(</span><span class="n">type_name</span><span class="p">)</span>
<span class="k">super</span><span class="p">(</span><span class="n">type_name</span><span class="p">.</span><span class="nf">lower</span><span class="p">.</span><span class="nf">camelize</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">sti_name</span>
<span class="nb">name</span><span class="p">.</span><span class="nf">upcase</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Success! We have refactored the chonky <code class="language-plaintext highlighter-rouge">Vegetable</code>, and we can work with our subclasses just like we would expect:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Potato</span><span class="p">.</span><span class="nf">all</span>
<span class="o">=></span> <span class="c1">#<ActiveRecord::Relation [#<Potato id: 3, name: "POTATO", color: "yellow", created_at: "2021-07-25 14:41:00.032041000 +0000", updated_at: "2021-07-25 14:41:00.032041000 +0000">]></span>
<span class="no">Potato</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">color: </span><span class="s1">'yellow'</span><span class="p">)</span>
<span class="o">=></span> <span class="c1">#<Potato id: nil, name: "POTATO", color: "yellow", created_at: nil, updated_at: nil></span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Single Table Inheritance can be useful to re-organize existing models that have grown too large. Ideally, you’d never have to reach for this approach, but when are things ever try ideal? If you got any use out of this short guide let me know on <a href="https://twitter.com/hschnedlitz">twitter</a> 🤗</p>Hans SchnedlitzHow to deal with a model that tries to do too much? Consider something like this:Real-Time Command Line Applications with Action Cable and Thor2021-04-04T17:00:00+00:002021-04-04T17:00:00+00:00https://hansschnedlitz.com/2021/04/04/build-real-time-clis-with-actioncable<p>If you build a Rails application that has any kind of real-time feature, chances are you use <a href="https://guides.rubyonrails.org/action_cable_overview.html">Action Cable</a>.</p>
<p>Action Cable allows you to build nice things such as feeds that automatically refresh as new content is published, or editors that display a list of users currently working on a document. Under the hood, it uses Websockets to stream changes to clients as they happen.</p>
<p>The most commonly used client is, of course, the web browser. But that doesn’t mean you can’t leverage Action Cable when using other kinds of clients - such as command line applications.</p>
<p>Imagine a command line client that triggers some long-running job on the server. Wouldn’t it be nice to give users live updates on how that job is advancing?</p>
<p>In this guide, I’ll show you how to build exactly that. We’ll create a command line app that connects to an Action Cable server, triggers a lengthy background job, and then displays live updates about its progress.</p>
<p>A (moving?) picture tells more than a thousand words.</p>
<p><a href="https://hansschnedlitz.com/assets/images/posts/2021-04-05/demo.gif"><img src="https://hansschnedlitz.com/assets/images/posts/2021-04-05/demo.gif" alt="demo" /></a></p>
<p class="notice--info">This is a long post. If you have no patience for words, you can find the source code of the result on <a href="https://github.com/hschne/actioncable-cli">GitHub</a>.</p>
<h2 id="the-server">The Server</h2>
<p>To start things off let’s create a new Rails application. We don’t need most of Rails’ functionality in this guide, so we can skip a lot of things.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails new actioncable-cli \
--skip-action-mailer \
--skip-action-mailbox \
--skip-action-text \
--skip-active-job \
--skip-active-record \
--skip-active-storage \
--skip-javascript \
--skip-jbuilder \
--skip-spring \
--skip-test \
--skip-system-test \
--skip-webpack-install \
--skip-turbolinks
</code></pre></div></div>
<p>We will create our command line app later. First, we have to make some changes to the Action cable connection. Usually, clients provide information about the currently logged-in user, for example through session cookies, which then serves as a connection identifier. See the <a href="https://guides.rubyonrails.org/action_cable_overview.html#connection-setup">official connection docs</a>.</p>
<p>Our command line app offers no such thing. We could add some sort of authentication mechanism, but to keep things simple we won’t. We will use a simple UUID to identify connections.</p>
<p>Open and modify <code class="language-plaintext highlighter-rouge">app/channels/application_cable/connection.rb</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">ApplicationCable</span>
<span class="k">class</span> <span class="nc">Connection</span> <span class="o"><</span> <span class="no">ActionCable</span><span class="o">::</span><span class="no">Connection</span><span class="o">::</span><span class="no">Base</span>
<span class="n">identified_by</span> <span class="ss">:client_id</span>
<span class="k">def</span> <span class="nf">connect</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">client_id</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">params</span><span class="p">[</span><span class="ss">:client_id</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Next, create a worker channel, through which we’ll later publish updates. Create a new file <code class="language-plaintext highlighter-rouge">app/channels/worker_channel.rb</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">WorkerChannel</span> <span class="o"><</span> <span class="no">ApplicationCable</span><span class="o">::</span><span class="no">Channel</span>
<span class="k">def</span> <span class="nf">subscribed</span>
<span class="n">stream_for</span> <span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">unsubscribed</span>
<span class="n">stop_all_streams</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Because we’ll be connecting from the command line, we’ll have to disable some security measures Rails enables by default. Uncomment this line in <code class="language-plaintext highlighter-rouge">development.rb</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>config.action_cable.disable_request_forgery_protection = true
</code></pre></div></div>
<p>Now that we have the connection and channel set up, let’s create a background job. We’ll be using <a href="https://github.com/mperham/sidekiq">Sidekiq</a>, so add this to your Gemfile:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'sidekiq'</span><span class="p">,</span> <span class="s1">'~> 6.1'</span>
</code></pre></div></div>
<p>We must also make sure that <a href="https://redis.io/">Redis</a> is up and running because Sidekiq relies on that for managing background workers. If you use <code class="language-plaintext highlighter-rouge">docker-compose</code>, add the following to <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">actioncable-cli-redis</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">redis</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">redis</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">6379:6379</span>
</code></pre></div></div>
<p>Next, create a new worker in <code class="language-plaintext highlighter-rouge">app/workers</code>. It won’t be doing any actual work, mostly it will be taking a nap.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Worker</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span>
<span class="k">def</span> <span class="nf">perform</span>
<span class="n">steps</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="n">steps</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">progress</span><span class="o">|</span>
<span class="nb">sleep</span><span class="p">(</span><span class="nb">rand</span><span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="mi">3</span><span class="p">))</span>
<span class="no">Sidekiq</span><span class="p">.</span><span class="nf">logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="s2">"Step </span><span class="si">#{</span><span class="n">progress</span><span class="si">}</span><span class="s2"> for client"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>To offer a way to start the background job we just created, add a new controller with the following contents:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">WorkersController</span> <span class="o"><</span> <span class="no">ApplicationController</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="no">Worker</span><span class="p">.</span><span class="nf">perform_async</span>
<span class="n">head</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Don’t forget to also add a new route to your <code class="language-plaintext highlighter-rouge">routes.rb</code>!</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">get</span> <span class="s1">'/workers/start'</span><span class="p">,</span> <span class="ss">to: </span><span class="s1">'workers#start'</span>
</code></pre></div></div>
<p>This is a good point to stop and check how badly broken everything is :crossed_fingers:</p>
<p>Start your Rails app, Sidekiq, and start the worker. If all is in order, you should see your worker writing to the Sidekiq logs.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails start
bundle <span class="nb">exec </span>sidekiq
<span class="c"># Send a request to trigger the worker</span>
curl <span class="s2">"http://localhost:3000/workers/start"</span>
</code></pre></div></div>
<p>Success? Then on to the next part.</p>
<h2 id="the-command-line-app">The Command Line App</h2>
<p>Our command line application will offer just a single command - one that starts the worker. <a href="https://github.com/erikhuda/thor">Thor</a> is a simple way to create command line apps, and it’s bundled with Rails, so we’ll be using that to implement that command.</p>
<p>Create <code class="language-plaintext highlighter-rouge">worker.thor</code> in your <code class="language-plaintext highlighter-rouge">lib/tasks</code> directory:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="nb">puts</span> <span class="s1">'Hello there!'</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You can test your command using <code class="language-plaintext highlighter-rouge">bundle exec thor worker:start</code>.</p>
<p>To receive live updates using Websockets we’ll need a Websocket client. I used <a href="https://socketry.github.io/async-websocket/">async-websocket</a>. Add it to your Gemfile:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'async-websocket'</span><span class="p">,</span> <span class="s1">'~> 0.17'</span>
</code></pre></div></div>
<p>Then update your command to connect to the server. Note that we generate a UUID to identify the connection. Remember that we adapted the Action Cable connection to make use of this <code class="language-plaintext highlighter-rouge">client_id</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="nb">require</span> <span class="s1">'securerandom'</span>
<span class="nb">require</span> <span class="s1">'async'</span>
<span class="nb">require</span> <span class="s1">'async/http/endpoint'</span>
<span class="nb">require</span> <span class="s1">'async/websocket/client'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="vi">@client_id</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">uuid</span>
<span class="n">url</span> <span class="o">=</span> <span class="s2">"ws://localhost:3000/cable?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Async</span> <span class="k">do</span> <span class="o">|</span><span class="n">_</span><span class="o">|</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="no">Async</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Endpoint</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">Async</span><span class="o">::</span><span class="no">WebSocket</span><span class="o">::</span><span class="no">Client</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">endpoint</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">connection</span><span class="o">|</span>
<span class="k">while</span> <span class="p">(</span><span class="n">message</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">read</span><span class="p">)</span>
<span class="nb">puts</span> <span class="n">message</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Run the command and check the server logs. You should see that a connection has been established, and should start receiving ping messages on the command line.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bundle exec thor worker:start
{:type=>"welcome"}
{:type=>"ping", :message=>1617639988}
...
</code></pre></div></div>
<p>Now we need to subscribe to the worker channel. As soon as the subscription was confirmed, we are ready to receive messages. We can then start the worker.</p>
<p>Adapt the Thor command as follows:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="nb">require</span> <span class="s1">'securerandom'</span>
<span class="nb">require</span> <span class="s1">'net/http'</span>
<span class="nb">require</span> <span class="s1">'async'</span>
<span class="nb">require</span> <span class="s1">'async/http/endpoint'</span>
<span class="nb">require</span> <span class="s1">'async/websocket/client'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="vi">@client_id</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">uuid</span>
<span class="n">url</span> <span class="o">=</span> <span class="s2">"ws://localhost:3000/cable?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Async</span> <span class="k">do</span> <span class="o">|</span><span class="n">_</span><span class="o">|</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="no">Async</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Endpoint</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">Async</span><span class="o">::</span><span class="no">WebSocket</span><span class="o">::</span><span class="no">Client</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">endpoint</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">connection</span><span class="o">|</span>
<span class="k">while</span> <span class="p">(</span><span class="n">message</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">read</span><span class="p">)</span>
<span class="n">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">type</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="k">case</span> <span class="n">type</span>
<span class="k">when</span> <span class="s1">'welcome'</span>
<span class="n">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">'confirm_subscription'</span>
<span class="n">on_subscribed</span>
<span class="k">else</span>
<span class="nb">puts</span> <span class="n">message</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">command: </span><span class="s1">'subscribe'</span><span class="p">,</span> <span class="ss">identifier: </span><span class="p">{</span> <span class="ss">channel: </span><span class="s1">'WorkerChannel'</span> <span class="p">}.</span><span class="nf">to_json</span> <span class="p">}</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">flush</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_subscribed</span>
<span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">start</span><span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">3000</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">http</span><span class="o">|</span>
<span class="n">http</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="s2">"/workers/start?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>All that is missing is to stream updates from the worker to the connected clients. We’ll have to make some small changes to our worker and worker controller:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Worker</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span>
<span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">client_id</span><span class="p">)</span>
<span class="n">steps</span> <span class="o">=</span> <span class="mi">5</span>
<span class="no">WorkerChannel</span><span class="p">.</span><span class="nf">broadcast_to</span><span class="p">(</span><span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="ss">type: :worker_started</span><span class="p">,</span> <span class="ss">total: </span><span class="n">steps</span><span class="p">)</span>
<span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="n">steps</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">progress</span><span class="o">|</span>
<span class="nb">sleep</span><span class="p">(</span><span class="nb">rand</span><span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="mi">3</span><span class="p">))</span>
<span class="no">Sidekiq</span><span class="p">.</span><span class="nf">logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="s2">"Step </span><span class="si">#{</span><span class="n">progress</span><span class="si">}</span><span class="s2"> for client </span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="no">WorkerChannel</span><span class="p">.</span><span class="nf">broadcast_to</span><span class="p">(</span><span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="ss">type: :worker_progress</span><span class="p">,</span> <span class="ss">progress: </span><span class="n">progress</span><span class="p">)</span>
<span class="k">end</span>
<span class="no">WorkerChannel</span><span class="p">.</span><span class="nf">broadcast_to</span><span class="p">(</span><span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="ss">type: :worker_done</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">WorkersController</span> <span class="o"><</span> <span class="no">ApplicationController</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="no">Worker</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span><span class="n">params</span><span class="p">[</span><span class="ss">:client_id</span><span class="p">])</span>
<span class="n">head</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Note that the worker uses the <code class="language-plaintext highlighter-rouge">client_id</code> to publish messages to the correct clients. We publish messages when the worker has started, when there is progress, and when the worker has finished.</p>
<p>We’ll update the command line app to handle these messages. Let’s also add <code class="language-plaintext highlighter-rouge">ruby_progressbar</code> so we can display the progress to the user.</p>
<p>Add this to your Gemfile.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'ruby-progressbar'</span><span class="p">,</span> <span class="s1">'~> 1.11'</span>
</code></pre></div></div>
<p>Then update the Thor command once again. In the end, it should look like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="nb">require</span> <span class="s1">'securerandom'</span>
<span class="nb">require</span> <span class="s1">'net/http'</span>
<span class="nb">require</span> <span class="s1">'async'</span>
<span class="nb">require</span> <span class="s1">'async/io/stream'</span>
<span class="nb">require</span> <span class="s1">'async/http/endpoint'</span>
<span class="nb">require</span> <span class="s1">'async/websocket/client'</span>
<span class="nb">require</span> <span class="s1">'ruby-progressbar'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="vi">@client_id</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">uuid</span>
<span class="n">url</span> <span class="o">=</span> <span class="s2">"ws://localhost:3000/cable?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Async</span> <span class="k">do</span> <span class="o">|</span><span class="n">_</span><span class="o">|</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="no">Async</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Endpoint</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">Async</span><span class="o">::</span><span class="no">WebSocket</span><span class="o">::</span><span class="no">Client</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">endpoint</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">connection</span><span class="o">|</span>
<span class="k">while</span> <span class="p">(</span><span class="n">message</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">read</span><span class="p">)</span>
<span class="n">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">if</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="n">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">handle_channel_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">type</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="k">case</span> <span class="n">type</span>
<span class="k">when</span> <span class="s1">'welcome'</span>
<span class="n">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">'confirm_subscription'</span>
<span class="n">on_subscribed</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_channel_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:message</span><span class="p">]</span>
<span class="n">type</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="k">case</span> <span class="n">type</span>
<span class="k">when</span> <span class="s1">'worker_started'</span>
<span class="n">total</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:total</span><span class="p">]</span>
<span class="vi">@bar</span> <span class="o">=</span> <span class="no">ProgressBar</span><span class="p">.</span><span class="nf">create</span><span class="p">(</span><span class="ss">title: </span><span class="s1">'Worker Progress'</span><span class="p">,</span> <span class="ss">total: </span><span class="n">total</span><span class="p">,</span> <span class="ss">format: </span><span class="s1">'%t %B %c/%C %P%%'</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">'worker_progress'</span>
<span class="vi">@bar</span><span class="p">.</span><span class="nf">increment</span>
<span class="k">when</span> <span class="s1">'worker_done'</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">close</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">command: </span><span class="s1">'subscribe'</span><span class="p">,</span> <span class="ss">identifier: </span><span class="p">{</span> <span class="ss">channel: </span><span class="s1">'WorkerChannel'</span> <span class="p">}.</span><span class="nf">to_json</span> <span class="p">}</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">flush</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_subscribed</span>
<span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">start</span><span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">3000</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">http</span><span class="o">|</span>
<span class="n">http</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="s2">"/workers/start?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The most important change here is the addition of <code class="language-plaintext highlighter-rouge">handle_channel_message</code> where we handle the messages we receive from the worker to create and update the progress bar.</p>
<p>Before wrapping up, we need to make one final change. Update <code class="language-plaintext highlighter-rouge">cable.yml</code> to use Redis in development. We need to do this so that our Sidekiq process knows about subscriptions made using the main process.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">development</span><span class="pi">:</span>
<span class="na">adapter</span><span class="pi">:</span> <span class="s">redis</span>
<span class="na">url</span><span class="pi">:</span> <span class="s"><%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/0" } %></span>
</code></pre></div></div>
<p class="notice--info">The default mechanism for managing Action Cable connections in development is <code class="language-plaintext highlighter-rouge">async</code>, which uses in-memory structures. These are accessible only by the current process. That is no good when multiple processes need to utilize the same connections.</p>
<p>Restart your Rails server and, for good measure, Sidekiq process if you haven’t already and run the worker command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle exec thor worker:start
</code></pre></div></div>
<h2 id="just-the-beginning">Just the beginning…</h2>
<p>This guide is done, but the story of ActionCable and command line applications isn’t. Updating a progress bar is nice and all, but it is only scratching the surface.</p>
<p>There is much more to explore. How about streaming process logs live to clients? Or what about streaming user inputs directly to the server?</p>
<p>Anything is possible - you just have to try it! :woman_scientist:</p>Hans SchnedlitzIf you build a Rails application that has any kind of real-time feature, chances are you use Action Cable.Processing images with ActiveStorage and Imgproxy2021-03-19T09:11:00+00:002021-03-19T09:11:00+00:00https://hansschnedlitz.com/2021/03/19/user-avatars-with-imgproxy-and-activestorage<p><a href="https://edgeguides.rubyonrails.org/active_storage_overview.html">ActiveStorage</a> has a nifty feature that allows you to serve variants of uploaded images. Think of a user uploading a profile image. You won’t need the full-sized original most of the time. Instead, you can serve smaller versions of that image, which, of course, consume less bandwidth and load faster.</p>
<p>ActiveStorage uses the <a href="https://github.com/janko/image_processing">image_processing</a> gem under the hood to accomplish this. Here is how you could render downsized version of a user avatar:</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">variant</span><span class="p">(</span><span class="ss">resize: </span><span class="s2">"100x100!"</span><span class="p">)</span> <span class="cp">%></span>
</code></pre></div></div>
<p>I’ve found ActiveStorage’s variant mechanism very easy to use, but it does have its downsides. Underlying libraries, such as <code class="language-plaintext highlighter-rouge">libvips</code> or <code class="language-plaintext highlighter-rouge">minimagick</code> have to be installed and kept up to date. Additionally, transforming images is CPU- and memory-intensive work, and doing this on your app servers may cause issues as your app grows.</p>
<p><a href="https://github.com/imgproxy/imgproxy">Imgproxy</a> is an alternative way to process images. A small service written in Go, it does one thing, and one thing only: processing images. As such, it allows us to offload the job of processing images to specialized machines, alleviating some of the problems described above.</p>
<p>To show you how that works we’ll create a small application that allows users to upload profile images. We’ll then use <code class="language-plaintext highlighter-rouge">image_processing</code> and later Imgproxy to serve downsized versions of those same images.</p>
<p class="notice--info">Words are nice and all, but code is pretty cool too, right? You can find the source code for this guide <a href="https://github.com/hschne/avatars">on GitHub</a> if you prefer that.</p>
<h2 id="setting-up-the-app">Setting up the App</h2>
<p>The simplest way to get a user model and user profile page is by using <a href="https://github.com/heartcombo/devise">Devise</a>.</p>
<p>I set up a demo application called ‘Avatars’ using <a href="https://github.com/hschne/schienenzeppelin">Schienenzeppelin</a>, my own Rails template that is configured with Devise, Tailwind, and other libraries.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sz avatars
</code></pre></div></div>
<p>If you are starting with a fresh app, you’ll have to install and configure Devise yourself. I recommend you follow the instructions <a href="https://github.com/heartcombo/devise#getting-started">here</a>.</p>
<p>Once your app is up and running, install ActiveStorage:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails active_storage:install
</code></pre></div></div>
<p>Next, update your user model to allow attaching images to it.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">has_one_attached</span> <span class="ss">:avatar</span>
</code></pre></div></div>
<p>We’ll now update some views and controllers to enable users to upload profile images. There are <a href="https://github.com/heartcombo/devise#strong-parameters"> several ways </a> to go about this, I prefer ejecting Devises controllers and modifying them as necessary.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails generate devise:controllers <span class="nb">users</span>
</code></pre></div></div>
<p>We’ll need to update only the <code class="language-plaintext highlighter-rouge">registrations_controller.rb</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Users::RegistrationsController</span> <span class="o"><</span> <span class="no">Devise</span><span class="o">::</span><span class="no">RegistrationsController</span>
<span class="n">before_action</span> <span class="ss">:configure_account_update_params</span><span class="p">,</span> <span class="ss">only: </span><span class="p">[</span><span class="ss">:update</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">configure_account_update_params</span>
<span class="n">devise_parameter_sanitizer</span><span class="p">.</span><span class="nf">permit</span><span class="p">(</span><span class="ss">:account_update</span><span class="p">,</span> <span class="ss">keys: </span><span class="sx">%i[name avatar]</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>To tell Devise to use this controller, update <code class="language-plaintext highlighter-rouge">routes.rb</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">devise_for</span> <span class="ss">:users</span><span class="p">,</span> <span class="ss">controllers: </span><span class="p">{</span> <span class="ss">registrations: </span><span class="s1">'users/registrations'</span> <span class="p">}</span>
</code></pre></div></div>
<p>We are now ready to update the view where users can update their profile. Open <code class="language-plaintext highlighter-rouge">app/views/devise/registrations/edit.html.erb</code>. If that file is not present you have to run <code class="language-plaintext highlighter-rouge">rails generate devise:views users</code> first.</p>
<p>The code below allows users to upload a new profile image, displays it if it is present, and shows a fallback icon using <a href="https://github.com/jamesmartin/inline_svg">inline_svg</a> otherwise.</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">class=</span><span class="s">"input-group"</span><span class="nt">></span>
<span class="cp"><%</span> <span class="k">if</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">attached?</span> <span class="cp">%></span>
<span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">url_for</span><span class="p">(</span><span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">),</span> <span class="ss">class: </span><span class="s2">"rounded"</span> <span class="cp">%></span>
<span class="cp"><%</span> <span class="k">else</span> <span class="cp">%></span>
<span class="cp"><%=</span> <span class="n">inline_svg_pack_tag</span><span class="p">(</span><span class="s1">'media/images/user.svg'</span><span class="p">,</span> <span class="ss">class: </span><span class="s2">"rounded"</span><span class="p">,</span> <span class="ss">size: </span><span class="s2">"5rem * 5rem"</span><span class="p">)</span> <span class="cp">%></span>
<span class="cp"><%</span> <span class="k">end</span> <span class="cp">%></span>
<span class="cp"><%=</span> <span class="n">f</span><span class="p">.</span><span class="nf">file_field</span> <span class="ss">:avatar</span> <span class="cp">%></span>
<span class="nt"></div></span>
</code></pre></div></div>
<p>With these changes, you should be able to upload user profile images and display them.</p>
<p><img src="https://hansschnedlitz.com/assets/images/posts/2021-03-19/profile.png" alt="Profile" class="align-center" /></p>
<h2 id="processing-images">Processing Images</h2>
<p>Our current implementation renders exactly the image that the user uploaded. That’s wasteful. What if they upload a 15 megapixel, high-res picture of their face? To serve processed images - variants - we’ll need to install the <code class="language-plaintext highlighter-rouge">image_processing</code> gem.</p>
<p>Add this to your <code class="language-plaintext highlighter-rouge">Gemfile</code> and run <code class="language-plaintext highlighter-rouge">bundle install</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'image_processing'</span>
</code></pre></div></div>
<p>Now replace <code class="language-plaintext highlighter-rouge">url_for(resource.avatar)</code> with <code class="language-plaintext highlighter-rouge">resource.avatar.variant</code> to serve a resized image.</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">variant</span><span class="p">(</span><span class="ss">resize: </span><span class="s2">"100x100!"</span><span class="p">),</span> <span class="ss">class: </span><span class="s2">"rounded"</span> <span class="cp">%></span>
</code></pre></div></div>
<p>You can find additional info on variants in the <a href="https://edgeguides.rubyonrails.org/active_storage_overview.html#transforming-images">official documentation</a>.</p>
<h2 id="adding-imageproxy">Adding Imageproxy</h2>
<p>So far so good. Image upload and processing using <code class="language-plaintext highlighter-rouge">image_processing</code> works, now let’s use Imgproxy. Add the <a href="https://github.com/imgproxy/imgproxy.rb">Imgproxy client</a>, and once again run <code class="language-plaintext highlighter-rouge">bundle install</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'imgproxy'</span>
</code></pre></div></div>
<p>There are several ways to configure the Imgproxy client. I prefer using an initializer, <code class="language-plaintext highlighter-rouge">initializers/imgproxy.rb</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Imgproxy</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">endpoint</span> <span class="o">=</span> <span class="s1">'http://localhost:8080'</span>
<span class="n">config</span><span class="p">.</span><span class="nf">key</span> <span class="o">=</span> <span class="s1">'696d6770726f7879'</span> <span class="c1"># imgproxy</span>
<span class="n">config</span><span class="p">.</span><span class="nf">salt</span> <span class="o">=</span> <span class="s1">'73616c74'</span> <span class="c1"># salt</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Key and salt are not strictly required but enable URL signing, which is a good security practice. You can find more information about URL signing in the <a href="https://docs.imgproxy.net/#/signing_the_url">Imgproxy documentation</a>.</p>
<p>To replace <code class="language-plaintext highlighter-rouge">image_processing</code> with <code class="language-plaintext highlighter-rouge">imgproxy</code> update the view and replace the line where we retrieve the variant with:</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">imgproxy_url</span><span class="p">(</span><span class="ss">width: </span><span class="mi">100</span><span class="p">,</span> <span class="ss">height: </span><span class="mi">100</span><span class="p">,</span> <span class="ss">format: </span><span class="s1">'jpg'</span><span class="p">),</span> <span class="ss">class: </span><span class="s2">"rounded"</span> <span class="cp">%></span>
</code></pre></div></div>
<p class="notice--info">In my testing I faced an issue where <code class="language-plaintext highlighter-rouge">imgproxy_url</code> failed. If that is the case for you as well, you may need to add <code class="language-plaintext highlighter-rouge">Rails.application.routes.default_url_options[:host] = 'localhost:3000'</code> to your <code class="language-plaintext highlighter-rouge">development.rb</code>.</p>
<p>Our Rails app is good-to-go. The last thing remaining is to start the Imgproxy service. If you use <code class="language-plaintext highlighter-rouge">docker-compose</code> add the following to your compose file and run <code class="language-plaintext highlighter-rouge">docker-compose up</code>.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">imgproxy</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">imgproxy</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">darthsim/imgproxy:latest</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">IMGPROXY_KEY=696d6770726f7879</span> <span class="c1"># imgproxy</span>
<span class="pi">-</span> <span class="s">IMGPROXY_SALT=73616c74</span> <span class="c1"># salt</span>
<span class="pi">-</span> <span class="s">IMGPROXY_LOCAL_FILESYSTEM_ROOT=/storage</span>
<span class="na">network_mode</span><span class="pi">:</span> <span class="s">host</span>
</code></pre></div></div>
<p>Note that <code class="language-plaintext highlighter-rouge">network_mode: host</code> is required, as Imgproxy needs to connect to your Rails application to retrieve original images. For alternative ways to run Imgproxy consult the <a href="https://docs.imgproxy.net/#/installation">documentation</a>.</p>
<p>If you reload your profile page, you should see something like this in your Docker logs, which of course means that Imgproxy is doing its job.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>imgproxy | INFO [2021-03-19T07:57:06Z] Completed in 61.071755ms /../s:100:100/plain/http://localhost:3000/rails/active_storage/blobs/redirect/../avatar.png@jpg request_id=7l4aJfTZ6kdjq0Ul1ITx9 method=GET status=200 image_url="http://localhost:3000/rails/active_storage/blobs/redirect/.../Hans.png" processing_options="Width: 100; Height: 100; Format: jpeg"
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Imgproxy can alleviate some of the pains associated with running ActiveStorage as your app grows - even if it requires additional infrastructure. In the context of this post, we didn’t see much of that.</p>
<p>A word of warning: Don’t get all hyped up. Using Imgproxy is probably not worth it unless your app is large enough. But still: It’s good to know what’s out there and what is possible, right?</p>
<p>I hope you learned a thing or two - let me know if you found this useful!</p>Hans SchnedlitzActiveStorage has a nifty feature that allows you to serve variants of uploaded images. Think of a user uploading a profile image. You won’t need the full-sized original most of the time. Instead, you can serve smaller versions of that image, which, of course, consume less bandwidth and load faster.Load testing GraphQL with WRK2021-03-09T17:00:00+00:002021-03-09T17:00:00+00:00https://hansschnedlitz.com/2021/03/09/load-testing-graphql-with-wrk<p>For performance testing <a href="https://github.com/wg/wrk">wrk</a> is one of my favorite tools. Whether you are trying to get a quick benchmark or building a performance test suite - it is fairly simple using <code class="language-plaintext highlighter-rouge">wrk</code>.</p>
<p>That is, as long as your requests are fairly simple. To benchmark a GET request against a classic REST API you could run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk -c <user-count> -t <cpu-core-count> -d 10 --latency https://your-server/api/books
</code></pre></div></div>
<p>Easy as pie, right? But what about <em>less simple</em> requests? Specifically, does <code class="language-plaintext highlighter-rouge">wrk</code> work for, say, <a href="https://graphql.org/">GraphQL</a> APIs?</p>
<p>While are other tools that you could use (for example <a href="https://k6.io/">k6</a>), <code class="language-plaintext highlighter-rouge">wrk</code> does the job well enough - if you aren’t scared of a little <a href="http://www.lua.org/">Lua</a> scripting.</p>
<p class="notice--warning">What you are seeing in this post is pretty much the entirety of all the Lua I’ve written in my entire life. So. Now you know. Take the Lua parts with a grain of salt.</p>
<h2 id="using-lua-to-perform-graphql-queries">Using Lua to perform GraphQL Queries</h2>
<p>You may pass a Lua script to <code class="language-plaintext highlighter-rouge">wrk</code> using the <code class="language-plaintext highlighter-rouge">-s</code> option.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk <span class="nt">-s</span> script.lua https://your-server/api/graphql
</code></pre></div></div>
<p>To perform a GraphQL query the script could look something like this:</p>
<div class="language-lua highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrk</span><span class="p">.</span><span class="n">method</span> <span class="o">=</span> <span class="s2">"POST"</span>
<span class="n">wrk</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s2">"Content-Type"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"application/json"</span>
<span class="n">wrk</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s2">"Accept"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"application/json"</span>
<span class="n">query</span> <span class="o">=</span> <span class="s">[[
query books($ids: [Integer]) {
books(ids: $ids) {
id
name
price
}
}
}
]]</span>
<span class="n">variables</span> <span class="o">=</span> <span class="s">[[
"ids": [1,2,3,4]
]]</span>
<span class="n">wrk</span><span class="p">.</span><span class="n">body</span> <span class="o">=</span><span class="s1">'{"query": "'</span> <span class="o">..</span> <span class="nb">string.gsub</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span> <span class="o">..</span> <span class="s1">'", "variables": {'</span> <span class="o">..</span> <span class="nb">string.gsub</span><span class="p">(</span><span class="n">variables</span><span class="p">,</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span> <span class="o">..</span> <span class="s1">'} }'</span>
</code></pre></div></div>
<p>Even if you have no clue about Lua you can most likely infer what is going on here. A quick explanation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk.method = "POST"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Accept"] = "application/json"
</code></pre></div></div>
<p>Most GraphQL servers require specific headers and HTTP verbs when processing requests. If your API has some specific requirements you will need to change this accordingly.</p>
<div class="language-lua highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">query</span> <span class="o">=</span> <span class="s">[[
query books($ids: [Integer]) {
books(ids: $ids) {
id
name
price
}
}
}
]]</span>
<span class="n">variables</span> <span class="o">=</span> <span class="s">[[
"ids": [1,2,3,4]
]]</span>
</code></pre></div></div>
<p>Double-brackets (<code class="language-plaintext highlighter-rouge">[[...]]</code>) denote multi-line strings in Lua, which allow us to specify our query and variables in a readable way.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk.body ='{"query": "' .. string.gsub(query, '\n', '') .. '", "variables": {' .. string.gsub(variables, '\n', '') .. '} }'
</code></pre></div></div>
<p>This sets the request body - as you might expect. Because the request body must contain valid JSON, we remove any line breaks from our <code class="language-plaintext highlighter-rouge">query</code> and <code class="language-plaintext highlighter-rouge">variables</code> variables and use string interpolation (<code class="language-plaintext highlighter-rouge">..</code>) to insert them into the final payload.</p>
<p>And that’s it. Have fun performance testing your GraphQL APIs! :rocket:</p>Hans SchnedlitzFor performance testing wrk is one of my favorite tools. Whether you are trying to get a quick benchmark or building a performance test suite - it is fairly simple using wrk.CLI OAuth in Ruby2021-02-26T17:00:00+00:002021-02-26T17:00:00+00:00https://hansschnedlitz.com/2021/02/26/cli-oauth-in-ruby<p>Have you ever used a command-line application that triggered an <a href="https://oauth.net/2/">OAuth</a> authentication flow to log you in and wondered how that works? For example, <a href="https://cloud.google.com/sdk">Google Cloud SDK</a> does this, as does <a href="https://devcenter.heroku.com/articles/heroku-cli">Heroku CLI</a>.</p>
<p>I’ve always found that a pretty neat way to handle authorization on the command line because it feels so <em>effortless</em> to the user. You run a command, your browser opens, you log in like you would on a website, and <em>Bam!</em>, you’re logged in on the command line.</p>
<p>So how does that work?</p>
<p>To find out, we’ll build a simple Thor app that supports OAuth login with Google. If you don’t care about words and just want to see the code you can find it on <a href="https://github.com/hschne/googleme">GitHub</a>.</p>
<p class="notice--warning">This post assumes you are somewhat familiar with OAuth. Also, the demo app we are building here should be considered a proof-of-concept. There are lots of holes and rough edges that still need ironing out before it can be used productively.</p>
<h2 id="basics">Basics</h2>
<p>OAuth can be a bit complicated. I’m not going to get into any details - there are tons of articles explaining it much better than I could. If you need a refresher I’m sure you can find some detailed information on the web :wink:</p>
<p>For now, just consider that two things make OAuth for command-line applications interesting:</p>
<ol>
<li>You do not own a trusted domain. The component that is starting the OAuth flow is a command-line application. There simply is no webpage to redirect the authentication provider to.</li>
<li>The client itself is untrusted. You do not own the platform where the code initiating the OAuth flow is running. Similar to a mobile app, you must assume that you cannot keep secrets secret, and as such, your OAuth flow cannot use a client secret.</li>
</ol>
<p>The first issue we can solve by starting a local server that we can redirect to. So, <code class="language-plaintext highlighter-rouge">localhost</code> becomes our callback domain. When authorizing with Google, this is already accounted for when we create OAuth credentials for <em>Desktop</em> applications.</p>
<p>To solve the second issue we’ll use the <a href="https://tools.ietf.org/html/rfc7636">PKCE extension</a> for OAuth. This aspect of OAuth, and the security implications of not being able to keep the client secret a secret, is a bit complicated. <a href="https://developer.okta.com/docs/concepts/oauth-openid/#authorization-code-with-pkce-flow">This Okta post</a> does a good job of explaining why PKCE works as a solution.</p>
<h2 id="creating-the-oauth-client">Creating the OAuth Client</h2>
<p>Let’s start by creating a simple command-line application. Our app will only provide two commands: A <code class="language-plaintext highlighter-rouge">login</code> command, which triggers the OAuth flow, and a <code class="language-plaintext highlighter-rouge">user</code> command, which performs an authorized request to retrieve some information from the Google API.</p>
<p>We’ll use <a href="https://github.com/erikhuda/thor">Thor</a> to create the app:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Error</span> <span class="o"><</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Error</span><span class="p">;</span> <span class="k">end</span>
<span class="k">class</span> <span class="nc">Main</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="n">desc</span> <span class="s1">'login'</span><span class="p">,</span> <span class="s1">'Login with Google'</span>
<span class="k">def</span> <span class="nf">login</span>
<span class="c1"># TODO: Login code</span>
<span class="k">end</span>
<span class="n">desc</span> <span class="s1">'user'</span><span class="p">,</span> <span class="s1">'Retrieve user data'</span>
<span class="k">def</span> <span class="nf">user</span>
<span class="c1"># TODO: API Request</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Before we can implement the OAuth flow we need to create OAuth Client IDs in the Google Cloud Console. If you are starting with a new project, you must create a new <a href="https://console.cloud.google.com/apis/credentials/consent">consent screen</a> first.</p>
<p>Fill in the required information - you do not need to provide authorized domains or app domains. When selecting scopes we only need the <code class="language-plaintext highlighter-rouge">userinfo.profile</code> scope, as that is the only information we want access to.</p>
<p>Head over to <a href="https://console.cloud.google.com/apis/credentials">credentials</a> and create new <code class="language-plaintext highlighter-rouge">OAuth client ID</code> credentials. As application type select <code class="language-plaintext highlighter-rouge">Desktop app</code>. Take note of both client ID and secret, you’ll need them later</p>
<p class="notice--info">‘Didn’t you just say we can’t use client secrets on untrusted platforms?’ I hear you say. Well, yes, but it seems that Google is a bit, like, doing their own thing here. Even though the desktop client goes through a PKCE flow, it must <em>still</em> provide a client secret and that secret is essentially treated as public information. <a href="https://stackoverflow.com/a/61970107/2553104">This SO comment</a> sheds some light on this weird situation.</p>
<h2 id="implementing-the-oauth-flow">Implementing the OAuth Flow</h2>
<p>As mentioned previously, to receive callbacks from the authorization server, we need to start a local server to receive those callbacks. Let’s create it.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'socket'</span>
<span class="nb">require</span> <span class="s1">'uri'</span>
<span class="nb">require</span> <span class="s1">'cgi'</span>
<span class="k">module</span> <span class="nn">Goggleme</span>
<span class="k">class</span> <span class="nc">Server</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="vi">@state</span> <span class="o">=</span> <span class="n">state</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="n">server</span> <span class="o">=</span> <span class="no">TCPServer</span><span class="p">.</span><span class="nf">new</span> <span class="mi">9876</span>
<span class="k">while</span> <span class="n">connection</span> <span class="o">=</span> <span class="n">server</span><span class="p">.</span><span class="nf">accept</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">gets</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">handle</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">puts</span> <span class="s1">'OAuth request received. You can close this window now.'</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">close</span>
<span class="k">return</span> <span class="n">data</span> <span class="k">if</span> <span class="n">data</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">handle</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="n">_</span><span class="p">,</span> <span class="n">full_path</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
<span class="n">path</span> <span class="o">=</span> <span class="no">URI</span><span class="p">(</span><span class="n">full_path</span><span class="p">).</span><span class="nf">path</span>
<span class="n">handle_authorize</span><span class="p">(</span><span class="n">full_path</span><span class="p">)</span> <span class="k">if</span> <span class="n">path</span> <span class="o">==</span> <span class="s1">'/authorize'</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_authorize</span><span class="p">(</span><span class="n">full_path</span><span class="p">)</span>
<span class="n">params</span> <span class="o">=</span> <span class="no">CGI</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="no">URI</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">full_path</span><span class="p">).</span><span class="nf">query</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s1">'Invalid oauth request received'</span><span class="p">)</span> <span class="k">if</span> <span class="vi">@state</span> <span class="o">!=</span> <span class="n">params</span><span class="p">[</span><span class="s1">'state'</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">params</span><span class="p">[</span><span class="s1">'code'</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Executing this will start a server on port <code class="language-plaintext highlighter-rouge">9876</code> that listens for requests to the <code class="language-plaintext highlighter-rouge">/authorize</code> endpoint. Upon receiving such a request, we verify that it contains the correct parameters and return the authorization code.</p>
<p>After the local server is ready to receive requests we need to open the Browser to allow the user to login using the selected authentication provider - in our case Google. Because we use PKCE, there is a small twist. We need to create a <code class="language-plaintext highlighter-rouge">code_verifier</code> and a <code class="language-plaintext highlighter-rouge">code_challenge</code> additionally to the <code class="language-plaintext highlighter-rouge">state</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">state</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">base64</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="n">code_verifier</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">base64</span><span class="p">(</span><span class="mi">64</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'+/'</span><span class="p">,</span> <span class="s1">'-_'</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'='</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="n">code_challenge</span> <span class="o">=</span> <span class="no">Digest</span><span class="o">::</span><span class="no">SHA2</span><span class="p">.</span><span class="nf">base64digest</span><span class="p">(</span><span class="n">code_verifier</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'+/'</span><span class="p">,</span> <span class="s1">'-_'</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'='</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
</code></pre></div></div>
<p>We can then start the server in a background thread.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">server</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">report_on_exception</span> <span class="o">=</span> <span class="kp">false</span>
<span class="no">Server</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">state</span><span class="p">).</span><span class="nf">start</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We can use <code class="language-plaintext highlighter-rouge">state</code> and <code class="language-plaintext highlighter-rouge">code_challenge</code> to initialize the OAuth flow. Note that we are using the <code class="language-plaintext highlighter-rouge">code</code> response type and the <code class="language-plaintext highlighter-rouge">S256</code> code challenge method.</p>
<p>We’ll use <a href="https://github.com/copiousfreetime/launchy">Launchy</a> to open the browser window, and after that is done, we wait for the local server to receive the callback.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">response_type: </span><span class="s1">'code'</span><span class="p">,</span>
<span class="ss">code_challenge_method: </span><span class="s1">'S256'</span><span class="p">,</span>
<span class="ss">code_challenge: </span><span class="n">code_challenge</span><span class="p">,</span>
<span class="ss">client_id: </span><span class="s1">'591376582274-ctrjhsj8fjjhn4pk1rknfvcfhrcc3af7.apps.googleusercontent.com'</span><span class="p">,</span>
<span class="ss">redirect_uri: </span><span class="s1">'http://localhost:9876/authorize'</span><span class="p">,</span>
<span class="ss">scope: </span><span class="s1">'https://www.googleapis.com/auth/userinfo.profile'</span><span class="p">,</span>
<span class="ss">state: </span><span class="n">state</span><span class="p">,</span>
<span class="ss">access_type: </span><span class="s1">'offline'</span>
<span class="p">}.</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">=</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}.</span><span class="nf">reduce</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">&</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="no">Launchy</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"https://accounts.google.com/o/oauth2/v2/auth?</span><span class="si">#{</span><span class="n">params</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">exception</span><span class="o">|</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s2">"Attempted to open </span><span class="si">#{</span><span class="n">uri</span><span class="si">}</span><span class="s2"> and failed because </span><span class="si">#{</span><span class="n">exception</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">server</span><span class="p">.</span><span class="nf">join</span>
</code></pre></div></div>
<p>Once we have received the authorization code, we contact the authorization server to exchange it for an authorization token.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">code</span> <span class="o">=</span> <span class="n">server</span><span class="p">.</span><span class="nf">value</span>
<span class="n">uri</span> <span class="o">=</span> <span class="no">URI</span><span class="p">(</span><span class="s1">'https://oauth2.googleapis.com/token'</span><span class="p">)</span>
<span class="n">http</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">.</span><span class="nf">host</span><span class="p">,</span> <span class="n">uri</span><span class="p">.</span><span class="nf">port</span><span class="p">)</span>
<span class="n">http</span><span class="p">.</span><span class="nf">use_ssl</span> <span class="o">=</span> <span class="kp">true</span>
<span class="n">http</span><span class="p">.</span><span class="nf">verify_mode</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">SSL</span><span class="o">::</span><span class="no">VERIFY_NONE</span>
<span class="n">request</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Post</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">)</span>
<span class="n">request</span><span class="p">[</span><span class="s1">'content-type'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'application/x-www-form-urlencoded'</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">grant_type: </span><span class="s1">'authorization_code'</span><span class="p">,</span>
<span class="ss">code_verifier: </span><span class="n">code_verifier</span><span class="p">,</span>
<span class="ss">code: </span><span class="n">code</span><span class="p">,</span>
<span class="ss">client_id: </span><span class="s1">'591376582274-ctrjhsj8fjjhn4pk1rknfvcfhrcc3af7.apps.googleusercontent.com'</span><span class="p">,</span>
<span class="ss">client_secret: </span><span class="s1">'cZAXyEkeV9kZNmDQyZsNLHaj'</span><span class="p">,</span>
<span class="ss">redirect_uri: </span><span class="s1">'http://localhost:9876/authorize'</span>
<span class="p">}.</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">=</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}.</span><span class="nf">reduce</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">&</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="n">request</span><span class="p">.</span><span class="nf">body</span> <span class="o">=</span> <span class="n">params</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">http</span><span class="p">.</span><span class="nf">request</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s2">"Invalid token response, got </span><span class="si">#{</span><span class="n">response</span><span class="p">.</span><span class="nf">code</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="k">unless</span> <span class="n">response</span><span class="p">.</span><span class="nf">code</span> <span class="o">==</span> <span class="s1">'200'</span>
</code></pre></div></div>
<p>If all goes well we should receive an access token along with additional data - which we’ll ignore for now to keep things simple :grin:</p>
<p class="notice--info">As you probably know, authorization tokens issued via OAuth expire after some time. The lifetime of the authorization token is part of that ‘additional data’, and would normally be used to have the user reauthorize your application.</p>
<h2 id="performing-authorized-requests">Performing Authorized Requests</h2>
<p>Now we’ll use the token we just received in our <code class="language-plaintext highlighter-rouge">user</code> command. We simply dump it in a file at the end of the <code class="language-plaintext highlighter-rouge">login</code> command and retrieve it when we need it. This is not the <a href="https://medium.com/@calavera/stop-saving-credential-tokens-in-text-files-65e840a237bb">right way to store credentials</a> but it will do for now.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">data</span> <span class="o">=</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="nf">body</span><span class="p">)</span>
<span class="n">path</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="no">Dir</span><span class="p">.</span><span class="nf">home</span><span class="p">,</span> <span class="s1">'.googleme'</span><span class="p">)</span>
<span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">f</span><span class="o">|</span> <span class="n">f</span><span class="p">.</span><span class="nf">write</span> <span class="n">data</span><span class="p">.</span><span class="nf">to_json</span> <span class="p">}</span>
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">path</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="no">Dir</span><span class="p">.</span><span class="nf">home</span><span class="p">,</span> <span class="s1">'.googleme'</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s1">'No access token found, please login first'</span><span class="p">)</span> <span class="k">unless</span> <span class="no">File</span><span class="p">.</span><span class="nf">file?</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">path</span><span class="p">))</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s1">'No access token found, please login first'</span><span class="p">)</span> <span class="k">unless</span> <span class="n">data</span>
</code></pre></div></div>
<p>Now the only thing that remains is to retrieve user information:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">access_token</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s1">'access_token'</span><span class="p">]</span>
<span class="n">uri</span> <span class="o">=</span> <span class="no">URI</span><span class="p">(</span><span class="s1">'https://www.googleapis.com/oauth2/v1/userinfo?alt=json'</span><span class="p">)</span>
<span class="n">http</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">.</span><span class="nf">host</span><span class="p">,</span> <span class="n">uri</span><span class="p">.</span><span class="nf">port</span><span class="p">)</span>
<span class="n">http</span><span class="p">.</span><span class="nf">use_ssl</span> <span class="o">=</span> <span class="kp">true</span>
<span class="n">http</span><span class="p">.</span><span class="nf">verify_mode</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">SSL</span><span class="o">::</span><span class="no">VERIFY_NONE</span>
<span class="n">request</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Get</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">)</span>
<span class="n">request</span><span class="p">[</span><span class="s1">'Authorization'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"Bearer </span><span class="si">#{</span><span class="n">access_token</span><span class="si">}</span><span class="s2">"</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">http</span><span class="p">.</span><span class="nf">request</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s2">"Invalid token response, got </span><span class="si">#{</span><span class="n">response</span><span class="p">.</span><span class="nf">code</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="k">unless</span> <span class="n">response</span><span class="p">.</span><span class="nf">code</span> <span class="o">==</span> <span class="s1">'200'</span>
<span class="nb">puts</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="nf">body</span><span class="p">)</span>
</code></pre></div></div>
<p>And that’s it! Running this little demo should now give you the user data of the authorized user.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Login first</span>
<span class="nv">$ </span>googleme login
<span class="c"># Show me the profile info!</span>
<span class="nv">$ </span>googleme user
<span class="o">{</span>
<span class="s2">"id"</span> <span class="o">=></span> <span class="s2">"123455"</span>,
<span class="s2">"name"</span> <span class="o">=></span> <span class="s2">"Hans Schnedlitz"</span>,
<span class="s2">"given_name"</span> <span class="o">=></span> <span class="s2">"Hans"</span>,
<span class="s2">"family_name"</span> <span class="o">=></span> <span class="s2">"Schnedlitz"</span>,
<span class="s2">"picture"</span> <span class="o">=></span> <span class="s2">"https://lh3.googleusercontent.com/a-/xyz"</span>,
<span class="s2">"locale"</span> <span class="o">=></span> <span class="s2">"en"</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>This was a fun little exercise that needed way more research than I expected. I learned a thing or two about OAuth that I didn’t know before, and I hope you did too while reading this. As mentioned before, the implementation is a very rough prototype and there are a bunch of things that can be improved.</p>
<p>Taking care of token expiry and re-authentication for one. You also should not store credentials the way I did, but rather take advantage of secure vaults that your operating system provides. And last but not least, this prototype implementation’s error handling is practically non-existent, so that should probably be changed:sweat_smile:</p>
<p>That being said, I’m still pretty happy with the result and am looking forward to using this in the future.</p>Hans SchnedlitzHave you ever used a command-line application that triggered an OAuth authentication flow to log you in and wondered how that works? For example, Google Cloud SDK does this, as does Heroku CLI.Use GitHub Actions to find Outdated Dependencies2021-02-10T21:00:00+00:002021-02-10T21:00:00+00:00https://hansschnedlitz.com/2021/02/10/using-github-actions-to-detect-outdated-dependencies<p>Keeping up with dependencies can be a pain. That is especially true if you build a tool that heavily relies on some library. If that library changes in a major way, you’ll have to be quick with updating or risk issues piling up.</p>
<p>But how can you efficiently keep track of dependency updates?</p>
<h2 id="scheduled-github-actions">Scheduled GitHub Actions</h2>
<p>I’ve found GitHub Actions to be a simple, yet effective, solution for that particular problem.</p>
<p>A small workflow that runs on a schedule and checks if there are outdated dependencies does the job. I use something like this for <a href="https://github.com/hschne/reveal.js-starter">reveal.js-starter</a> to get notified when a new <a href="https://revealjs.com/">reveal.js</a> version is released:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Check Outdated</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">schedule</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">cron</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0</span><span class="nv"> </span><span class="s">12</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">1"</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">check_updates</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout repository</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install npm</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm install</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check outdated</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm outdated reveal.js</span>
</code></pre></div></div>
<p>If you are thinking “Well, that doesn’t look hard” - you are right. And isn’t that nice? :wink:</p>
<p>There isn’t a lot for me to cover here, but let’s go over the interesting parts real quick.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on</span><span class="pi">:</span>
<span class="na">schedule</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">cron</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0</span><span class="nv"> </span><span class="s">12</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">1"</span>
</code></pre></div></div>
<p>Here we specify when our action should run. Consult the <a href="https://docs.github.com/en/actions">official documentation</a> for details, but I’ve found checking dependencies each Monday at noon to work quite nicely.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check outdated</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm outdated reveal.js</span>
</code></pre></div></div>
<p>Luckily, <code class="language-plaintext highlighter-rouge">npm</code> offers a simple command to check if any dependency is outdated (based on your current lock file). The nice thing here is that this command will return with exit code <code class="language-plaintext highlighter-rouge">1</code> if newer versions were detected. As a result, the GitHub Action will fail without us needing to do anything else. Easy!</p>
<p><img src="https://i.kym-cdn.com/photos/images/original/000/875/422/11f.gif" alt="Chefs Kiss" class="align-center" /></p>
<p>The same approach works quite well for Ruby-based applications since <a href="https://bundler.io/">bundler</a> offers an <code class="language-plaintext highlighter-rouge">outdated</code> command as well. I use this workflow for my custom Rails generator <a href="https://github.com/hschne/schienenzeppelin">Schienenzeppelin</a>:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Check Outdated</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">schedule</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">cron</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0</span><span class="nv"> </span><span class="s">12</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">1"</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">check_updates</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout repository</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@latest</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ruby-version</span><span class="pi">:</span> <span class="s">3.0.0</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check Outdated</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">bundle config unset deployment</span>
<span class="s">bundle outdated rails</span>
</code></pre></div></div>
<p>Now, I’ve kept these actions very simple. Of course, there are tons of things you can improve upon! Customizing how notifications are sent or only failing when a new major version is released are things that come to mind.</p>
<p>What do you think? Let me know if you’ve encountered other interesting uses for GitHub actions :hugs:</p>Hans SchnedlitzKeeping up with dependencies can be a pain. That is especially true if you build a tool that heavily relies on some library. If that library changes in a major way, you’ll have to be quick with updating or risk issues piling up.