In this episode, Vincent shares insights into his latest open-source venture, the MCP Toolkit, which aims to provide tools for writing with Model Context Protocol (MCP) applications using Clojure. Alongside Vincent's personal narrative, we delve into the technical aspects of his projects and discuss the challenges and opportunities in the open-source world—especially the importance of community collaboration and user contributions. Vincent also explores the intersection of AI and programming, discussing his experiments and learnings about AI-driven development.

His insights into using AI for code generation and tooling provide a fresh perspective on modern development practices. Throughout the discussion, Vincent provides a candid look at his professional journey, including his decision to move from Taiwan to Finland and the lifestyle and cultural differences encountered along the way. His story is one of perseverance and adaptability, offering valuable lessons for developers navigating career transitions and exploring the vast potential of open-source collaboration.

Whether you're interested in Clojure, open-source development, or the practical applications of AI in programming, this episode provides a wealth of knowledge and inspiration. Don't forget to subscribe for more insightful conversations.

Vincent:

• Github: https://github.com/green-coder

• Blog: https://blog.404.taipei/

Open Source Projets:

• MCP-toolkit: https://github.com/metosin/mcp-toolkit

• Vrac: https://github.com/metosin/vrac

• Signaali: https://github.com/metosin/signaali

• Siagent: https://github.com/metosin/siagent

• Si-Frame: https://github.com/metosin/si-frame

About Our Guest:

Miikka Koskinen, based in Helsinki, is a seasoned software engineer with over 15 years of experience spanning mathematics, technology, and open source development. Miikka's journey began with QBasic at age 10 and has evolved through a variety of languages and domains—from embedded systems to large-scale consulting—and contributions to the Clojure ecosystem at Metosin.

Episode Highlights:

• Code Review Culture: Learn how Miikka approaches code review as a process of collaboration and mutual learning, not just gatekeeping. We discuss the value of early feedback, building shared understanding, and fostering psychological safety within engineering teams.

• Teamwork & Trust: Miikka shares insights on establishing trust in remote and distributed teams, the dangers of defaulting to blame, and how shared ownership of code leads to healthier team dynamics.

• Career Path: Generalist vs. Specialist: Reflect with us on the pros and cons of being a generalist versus a specialist in the software industry. Miikka offers candid thoughts about career development, lifelong learning, and finding your niche (or not!).

• Technical Writing & Thought Leadership: Discover the benefits of maintaining a technical blog and how publishing your thoughts—whether on engineering practices or database migrations—can enhance your professional reputation and open new career opportunities.

• Open Source & Community: We discuss Miikka’s journey with open source, contributing and maintaining libraries, and the dynamics of knowledge sharing in public and private codebases.

• Emerging Tech: Get Miikka's take on S3 object storage’s new capabilities and potential impacts on distributed system design and next-generation database architectures.

Why Watch?

Whether you're interested in evolving your team's code review practices, looking to strengthen collaboration and trust, or evaluating your own growth as a software professional, this episode offers practical strategies and honest reflections.

The conversation weaves through his experience moving to Finland in 2017, finding love on Tinder, and building a new life in Helsinki. Daniel offers valuable insights into design systems, explaining how they serve as the language of digital products and why timing is crucial when implementing them. He emphasizes the importance of human relationships and trust in successful design system adoption.

Beyond tech, Daniel opens up about his transition from competitive basketball to more "proper 38-year-old sports" like squash, his enduring love for metal music, and how Finland's legendary Tuska festival coinciding with his birthday weekend feels like destiny. His story is a compelling blend of professional expertise, personal growth, and finding belonging in a new culture.

This episode offers an authentic look at expatriate life in Finland, the evolution of creative passions, and the intersection of technology and human connection in modern digital product development.

In this episode, we sit down with Valtteri Harmainen, the CEO of Metosin, to explore his unconventional journey into the world of technology and leadership. You might remember Valtteri from his talk at ClojuTRE, where he shared how he "created his own Clojure job". We delve into his early life in Finland, his accidental entry into programming, and how he ultimately landed in the CEO role at Metosin

This isn't just a tech story; it's a personal one. Valtteri recounts how a functional programming course that never happened sparked his interest in Clojure, and how his initial programming project involved fixing a university system that "doesn't work". His path wasn't straightforward, going from philosophy and cognitive science to usability and eventually to programming by necessity. This led to a career that saw him move from developer roles to leadership positions, a pattern he says he kept experiencing: "I ended up in some kind of leadership position"

We discuss his experiences at various companies, including his time at a startup that grew rapidly during the pandemic. Valtteri also shares his transition to Metosin, where he pioneered the account manager role and then stepped into the CEO position. His view on leadership has changed, realizing that "people are not as easy as writing code". He values honesty and transparency in leadership acknowledging that "everything is a compromise" when making decisions

Looking ahead, Valtteri shares Metosin's vision for the future, emphasizing a holistic approach that includes not just technical skills, but also business acumen and people skills, which are described as a "triangle". He explains that while they are strong in Clojure, the real problems companies face are more often related to people, processes, and organization rather than technology. This approach enables Metosin to work with clients on all levels. Valtteri also shares a few laughs about the unexpected challenges in communication, mentioning how "people are the weirdest part" of the entire process.

This episode is a deep dive into the career path of an unconventional leader in tech and the evolution of a software company.

Key Topics:

• Valtteri's journey from philosophy to tech

• The accidental start of a programming career

• Learning functional programming and Clojure

• Transition from programmer to leader

• The importance of people skills in business and tech

• Metosin's future and its focus on a holistic approach

• The challenges of leadership and the value of transparency

This episode provides a comprehensive overview of Valtteri's career and Metosin's approach to software and consulting, making it a must-listen for anyone interested in technology, leadership, or the human side of business.

Please bear with us. The episode was recorded in a bustling food court, so there might be some background noise. However, the conversation is packed with insights and ideas that are sure to interest you if you're into web development or Clojure.

Topics Covered:

• Enhancing the Clojure web stack

• Middleware ordering challenges and solutions

• Ben's career journey from Intel and Apple to Clojure

• Discussion on Rama and its applications

• Performance optimizations in Clojure

• The future of Clojure web development

Show Highlights:

• Innovative ideas for middleware management

• Insights into Ben's professional background

• Practical tips on improving web development workflows

• Exploration of performance optimization in Clojure

Stay tuned for our next episode featuring the CEO of Metosin, Valtteri Harmainen. Don't forget to subscribe and hit the notification bell to stay updated with our latest content.

If you enjoyed this episode, please like, share, and subscribe. Your support helps us bring more insightful conversations to the community.

Find Q4US online: https://q4us.dev

Connect with Sandun on LinkedIn https://www.linkedin.com/in/sandundasanayake/

At that point, I was using a smaller hosting service to manage my DNS, but they didn't have the option to redirect the subdomain to a different domain.

I ended up switching to Cloudflare and using their "rules" to route from the old domain to the new one. From blog.tvaisanen.com to tonitalksdev.com.

The process was pain-free, and now Cloudflare has become my go-to domain registrar and DNS management tool. Let me show you how simple my configuration ended up being.

Redirect Rule

All it took was to add one redirect rule for the subdomain.

To reroute all the traffic to the new domain.

Conclusion

Cloudflare's free plan includes tools for custom redirect rules and domain DNS management. You can use your current domain registrar and move the DNS management to Cloudflare, but they also provide domain registration services at a cost. I'm not sponsored by them; I just like the service, and I've transferred all my domains to their platform.

Learn how to add your domain to Cloudflare from their documentation.

Once again, thanks for reading.

Feel free to reach out and let me know what you think—social links in the menu.

Cat Rivers shares her journey from tattooing to software development and back, discussing burnout, coding experiences, and the challenges of career changes. Highlights

• 🎨 Cat Rivers is a tattoo artist and psychologist who briefly ventured into coding.

• 💻 She experienced burnout while juggling entrepreneurship and sought stability in tech.

• 🔄 Coding was seen as a potential career change but led to emotional challenges.

• 📚 She learned JavaScript, TypeScript, and Clojure, noting their complexities.

• 🤝 Networking helped her land an internship and eventually a job in software development.

• 🧠 Imposter syndrome and communication barriers are common in tech environments.

• 🦆 Cat returned to tattooing, emphasizing the need for creative fulfillment.

Spotify version will be published on the 22. of August.

If you are familiar with Aero and Malli jump till the end to see how to write configurations as Malli schemas.

Let's go through the basics of Aero and Malli first before putting them together.

Aero

Aero is a popular Clojure library from JUXT that enables us to write the environment variables into the EDN configuration file with #ENV tag literals. It has other features, but for our purposes here, this is enough. Let's see what this means by creating a configuration file for an application that needs to know the user and the shell of the system the application runs on.

Without Aero, we might write something like

(System/getenv "USER") ;; => "tvaisanen"
(System/getenv "SHELL");; => "/bin/zsh"

But we prefer the declarative way

;; sample-01.edn
{:user #env USER
 :shell #env SHELL}

(ns core
  (:require [aero.core :as aero]
            [clojure.java.io :as io]))

(aero/read-config (io/resource "sample-01.edn"))
;; => 
{:user "tvaisanen" :shell "/bin/zsh"}

Now that the application can access the configuration data, we'd usually do the validation step.

Let's say that we expect the username to be at least five characters and the shell to be bash instead of zsh and we want to throw an error when the requirements are not met.

(defn validate-config [config]
  (when-not (and (count (get config :user))
                 (clojure.string/ends-with? (get config :shell)
                                            "bash"))
    (throw (ex-info "invalid config" config))))

(-> (io/resource "sample-01.edn")
    (aero/read-config)
    (validate-config))

  Show: Project-Only All 
  Hide: Clojure Java REPL Tooling Duplicates  (26 frames hidden)

1. Unhandled clojure.lang.ExceptionInfo
   invalid config
   {:user "tvaisanen", :shell "/bin/zsh"}
                      REPL:   47  core/validate-config
                      REPL:   43  core/validate-config
                      REPL:   51  core/eval17603
                      REPL:   49  core/eval17603

As a developer, I'm already thinking that this approach will be a pain. I'll probably need to add many different configuration parameters, validate for all of them, and then provide descriptive error messages. It'd still be manageable for a small toy application like ours, but it's most likely not the case for larger projects.

Validation With Malli

Malli to the rescue.

If you're unfamiliar with Malli, refer to my earlier post or the official documentation for the basics.

Let's start by defining the expected values and their types. At this point, we'll assume both values as strings and attack the constraints in a moment.

(require '[malli.core :as m])

(def schema
  [:map
   [:user :string]
   [:shell :string]])

(->> (io/resource "sample-01.edn")
     (aero/read-config)
     (m/validate schema))
;; => 
true

Everything is OK so far. Next, add the constraints from before.

(def schema
  [:map
   [:user {:min 5} :string]
   [:shell [:re "^.*bash$"]]])

(->> (io/resource "sample-01.edn")
     (aero/read-config)
     (m/validate schema))
;; => 
false

Now, we are at the same stage with our validate-config function. Let's take it a step further. Do you remember we talked about the descriptive error messages? Malli provides us with out-of-the-box tools to make this easy for us.

(->> (io/resource "sample-01.edn")
     (aero/read-config)
     (m/explain schema))
;; =>
{:schema
 [:map 
   [:user {:doc "At least 5 characters long", :min 5} :string] 
   [:shell {:doc "String needs to end with `bash`"} [:re "^.*bash$"]]],
 :value {:user "tvaisanen", :shell "/bin/zsh"},
 :errors
 ({:path [:shell], 
   :in [:shell], 
   :schema [:re "^.*bash$"], 
   :value "/bin/zsh"})}

The output is still somewhat cryptic for the casual reader. We are not satisfied until the resulting error reads like English.

(require '[malli.error :as me])

(->> (io/resource "sample-01.edn")
     (aero/read-config)
     (m/explain schema)
     (me/humanize))
;; => 
{:shell ["should match regex"]}

Now we have the error in English, but we still don't know what the regex part contains.

(def schema
  [:map
   [:user
    {:min 5}
    :string]
   [:shell
    [:re
     {:error/message "String needs to end with `bash`"}
     "^.*bash$"]]])

(->> (io/resource "sample-01.edn")
     (aero/read-config)
     (m/explain schema)
     (me/humanize))
;; 
=> {:shell ["String needs to end with `bash`"]}

One last thing before we move forward.

Usually, we need to read some values as specific types.

Assume that we expect a comma-separated string for the features application and want to use it as a set of strings. Again, we can use Malli decode for this at load time.

Let's see how decoding works first.

(require '[malli.transformer :as mt])

(m/decode [:map [:int int?]]
          {:int "1"}
          mt/string-transformer)
;; 
=> {:int 1}

Then, we adapt the example to our use case.

(m/decode [:map [:features [:set :string]]]
          {:features "foo,bar,fizz"}
          mt/string-transformer)
;; 
=> {:features "foo,bar,fizz"}

It didn't work as expected! But of course, how could Malli know what string encoding we use? We can fix this with custom decoders.

(m/decode 
  [:map [:features
          {:decode/string (fn [v] 
                            (set (clojure.string/split v #",")))}
          [:set :string]]]
          {:features "foo,bar,fizz"}
          mt/string-transformer)
;; 
=> {:features #{"foo" "bar" "fizz"}}

Let's wrap all of this in a function to load our configuration. I also updated the expected shell to be zsh so the validation step won't throw.

(defn load-config [filename]
  (let [config (aero/read-config (io/resource filename))
        decoded (m/decode schema config mt/string-transformer)]
    (when-not (m/validate schema decoded)
      (throw (ex-info "invalid schema"
                      (->> decoded
                           (m/explain schema)
                           (me/humanize)))))
    decoded))

;; let's see what happens
(load-config "sample-01.edn")

1. Unhandled clojure.lang.ExceptionInfo
   invalid schema
   {:features ["missing required key"]}
                      REPL:  132  core/load-config
                      REPL:  128  core/load-config
                      REPL:  138  core/eval17775

The error shows that we are missing the :features key. Update the configuration file to fix this.

{:user #env USER
 :shell #env SHELL
 :features "foo,bar,fizz"}

And try again.

(load-config "sample-01.edn")
;; =>
{:user "tvaisanen"
 :shell "/bin/zsh"
 :features #{"foo" "bar" "fizz"}}

It works as expected! We save a lot of work when dealing with configuration updates. In the future, we'll keep the schema updated with the configuration file itself. Simple.

Keep Schema Up-To-Date

Keeping the config and schema up to date is easier said than done. We are people, after all. If we want to ensure that the schema is updated when the configuration changes, we most likely need to add a step to remove all undefined keys from the loaded configuration to force each developer to update the schema on changes.

We can do this by adding a mt/strip-extra-keys-transformer.

(def transformer
  (mt/transformer mt/strip-extra-keys-transformer
                  mt/string-transformer))

(defn load-config [filename]
  (let [config (aero/read-config (io/resource filename))
        decoded (m/decode schema config transformer)]
    (when-not (m/validate schema decoded)
      (throw (ex-info "invalid schema"
                      (->> decoded
                           (m/explain schema)
                           (me/humanize)))))
    decoded))

Now, we force developers to add their changes to the schema configuration since if they don't, the strip-extra-keys-transformer drops the undefined keys, and it'll throw if an update doesn't match the schema.

But why stop here?

Configuration File as Malli Schema

We can define Malli schemas as EDN files.

Aero reads EDN files.

We can write our configurations and type definitions in the same file. As long as we reference the decoder functions in the EDN, if we have any, we should be good to go. Let's update our config file.

;; sample-02.edn
[:map
 [:user
  {:error/message "At least 5 characters long"
   :default #env USER
   :min 5}
  :string]
 [:shell
  [:re
   {:error/message "String needs to end with `zsh`"
    :default #env SHELL}
   "^.*zsh$"]]
 [:features
  {:default "foo,bar,bizz"}
  [:set :string]]]

Now, we have the schema with default values.

(aero/read-config (io/resource "sample-02.edn"))
;; => 
[:map
 [:user
  {:error/message "At least 5 characters long"
   :default "tvaisanen"
   :min 5}
  :string]
 [:shell
  [:re
   {:error/message "String needs to end with `zsh`"
    :default "/bin/zsh"}
   "^.*zsh$"]]
 [:features {:default "foo,bar,bizz"} [:set :string]]]

The last remaining step is to transform this into a configuration map. We'll do this by creating a transformer using the default-value-transformer .

(def defaults-transformer
  (mt/default-value-transformer
   {:key :value
    :defaults {:map    (constantly {})
               :string (constantly "")
               :vector (constantly [])}}))

We must define defaults for maps to populate nested maps in the configuration. Depending on your case, doing the same for vectors or even strings might be necessary.

(m/decode [:map
           [:int {:value 1} :int]
           [:map :map]
           [:nested-map [:map [:a :string]]]
           [:vector [:vector :any]]
           [:string :string]]
          nil
          defaults-transformer)
;; =>
{:int 1
 :map {}
 :nested-map {:a ""}
 :vector []
 :string ""}

Then, we use this to load our configuration schema with the defaults.

(defn read-config
  ([filename]
   (read-config filename {:transformer transformer}))
  ([filename {:keys [transformer]}]
   (let [schema-data (aero/read-config (io/resource filename))
         schema (try (m/schema schema-data)
                     (catch Exception e
                       (throw (ex-info "Invalid configuration schema"
                                       {:malli/error (ex-data e)}))))
         config (m/decode schema nil defaults-transformer)]
     (when-not (m/validate schema config)
       (throw (ex-info "Invalid configuration"
                       {:value config
                        :error  (me/humanize (m/explain schema config))})))
     config)))

The last problem we need to solve is including the decoder functions.

1. Unhandled clojure.lang.ExceptionInfo
   Invalid configuration
   {:value {:user "tvaisanen", :shell "/bin/zsh", :features "foo,bar,bizz"},
    :error {:features ["invalid type"]}}
                      REPL:  158  core/read-config
                      REPL:  148  core/read-config
                      REPL:  167  core/eval17934

Extend Aero Readers

Let's create our reader #decoder ... to reference the custom decoder functions.

(require '[aero.core :as aero]

(defmethod aero/reader 'decoder
  [opts tag symbol-pointing-to-a-fn]
  (if-let [decoder-fn (resolve symbol-pointing-to-a-fn)]
    decoder-fn
    (throw (ex-info "Can't load decoder function"
                    {:fn symbol-pointing-to-a-fn}))))

Now, we can add #decoder config/decode-set-of-strings to the schema file.

[:map
 [:user
  {:doc "At least 5 characters long"
   :value #env USER
   :min 5}
  :string]
 [:shell
  [:re
   {:error/message "String needs to end with `zsh`"
    :value #env SHELL}
   "^.*zsh$"]]
 [:features
  {:value "foo,bar,bizz"
   :decode/string #decoder config/decode-set-of-strings}
  [:set :string]]]

And that's it! We can now load the configuration from the schema file by decoding the values.

(read-config "sample-02.edn")
;; => 
{:user "tvaisanen"
 :shell "/bin/zsh"
 :features #{"foo" "bar" "bizz"}}

The Final Result

Here's everything put together.

(ns config
  (:require [aero.core :as aero]
            [malli.core :as m]
            [malli.transform :as mt]
            [malli.error :as me]
            [clojure.java.io :as io]))

(def defaults-transformer
  (mt/transformer
   (mt/default-value-transformer
    ;; look default value under the key `value`
    {:key :value
     :defaults {:map    (constantly {})
                :string (constantly "")
                :vector (constantly [])}})
   ;; use string-transformer to "enable" string decoders
   (mt/string-transformer)))

(defn decode-set-of-strings [string-value]
  (set (clojure.string/split string-value #",")))

(defmethod aero/reader 'decoder
  [opts tag symbol-pointing-to-a-fn]
  (if-let [decoder-fn (resolve symbol-pointing-to-a-fn)]
    decoder-fn
    (throw (ex-info "Can't load decoder function"
                    {:fn symbol-pointing-to-a-fn}))))

(defn read-config
  ([filename]
   (read-config filename {:transformer defaults-transformer}))
  ([filename {:keys [transformer]}]
   (let [schema-data (eval (aero/read-config (io/resource filename)))
         schema (try (m/schema schema-data)
                     (catch Exception e
                       (throw (ex-info "Invalid configuration schema"
                                       {:malli/error (ex-data e)}))))
         config (m/decode schema nil transformer)]
     (when-not (m/validate schema config)
       (throw (ex-info "Invalid configuration"
                       {:value config
                        :error  (me/humanize (m/explain schema config))})))
     config)))

(read-config "sample-02.edn")
;; => 
{:user "tvaisanen" 
 :shell "/bin/zsh" 
 :features #{"foo" "bar" "bizz"}}

Conclusions

The benefits of having the validation and the configuration in the same file are that everything is in the same place, including the definition and the validation. It also makes it easier to maintain consistent validation practices since Malli takes care of the validation process. But all of this can make the configurations more difficult to read, which might be a problem, especially if the files need to be understood by people who don't know Clojure, for example, in DevOps.

If your project is already all in on Malli, this makes sense. The developers should already be familiar with the schema syntax, so there shouldn't be too much adoption friction.

I've never used this approach in production, but I'll try it out next time I need to configure a new project.

Once again, thanks for reading.

Feel free to reach out and let me know what you think—social links in the menu.

My friend and colleague from Metosin, Martin Varela, was happy to jump in front of the camera to capture the following conversation without hesitation.

I hope you found this interesting, and thank you for reading, watching, and listening.

Feel free to reach out and let me know what you think—social links in the menu.

Listen on Spotify

Podcast: YouTube / Spotify

This post is a follow-up to an earlier post where we set up a project for testing. Follow up on the project setup from here if you haven't already done so. Once again, this post is not about what you should test but to show you the bare minimum of creating mocks and fixtures to get you started.

We'll cover some simple examples to get you started with-redefs to mock functions and variables and use-fixtures to configure test fixtures. Let's start with the first one.

Mocking Functions and Variables

Clojure has a core feature macro with-redefs that can temporarily override a var (function or variable). Within the with-redefs closure, the program will interpret the var as we've defined locally, and this is useful when mocking some functions or variables in tests.

(with-redefs [var-to-override mock]
    ;; var-to-override is replaced with the mock
    )
;; var-to-override is interpreted normally again

Let's see how to use this in a test setup by mocking the multiply function from the previous post as an example. Let's return the operation as a string instead of computing the value.

(require '[clojure.test :as t])

(defn multiply-mock [a b]
  (str a " * "  b " = ?"))

(t/deftest using-with-redefs
  (t/testing "we can override multiply implementation"
    (with-redefs [sut/multiply multiply-mock]
      (t/is (= "2 * 2 = ?" (sut/multiply 2 2))))))

The same approach works with other namespaces, not just with functions we have defined. For example, we could temporarily change the implementation of clojure.edn/read-string.

(defn read-string-mock []
  "are you trying to read EDN?")

(t/deftest mock-edn
  (t/testing "we can override multiply implementation"
    (with-redefs [clojure.edn/read-string read-string-mock]
      (t/is (= {:a 1} (clojure.edn/read-string "{:a 1}"))))))

Fail in mock-edn
we can override multiply implementation

expected: {:a 1}
  actual: "are you trying to read EDN?"

In a real-world scenario, we might want to replace a function in a test that calls to an external service that we don't have a test environment or a process that takes time to complete. And that's pretty much what we need to know to get started with mocking using with-redefs. Let's take a look at setting up fixtures next.

Test Fixtures

What are text fixtures? Wikipedia defines them as follows.

A test fixture is a device used to consistently test some item, device, or piece of software. Test fixtures are used in the testing of electronics, software and physical devices.

In software, this usually means setting up an API, a database, or both so that the tests have a consistent environment in which to run.

Fixtures allow you to run code before and after tests, to set up the context in which tests should be run.

ClojureDocs

In Clojure, a fixture is just a function that we "connect" to the test execution with clojure.test/use-fixtures macro. We can wrap setup and teardown logic in this fixture function.

(defn logger-fixture [test-fn]
  (prn "start test")
  (test-fn)
  (prn "end test"))

; run once per namespace
(t/use-fixtures :once logger-fixture)
; run once pre deftest, with-test
(t/use-fixtures :each logger-fixture)

Let's see how this works with an HTTP API.

We'll continue the setup from the last post and create a new namespace api for the API code to start testing with the tools we just learned.

(ns api
  (:require [ring.adapter.jetty :as jetty]))

(def api-state (atom {}))

(defn read-database []
  @api-state)

(defn ring-handler [{:keys [method] :as req}]
  {:status  200
   :headers {"Content-Type" "application/edn"}
   :body    (str (read-database))})

(defn start! [opts]
  (jetty/run-jetty #'ring-handler opts))

To test the API over HTTP, we need an HTTP client. Add Hato to your deps.edn as the client, we are ready to start testing. Let's write the first test without setting up a fixture.

(ns api-test
  (:require [clojure.test :as t]
            [api :as sut]
            [hato.client :as http]))

(t/deftest test-api
  (t/testing "API works"
    (t/is (= 200 (:status (http/get "localhost:8080"))))))

If we run the test before setting up the fixture, we should get an error since we are trying to call an HTTP server that is not running.

Error in test-api
API works

expected: (= 200 (:status (http/get "http://localhost:8080")))
   error: java.net.ConnectException

To fix the problem, let's create a fixture around the tests that ensures the server is running and ready to serve requests when we run our tests.

(ns api-test
  (:require [clojure.test :as t]
            [api :as sut]
            [hato.client :as http]))

(defn api-fixture [test-fn]
  (let [;; store the handle to the server to be able
        ;; to stop the server after the test is run
        server (atom nil)]
    ;; setup test fixture by starting the server
    (reset! server (sut/start! {:port 8080 :join? false}))

    ;; run the test 
    (test-fn)

    ;; teardown test fixture by stopping the server
    (.stop @server)))

(t/use-fixtures :once api-fixture)

(t/deftest test-api
  (t/testing "API works"
    (t/is (= 200 (:status (http/get "localhost:8080"))))))

And now we should have a passing test!

Tested 1 namespaces in 61 ms
Ran 1 assertions, in 1 test functions
1 passed
cider-test-fail-fast: t

And that's it. Now, we have a working API fixture. The same logic applies to databases and other services.

Combine Both Mocks and Fixtures

We can mix and match these tools to meet our needs. Let's continue on the API tests by validating that the API returns the expected value, which is an empty map.

(t/deftest test-api
  (t/testing "API works"
    (t/is (= 200 (:status (http/get "http://localhost:8080")))))

  (t/testing "API state is initalized as an empty map"
    (let [response (http/get "http://localhost:8080"
                             {:as :clojure})]
      (t/is (= {} (get response :body ))))))

Tested 1 namespaces in 13 ms
Ran 2 assertions, in 1 test functions
2 passed
cider-test-fail-fast: t

Let's say we wanted to replace the "database state." We could mock the value by using with-redefs by referring to it with sut/api-state.

(t/testing "API response is altered by with-redefs"
    (with-redefs [sut/api-state (atom {:new "state"})]
      (let [response (http/get "http://localhost:8080"
                               {:as :clojure})]
        (t/is (= {} (get response :body))))))

This time, running the test, we can see that the returning value is indeed the one defined in the with-redefs closure.

api-test
1 non-passing tests:

Fail in test-api
API response is altered by with-redefs

expected: {}
  actual: {:new "state"}          

    diff: - nil          
          + {:new "state"}

Or, in case we wanted to mock the function reading the database, we could do the same for the read-database function.

(t/testing "read-database handler is mocked by with-redefs"
    (with-redefs [sut/read-database (constantly {:mocked "value"})]
      (let [response (http/get api-url {:as :clojure})]
        (t/is (= {} (get response :body ))))))

This time, we've successfully mocked the function that returns the state.

Fail in test-api
read-database handler is mocked by with-redefs

expected: {}            
  actual: {:mocked "value"}          

    diff: - nil          
          + {:mocked "value"}

Conclusion

Testing in Clojure is simple and relatively painless. This is because the language provides the essential tools without the need to write boilerplate code to set up mocks and fixtures. The examples here are just toy examples. Still, they show how these tools work and how to get started.

I hope you found this helpful, and thank you for reading.

Feel free to reach out and let me know what you think—social links in the menu.

You can find the previous steps from the earlier posts in the Building on DigitalOcean App Platform series.

Refactoring the API

First, let's update the source file structure from this

❯ tree src
src
└── main.clj

to this.

❯ tree src
src
└── clj
   └── api
      ├── db.clj
      └── main.clj

We'll add a new path src/cljs for the frontend-related code in a moment.

Let's continue where we left off last time after we created the migrations and moved the database-related code from the original namespace main to api.db.

(ns api.db
  (:require [migratus.core :as migratus]
            [next.jdbc :as jdbc]
            [next.jdbc.date-time]))

;; Use hardcoded default for now.
;; Configuration management needs to dealt with later.
(def dev-jdbc-url
  "jdbc:postgresql://localhost:5432/db?user=user&password=password")

(defn get-db-conf []
  {:dbtype  "postgres"
   :jdbcUrl (or (System/getenv "JDBC_DATABASE_URL")
                dev-jdbc-url)})

(defn migrations-config []
  {:db                   (get-db-conf)
   :store                :database
   :migration-dir        "migrations/"
   :migration-table-name "migrations"
   :init-in-transaction? false})

(defn create-migration [{:keys [name]}]
  (migratus.core/create (migrations-config)
                        name))

(defn datasource []
  (jdbc/get-datasource (get-db-conf)))

(defn get-migrations []
  (jdbc/execute! (datasource) ["SELECT * FROM migrations"]))

(defn run-migrations []
  (migratus/migrate (db/migrations-config)))

And then use the api.db in api.main and migrate the rest of the code.

(ns api.main
  (:require [ring.adapter.jetty :as jetty]
            [reitit.ring :as ring]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [api.db :as db])
  (:gen-class))

(defn get-port []
  (Integer/parseInt (or (System/getenv "PORT")
                        "8000")))

(defn app [_request]
  (let [migrations (db/get-migrations)]
    {:status  200
     :headers {"Content-Type" "application/edn"}
     :body    (str migrations)}))

(defn -main [& _args]
  (db/run-migrations)
  (jetty/run-jetty #'app {:port (get-port)}))

One last thing: update the paths in deps.edn and the :exec-fn for run and create-migration.

modified   api/deps.edn
@@ -1,4 +1,4 @@
-{:paths ["src" "resources"]
+{:paths ["src/clj" "resources" "compiled-resources"]

  :deps {org.clojure/clojure {:mvn/version "1.11.0"}
         ring/ring-core {:mvn/version "1.6.3"}
@@ -10,7 +10,7 @@

  :aliases {:run
            {:main-opts ["-m" "main"]
-            :exec-fn   main/-main}
+            :exec-fn   api.main/-main}
@@ -24,4 +25,4 @@

            :create-migration
            {:exec-args {:name nil}
-            :exec-fn db/create-migration}}}
+            :exec-fn api.db/create-migration}}}

Now, we are good to continue on the CLJS side of things.

Configure Shadow CLJS

We'll be using Shadow CLJS to compile the ClojureScript.

shadow-cljs provides everything you need to compile your ClojureScript projects with a focus on simplicity and ease of use. The provided build targets abstract away most of the manual configuration so that you only have to configure the essentials for your build. Each target provides optimal defaults for each environment and get an optimized experience during development and in release builds.

Shadow-CLJS: UserGuide

Shadow CLJS has a built-in development time HTTP server to serve the files, but we won't use it this time since we want to serve the files from our backend to keep the development setup as close as possible to the deployment configuration. But before getting there, we must first configure the build process, which has three steps:

1. Configure Shadow CLJS

2. Define NPM dependencies

3. Write ClojureScript code

1. Configure Shadow CLJS

Shadow CLJS is configured with shadow-cljs.edn file. It contains all of the ClojureScript-related configuration, dependencies, source file paths, build configurations, etc. I have added the cider/cider-nrepl and binaryage/devtools for convenience, but they are not required for the build.

The configuration is minimal, with only the browser build target defined. This means that when we run shadow-cljs compile/release app we'll get a Javascript bundle meant to be used in a browser.

{:source-paths ["src/cljs"]

 :dependencies [[binaryage/devtools "0.9.7"]
                [cider/cider-nrepl "0.36.0"]
                ;; React wrapper
                [lilactown/helix "0.1.5"]]

 :builds {:app {:target     :browser
                :output-dir "compiled-resources/public/js"
                :asset-path "/js"
                :modules    {:main {:entries [app.main]
                                    :init-fn app.main/init}}
                :devtools   {:preloads [devtools.preload]
                             :after-load app.main/init}}}}

The application output-dir is where the results will be compiled.

2. Define NPM Dependencies

NPM dependencies are defined in the package.json file. It has the npm dependencies the project uses, and since we are configuring a Helix app, we'll need the react and react-dom as dependencies.

{
  "name": "web",
  "version": "0.0.1",
  "private": true,
  "devDependencies": {
    "shadow-cljs": "2.26.2"
  },
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  }
}

3. ClojureScript Source

For this step, we only need some ClojureScript code to get started. Let's start with something simple and return to the application code later.

(ns app.main)

(. js/console log  "Hello, app.main!")

Confirm the Build Works

Now that we have all the required steps completed let's compile the release build to see if we set up everything correctly.

❯ npx shadow-cljs release app

shadow-cljs - config: /home/tvaisanen/projects/digitalocean-clojure-minimal/api/shadow-cljs.edn
shadow-cljs - connected to server
[:app] Compiling ...
[:app] Build completed. (45 files, 1 compiled, 0 warnings, 1.25s)

❯ ls -ul compiled-resources/public/js

.rw-r--r--  175 tvaisanen  2 Jan 11:58 main.js
.rw-r--r-- 1.3k tvaisanen  2 Jan 11:58 manifest.edn

var shadow$provide = {};
(function(){
'use strict';/*

 Copyright The Closure Library Authors.
 SPDX-License-Identifier: Apache-2.0
*/
console.log("Hello, app.core!");
}).call(this);

It looks like it's working as expected. We are ready to continue to the next phase of serving the files from our backend.

Serving the SPA

In the previous step, we did not yet configure a way for the browser to load the built files. For this, we need an HTML file that loads the Javascript. So, let's create the file resources/public/index.html to do just that.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>
  <body>
    <div id="app" />
    <script src="/js/main.js" type="text/javascript"></script>
  </body>
</html>

As you might have noticed, the index file has a relative path to the compiled resources, but our files are in resources and compiled-resources. We want to keep the static files separate from the build artifacts. This helps keep the build process cleaner since we do not need to think whether the compile-resources folder has development files. It'll be created in the build time and kept from version control.

❯ tree resources

resources
├── migrations
│  ├── 20231208151434-test.down.sql
│  └── 20231208151434-test.up.sql
└── public
   └── index.html

❯ tree compiled-resources

compiled-resources
└── public
   └── js
      ├── main.js
      └── manifest.edn

Effectively, we want to merge the resources and compiled-resources in the backend, when the files are served, their respective paths are seen by the browser like they are coming from the same path. We can do this by adding both resources and compiled-resources to the paths in deps.edn.

modified   api/deps.edn
@@ -1,4 +1,4 @@
-{:paths ["src/clj" "resources"]
+{:paths ["src/clj" "resources" "compiled-resources"]

Then, we look at resources by their names in the backend.

(require '[clojure.java.io :as io])

(for [resource (concat (file-seq (io/file (io/resource "public")))
                       (file-seq (io/file (io/resource "public/js"))))]
   (.getName resource))
;; => ("public" "index.html" "js" "main.js" "manifest.edn")

As expected, we can find all of the files with their names. This enables us to create a resource handler for the API to serve both resource directories. Next, let's jump to the backend routing.

Backend Routing

We'll be using metosin/reitit for the backend routing. It also provides functions to serve the resources.

modified   api/deps.edn
@@ -3,9 +3,10 @@
-        migratus/migratus {:mvn/version "1.5.4"}}
+        migratus/migratus {:mvn/version "1.5.4"}
+        metosin/reitit {:mvn/version "0.7.0-alpha7"}}

We need a couple of updates on the API code.

1. Refactor the previous app to /api/* handler

2. Configure the router to serve both the api and the static files.

(ns api.main
  (:require [api.db :as db]
            [ring.adapter.jetty :as jetty]
            [reitit.ring :as ring]
            [clojure.java.io :as io]
            [clojure.string :as str])
  (:gen-class))

(defn get-port []
  (Integer/parseInt (or (System/getenv "PORT")
                        "8000")))

(defn handler [_request]
  (let [migrations (db/get-migrations)]
    {:status  200
     :headers {"Content-Type" "application/edn"}
     :body    (str migrations)}))

(defn index-handler
  [_]
  {:status  200
   :headers {"content-type" "text/html"}
   :body    (io/file (io/resource "public/index.html"))})

(def app
  (ring/ring-handler
   (ring/router
    [["/api"
      ["*" {:get handler}]]
     ["/" index-handler]
     ["/*" (ring/create-resource-handler
            {:not-found-handler
             (fn [{:keys [uri] :as r}]
               (if (str/starts-with? uri "/api")
                 {:status 404}
                 (index-handler r)))})]]
    {:conflicts (constantly nil)})
   (ring/create-default-handler)))

(defn -main [& _args]
  (db/run-migrations)
  (jetty/run-jetty #'app {:port (get-port)}))

Let's break the router definition into parts to see what's happening. Our requirements, at the time being, for the routing are:

1. to be able to serve both api and static assets.

2. return index.html if the static asset is not found

3. serve the App from the root

(ring/router
    [;; serve API
     ["/api" 
      ["*" handler]]
     ;; serve app from root
     ["/" index-handler]
     ;; in other case serve static
     ["/*" (ring/create-resource-handler
            {:not-found-handler
             (fn [{:keys [uri] :as r}]
               (if (str/starts-with? uri "/api")
                 ;; if the uri is an API path
                 ;; serve NOT FOUND
                 {:status 404}
                 ;; every other case serve the app
                 ;; so that FE routing can try to 
                 ;; resolve the path
                 (index-handler r)))})]]
    {:conflicts (constantly nil)})

Let's add a dev/clj/user.clj namespace to start the server via REPL

(ns user
  (:require [api.main :as main]))

(def server (atom nil))

(defn start! []
  (reset! server
          (main/start! {:port  8000
                        :join? false})))

(defn stop! []
  (.stop @server))

(comment
  (start!)
  (stop!))

And create an alias to load the user namespace by default.

modified   api/deps.edn
- :aliases {:run
+ :aliases {:dev
+           {:extra-paths ["env/clj"]
+            :ns-default user}

❯ clj -M:dev
2024-01-02 13:41:03.950:INFO::main: Logging initialized @670ms
Clojure 1.11.0

user=> (start!)

Jan 02, 2024 1:41:07 PM clojure.tools.logging$eval2887$fn__2890 invoke
INFO: Starting migrations
Jan 02, 2024 1:41:07 PM clojure.tools.logging$eval2887$fn__2890 invoke
INFO: Ending migrations
2024-01-02 13:41:07.296:INFO:oejs.Server:main: jetty-9.2.21.v20170120
2024-01-02 13:41:07.305:INFO:oejs.ServerConnector:main: Started ServerConnector@2c6f022d{HTTP/1.1}{0.0.0.0:8000}
2024-01-02 13:41:07.306:INFO:oejs.Server:main: Started @4026ms
#object[org.eclipse.jetty.server.Server 0x2e7e4480 "org.eclipse.jetty.server.Server@2e7e4480"]

Now, we should be able to fetch the static resources

❯ http :8000/
HTTP/1.1 200 OK
Content-Length: 307
Content-Type: text/html
Date: Tue, 02 Jan 2024 11:44:41 GMT
Server: Jetty(9.2.21.v20170120)

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>
  <body>
    <div id="app" />
    <script src="/js/main.js" type="text/javascript"></script>
    <script>
      app.core.init();
    </script>
  </body>
</html>

and access the API endpoints via HTTP.

❯ http :8000/api/
HTTP/1.1 200 OK
Content-Type: application/edn
Date: Tue, 02 Jan 2024 11:46:00 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

[{:migrations/id 20231208151434, 
  :migrations/applied #inst "2024-01-01T09:17:26.936000000-00:00", 
  :migrations/description "test"}]

And that's a wrap on the backend side.

Frontend Code

Finally, we can set up the React side of the application. Let's start by adding a utils namespace and define some utilities for rendering the application.

(ns app.utils
  (:require [helix.core :refer [$]]
            ["react" :as react]
            ["react-dom/client" :as rdom]))

(defn app-container []
  (js/document.getElementById "app"))

(defonce root (atom nil))

(defn react-root []
  (when-not @root
    (reset! root (rdom/createRoot (app-container))))
  @root)

(defn render
  [App]
  (.render (react-root)
           ($ react/StrictMode
              ($ App))))

Next, use the utils to render an App that renders the API response in the DOM.

(ns app.main
  (:require [app.utils :as utils]
            [cljs.pprint :refer [pprint]]
            [clojure.edn :as edn]
            [helix.core :refer [defnc]]
            [helix.dom :as d]
            [helix.hooks :as hooks]))

(defnc App []
  (let [[data set-data] (hooks/use-state {})]
    (hooks/use-effect
     :once
     (-> (js/fetch "/api")
         (.then (fn [res] (. res text)))
         (.then (fn [response-string]
                  (set-data (edn/read-string response-string))))))
    (d/div
     (d/div "App Here")
     (d/pre (with-out-str
              (pprint data))))))

(defn init []
  (utils/render App))

Development Setup

For development time, we can start the CLJS compiler in watch mode to get the updated code in the browser whenever the source code changes.

npx shadow-cljs watch app

After this step, start the API with the :dev profile.

❯ clj -M:dev
2024-01-02 13:41:03.950:INFO::main: Logging initialized @670ms
Clojure 1.11.0
user=> (start!)

This should be enough to get us started on writing code for the frontend. Shadow CLJS also provides a CLJS REPL in the browser for us to connect to, but that's out of the scope of this post. Read how to do this from the docs, or let me know if that'd be something you're interested in learning.

Deployment

We'll package the CLJS build into our Dockerfile so that the deployment process will be the same as previously. The only change is that we use the new Dockerfile for the build.

Let's create a new api/Dockerfile to build the frontend code as a separate step,

FROM clojure:openjdk-17-tools-deps-alpine AS frontend-build
RUN apk add --update nodejs npm
WORKDIR /app
COPY  . .
RUN npm install
RUN npx shadow-cljs release app

# Use a base without node dependencies for serving
FROM clojure:openjdk-17-tools-deps-alpine
WORKDIR /app
COPY . .
RUN ls /app
COPY --from=frontend-build /app/compiled-resources /app/compiled-resources
RUN clojure -P
CMD clojure -X:run

update the api/docker-compose.yml,

modified   api/docker-compose.yml
@@ -5,7 +5,7 @@ services:
   api:
     build:
       context: .
-      dockerfile: docker/dev.Dockerfile
+      dockerfile: Dockerfile
       tags:
         - "registry.digitalocean.com/clojure-sample-app/dev"
     environment:

and then build, start the service, and confirm the build works as expected.

docker-compose up --build

Navigate to localhost:8000 to ensure the app is rendering. You can also see the migrations vector rendered on the screen.

If you haven't followed along with the previous posts, you can find the deployment instructions here.

Conclusion

There are multiple ways to deploy frontend applications. You could bake the static files into a container with a web server like NGINX, serve the files from a bucket with CDN, use the DigitalOcean static App, use a service like Netlify, or host the static files from your GitHub pages. But if you have an API component in your App. It makes sense to pair the frontend code tightly with the API to make the deployment phase more robust. What do I mean by that in this context? If we have the exact version of the frontend shipped with the API itself, there's no room for version mismatch.

Once again, thanks for reading; I hope you found this helpful. Here's the accompanying GitHub repository.

This is the third post on a series building on DigitalOcean. Read the previous posts in the series to catch up on the context if you haven't done so already.

So, without further ado, let's get to the topic and configure Migratus for running the database migrations.

Configure Migrations

To use Migratus, first, we must add the dependency to our deps.edn file and get the latest version from Clojars. I also like to create an alias into the file to have a command line invocation to generate the migration files that we can later use with clj -X:create-migration <migration-name>.

{:paths ["src" "resources"]

 :deps {...
        migratus/migratus {:mvn/version "1.5.4"}}

 :aliases {...
           :create-migration
           {:exec-args {:name nil}
            :exec-fn main/create-migration}}}

The next step is to run the migrations on application startup. Let's do this by updating the main.clj by first requiring migratus, creating a configuration and the create-migration-function configured for the create-migration alias.

(ns main
  (:require [ring.adapter.jetty :as jetty]
            [migratus.core :as migratus] ;; Add migratus
            [next.jdbc :as jdbc])
  (:gen-class))

(defn get-port []
  (Integer/parseInt (System/getenv "PORT")))

(defn get-db-conf []
  {:dbtype  "postgres"
   :jdbcUrl (System/getenv "JDBC_DATABASE_URL")})

;; THIS IS NEW ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defn migrations-config []
  {:db                   (get-db-conf)
   :store                :database
   :migration-dir        "migrations/"
   :migration-table-name "migrations"
   :init-in-transaction? false})

(defn create-migration [{:keys [name]}]
  (migratus.core/create (migrations-config)
                        name))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defn datasource []
  (jdbc/get-datasource (get-db-conf)))

(defn app [_request]
  (let [migrations (jdbc/execute! (datasource) 
                                  ;; Get migrations instead of version
                                  ["SELECT * FROM migrations"])]
    {:status  200
     :headers {"Content-Type" "application/edn"}
     :body    (str migrations)}))

(defn -main [& _args]
  ;; Run migrations everytime the application starts!
  (migratus/migrate (migrations-config))
  (jetty/run-jetty #'app {:port (get-port)}))

We create a test migration with the alias we created in the first step.

❯ clj -X:create-migration :name test
❯ tree resources
resources
└── migrations
   ├── 20231208151434-test.down.sql
   └── 20231208151434-test.up.sql

And that's it for the configuration. If you're going to add PostgreSQL schemas in your test migration, note that if you use multiple SQL statements, you need to separate each of them with --;;.

Test the Migrations Locally

Now that we have the application updated and migrations configured, let's validate that everything works as expected (If you haven't already tested all of this in the REPL). But before that, let's create another Dockerfile for the development build and add the PostgreSQL package to have it available in the DigitalOcean application console. This will allow us to connect to the development database from DigitalOcean with a command-line interface.

FROM clojure:openjdk-17-tools-deps-alpine

RUN apk update; apk add postgresql

WORKDIR app
COPY . .
RUN clojure -P
CMD clojure -X:run

And next, update the docker-compose.yml to use the development Dockerfile.

services:

  api:
    build:
      context: .
      dockerfile: docker/dev.Dockerfile # <-- use the new file

After executing docker-compose up --build we should have the updated application running in a container with the database. Use another terminal window from the same directory to test that we can use the psql client from within our application container.

❯ docker-compose run api psql -h postgres -U user -d db

The password is defined in the docker-compose.yml's JDBC_DATABASE_URL. Now that we have successfully connected to the database, let's see if we have the expected table migrations listed with \dt and that we have the expected test migration applied.

db=# \dt
          List of relations
 Schema |    Name    | Type  | Owner
--------+------------+-------+-------
 public | migrations | table | user
(1 row)

db=# select * from migrations;
       id       |        applied         | description
----------------+------------------------+-------------
 20231208151434 | 2023-12-09 08:29:05.82 | test
(1 row)

It's looking good. As a last step, let's try to retrieve the same content via the HTTP API.

❯ http :8000
HTTP/1.1 200 OK
Content-Type: application/edn
Date: Sat, 09 Dec 2023 08:33:08 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

[{:migrations/id 20231208151434, 
  :migrations/applied #inst "2023-12-09T08:29:05.820000000-00:00", 
  :migrations/description "test"}]

Everything is working as expected. Let's see how we can follow the same steps on the deployed application.

Deploy the Changes to DigitalOcean

Before deploying the changes, we need to update the applications run_command since the Alpine image doesn't have rlwrap installed.

@@ -41,7 +41,7 @@ resource "digitalocean_app" "app" {
       source_dir = "api/"
       http_port  = 8000

-      run_command = "clj -X:run"
+      run_command = "clojure -X:run"
     }

     database {

First, push the image to your DigitalOcean Container Registry (DOCR) and wait for the deployment to finish. You can find the instructions for the deployment from the previous posts. Refer to the first post on how to set up the DigitalOcean project and push images to the DigitalOcean Container Registry and the second post to update the application to read the environment variables dynamically.

Now that the changes are deployed, we can visit our DigitalOcean dashboard and find the application console. This provides access similar to what we did earlier with the docker-compose exec ..., which is why we installed the Postgres client in the image itself. I wanted to show that it is possible to connect to the database for debugging, maintenance, or whatever reasons when using the application platform.

Finally, as a last step, let's call the endpoint to see how the API returns the migrations table data.

❯ http https://sample-app-mffks.ondigitalocean.app
HTTP/1.1 200 OK
CF-Cache-Status: MISS
CF-RAY: 8326da43ccded93f-HEL
Connection: keep-alive
Content-Type: application/edn
Date: Fri, 08 Dec 2023 17:50:46 GMT
Last-Modified: Fri, 08 Dec 2023 17:50:46 GMT
Server: cloudflare
Set-Cookie: __cf_bm=eZ9Xa2czsbrF7O71GudfKdryjh.VD8bBKAG5EcXof88-1702057846-0-Ack9rJbWGn1PH2yEzcqHNxJjqfzvEX2n8nWJybWQpIaP63X/LFMEaTMazT1dEjpfdcVYFd30zmZCkZ78wART9yM=; path=/; expires=Fri, 08-Dec-23 18:20:46 GMT; domain=.ondigitalocean.app; HttpOnly; Secure; SameSite=None
Transfer-Encoding: chunked
Vary: Accept-Encoding
cache-control: private
x-do-app-origin: 1a36f444-619d-4979-9436-c8765b32e6f6
x-do-orig-status: 200

[{:migrations/id 20231208151434, 
  :migrations/applied #inst "2023-12-08T17:46:28.392000000-00:00", 
  :migrations/description "test"}]

It works!

Conclusion

To run migrations, you can just run them on the application startup. Migratus is an excellent lightweight option for Clojure. Set up the migration files, create the configuration, and you're ready.

To my knowledge, the DigitalOcean app platform doesn't provide a direct way to connect to the development database from the command line. Still, we can use the app console as a bastion if we've installed psql in the container image. Even if you haven't installed it, you can do that in the console, but you'd need to do this again after each deployment.

Thank you for reading. I hope you found this helpful.

Feel free to reach out and let me know what you think—social links in the menu.

A software bill of materials is a list of all the open source and third-party components present in a codebase. An SBOM also lists the licenses that govern those components, the versions of the components used in the codebase, and their patch status, which allows security teams to quickly identify any associated security or license risks.

Synopsys: What is a Software Bill of Materials

When we use public container registries like the docker hub, some bad actors may have sneaked their malware into the image. If this piece of software becomes part of our SBOM, it can compromise the application. We should include only the essential dependencies in our containers to minimize the risk.

Let's take the Docker configuration used in the previous post, Deploying Clojure Like a Seasoned Hobbyist, as an example and use it as our vehicle to explore this topic further from the perspective of how much extra is included in the container and what known vulnerabilities we can find with a security scan, try reducing the image size and SBOM, and compare the final artifact sizes between the different approaches.

Examples are available in the repository.

Inspecting The Image

I'll tag the build (from the previous post) clj-tools-deps-buster to have a shorter descriptive name for this context.

❯ docker tag \
    registry.digitalocean.com/clojure-sample-app/dev \
    clj-tools-deps-buster

First, we want access to the manifest to find the container configuration, so let's start by creating an archive from all of the files in the container, including the manifest.

❯ docker save clj-tools-deps-buster > clj-tools-deps-buster.tar

Then, let's unpack the archive to get access to the files.

❯ tar -xvf clj-tools-deps-buster.tar

Now, we can find the configuration file from the manifest.json.

❯ cat manifest.json| jq ".[0].Config"
"3085b45633b1c0f87a3ea67000d7d77e2f14a74d1b2968966f866c7c3531f745.json"

The configuration file is too large to add here, so let's fire up a Babashka REPL and investigate the configuration file a bit further.

If we look at this container's history, the last four steps come from the Dockerfile we used, and the rest already exist in the container.

(require '[cheshire.core :as json])

(def config
  (->  "3085b45633b1c0f87a3ea67000d7d77e2f14a74d1b2968966f866c7c3531f745.json"
       slurp
       (json/parse-string true)))

(keys config)
;; => (:architecture :config :created :history :os :rootfs)

(count (get config :history))
;; => 23

(for [{:keys [created_by]} (take-last 4 (get config :history))]
   created_by)
;; =>
;; ("WORKDIR /tmp/app"
;;  "COPY . . # buildkit"
;;  "RUN /bin/sh -c clojure -P # buildkit"
;;  "CMD [\"/bin/sh\" \"-c\" \"clj -X:run\"]")

We can see that the commands are the same as configured in the previous post's Dockerfile. Other authors did all the previous steps, so we might not know what they did. Let's take the third entry from the history as an example.

(print (get-in config [:history 2 :created_by]))
;; /bin/sh -c set -eux; apt-get update; \
;; apt-get install -y --no-install-recommends -certificates curl netbase wget; \
;; rm -rf /var/lib/apt/lists/*

The original author of the base image had decided to install curl, netbase, and wget, which all ended up in our application image.

Let's take a look at what commands we have pre-installed on the container by first finding all apt-get install commands from the image history.

(def install-commands
  (for [{:keys [created_by]} (get config :history)
        :when (str/includes? created_by "apt-get install")]
    created_by))

(count install-commands)
;; => 5

It looks like we have five different steps using the apt-get install. Let's see what these are all about.

(for [command install-commands]
  (into []
        (comp 
         (filter #(str/includes? % "apt-get install"))
         (map str/trim))
        (-> command
            (str/replace #"\t" "")
            (str/split #";"))))

;; (["apt-get install -y --no-install-recommends ca-certificates curl netbase wget"]
;;  ["apt-get install -y --no-install-recommends gnupg dirmngr"]
;;  ["/bin/sh -c apt-get update && apt-get install -y --no-install-recommends git mercurial openssh-client subversion procps && rm -rf /var/lib/apt/lists/*"]
;;  ["apt-get install -y --no-install-recommends bzip2 unzip xz-utils binutils fontconfig libfreetype6 ca-certificates p11-kit"]
;;  ["/bin/sh -c apt-get update && apt-get install -y make rlwrap && rm -rf /var/lib/apt/lists/* && wget https://download.clojure.org/install/linux-install-$CLOJURE_VERSION.sh && sha256sum linux-install-$CLOJURE_VERSION.sh && echo \"7677bb1179ebb15ebf954a87bd1078f1c547673d946dadafd23ece8cd61f5a9f *linux-install-$CLOJURE_VERSION.sh\" | sha256sum -c - && chmod +x linux-install-$CLOJURE_VERSION.sh && ./linux-install-$CLOJURE_VERSION.sh && rm linux-install-$CLOJURE_VERSION.sh && clojure -e \"(clojure-version)\""])

The result is that we have a bunch of unnecessary executables for the final container. On top of these, to get the complete picture of what is installed, we would still need to go through the Clojure install scripts. But we're not going to go there this time around. Let's continue with the image itself.

Before moving forward, let's see the size of the image and if the base image has any known vulnerabilities.

❯ docker image ls | grep clj-tools-deps-buster
clj-tools-deps-buster ...  725MB

The image size is 725MB. A screenshot from the docker hub shows that the base image we used has 17 critical and 27 high-priority vulnerabilities. Let's get back to these numbers a bit later.

Reduce the Image Size

Next, let's see if we can reduce the image size and vulnerabilities by changing to smaller base images.

Using Slim Buster

Create a new file slim.Dockerfile

FROM clojure:openjdk-17-tools-deps-slim-buster
WORKDIR app
COPY . .

RUN clojure -P

CMD clj -X:run

Build the image.

❯ docker build \
  --file=slim.Dockerfile . \
  -t clj-tools-deps-slim --no-cache

Check the size of the container.

❯ docker image ls | grep clj-tools-deps-slim
clj-tools-deps-slim ... 553MB

We see that by using the clojure:openjdk-17-tools-deps-slim-buster , we successfully reduced the image size by 172 MB and made minor improvements in vulnerability counts.

And let's do the same once more with the Alpine version.

Using Alpine Linux Image

Once again, create a new docker file alpine.Dockerfile.

FROM clojure:openjdk-17-tools-deps-alpine
WORKDIR app
COPY . .

RUN clojure -P

CMD clj -X:run

Build the image and check the size.

❯ docker image ls | grep clj-tools-deps-alpine
clj-tools-deps-alpine ... 356MB

This is good progress. We went from 725MBs to 356MBs. The net total is -369MBs; we can see improvements with vulnerability counts.

We can still improve from here by going distroless. Yes, you read correctly, distroless.

Distroless Containers

Google provides language-specific base containers to run application code without a Linux distribution on the image. These base images don't even have shells. When using a language like Rust, You could have only the kernel and include the standard C-libraries in your binary.

🥑 Language focused docker images, minus the operating system.

Github: Google Container Tools

For Clojure code, we'll use the Java image gcr.io/distroless/java17-debian12. We must build a JAR file for the containers to run the application code without the Clojure dependencies.

So, let's get to it.

Build JAR

First, let's create a JAR build of the application following the instructions in the Clojure build.tools reference.

Let's start by updating the application to dynamically read the env variables since they are unavailable during the build time.

(ns main
  (:require [ring.adapter.jetty :as jetty]
            [next.jdbc :as jdbc])
  (:gen-class))

(defn get-port []
  (Integer/parseInt (System/getenv "PORT")))

(defn get-db-conf []
  {:dbtype "postgres"
   :jdbcUrl (System/getenv "JDBC_DATABASE_URL")})

(defn datasource []
  (jdbc/get-datasource (get-db-conf)))

(defn app [_request]
  (let [db-version (jdbc/execute! (datasource) ["SELECT version()"])]
    {:status  200
     :headers {"Content-Type" "application/edn"}
     :body    (str db-version)}))

(defn -main [& _args]
  (jetty/run-jetty #'app {:port (get-port)}))

Add new alias build into deps.edn

{...
 :aliases 
  {....
   :build {:deps {io.github.clojure/tools.build
                   {:git/tag "v0.9.6" :git/sha "8e78bcc"}}
           :ns-default build}}}

And then create a build.clj file.

(ns build
  (:require [clojure.tools.build.api :as b]))

(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file "target/standalone.jar")

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs   ["src" "resources"]
               :target-dir class-dir})
  (b/compile-clj {:basis      basis
                  :ns-compile '[main]
                  :class-dir  class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis     basis
           :main      'main}))

Now, we should be able to build the JAR file by running:

❯ clj -T:build uber

After this step, we should have the following files in the target folder.

❯ ls target
classes  standalone.jar

Using Distroless Java

Lastly, let's create a distroless image to run the application JAR.

FROM clojure:openjdk-17-tools-deps-buster as base

WORKDIR app
COPY . .
RUN clj -T:build uber

FROM gcr.io/distroless/java17-debian12
COPY --from=base /tmp/app/target/standalone.jar /tmp/app/standalone.jar

CMD ["/tmp/app/standalone.jar"]

As Timo Kramer pointed out, we can also create the container with our build script using Google's Jib library. Take a look at how Datahike uses it.

Build your image and check the image size.

❯ docker image ls | grep jvm-distroless
jvm-distroless ... 234MB

Let's run a security scan for all the images and see how the numbers compare.

Comparing Image Sizes and Known Vulnerabilities

For this step, I'll be using Trivy since the docker hub doesn't provide the numbers for the distroless images.

Trivy is a comprehensive and versatile security scanner. Trivy has scanners that look for security issues, and targets where it can find those issues.

Github: Trivy

Here's an example of a partial output. See the complete reports here.

❯ trivy image clj-tools-deps-alpine

clj-tools-deps-buster (debian 10.12)
======
Total: 1016 (UNKNOWN: 9, LOW: 611, MEDIUM: 170, HIGH: 177, CRITICAL: 49)

....

Java (jar)
=====
Total: 48 (UNKNOWN: 0, LOW: 10, MEDIUM: 20, HIGH: 14, CRITICAL: 4)

Trivy scans the system and the JAR vulnerabilities in one go, so I'll add both separately to the graphs.

We can see clearly that in the case of system vulnerabilities, the amount goes down with the container size. The image size doesn't affect JAR vulnerabilities significantly. Based on this, the clj-tools-deps-buster probably has some extra development time dependencies that are not required for the final image.

Conclusion

Distroless images might not suit your use case, and going distroless doesn't mean the container will be bulletproof. But it's still good to know that it's an option. There's much more to container security than just the packages installed on the image. Minimizing the SBOM is one more trick on your sleeve. I recommend reading OWASP's docker security cheatsheet for additional security steps.

I hope you found this helpful. Thank you for reading.

Feel free to reach out and let me know what you think—social links in the menu.

This post is not about what, when, and if you should test but a tutorial on getting started when you've decided that you need them and how to integrate the testing into your workflow.

How Clojure is Different From the Mainstream

In other dynamic languages, you often run tests in watch mode so that your test runner triggers a new run whenever you save the file. This offers a fast feedback loop on your changes. Clojure differs from the others in that it has the REPL that can be used to evaluate the code while developing and testing it without running all of the tests every time. Let's get back to the topic of running the tests after we've gone through how to write tests.

Writing Tests

Clojure has its unit testing framework core.test ; therefore, extra dependencies are unnecessary when writing the tests. At the core of testing are assertions, and we use the macro clojure.test/is that takes any predicate with an optional message describing it.

(require '[clojure.test :as t])

(t/is (true? true) "true is true")
;; => true

(t/is (= true false) "true equals false")
;; => false

;; REPL output
;;
;; FAIL in () (NO_SOURCE_FILE:4)
;; true equals false
;; expected: (= true false)
;;  actual: (not (= true false))

You can think of this as a function call to evaluate whether a given predicate (boolean or a function that produces a boolean) is true.

Assertions are used within deftests macro that returns a function to evaluate the assertions inside.

(t/deftest test-assertion
  (t/is (= true false) "true equals false")
  (t/is (true? true) "true is true"))
;; => #'core-test/test-assertion

(test-assertion)
;; REPL output
;;
;; FAIL in (test-assertion) (NO_SOURCE_FILE:17)
;; expected: (= true false)
;;  actual: (not (= true false))

Read the Clojure style guide on naming conventions

If you read the REPL outputs, you might have noticed we lost the message attached to our failing assertion. With deftests we can use testing to define the context for the test.

(t/deftest test-assertion
  (t/testing  "true equals false"
    (t/is (= true false)))
  (t/testing "true is true"
      (t/is (true? true))))
;; => #'core-test/test-assertion

(test-assertion)

;; REPL output
;;
;; FAIL in (test-assertion) (NO_SOURCE_FILE:25)
;; true equals false
;; expected: (= true false)
;;   actual: (not (= true false))

You can see that we got our descriptions back. And that's pretty much the basics of testing in Clojure. There are more tools in clojure.test like are and use-fixture. Let me know if you want to learn more about them. But now, back to test runners.

There are several test runner options to choose from. You could use the functions provided by core.test, Cognitect's test-runner, or Kaocha, to name a few. We'll be using Kaocha this time. It is an extendable, build-tool agnostic test runner.

Setup the project

Let's set up a project with Kaocha testing in mind so that we have a deps.edn alias for running the tests once and in watch mode.

Here's the file structure

├── deps.edn
├── src
│   └── core.clj
└── test
    └── core_test.clj

Let's fill in the deps.edn first and add Kaocha as a test dependency.

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}}

 :aliases
 {:test
  {:extra-paths ["test"]
   :extra-deps  {lambdaisland/kaocha {:mvn/version "1.77.1236"}}
   :exec-fn     kaocha.runner/exec-fn
   :exec-args   {:skip-meta :slow}}

  :watch
  {:exec-args   {:watch?     true
                 :skip-meta  :slow
                 :fail-fast? true}}}}

Under aliases, we have keys :test and :watch and these work together. With the Clojure CLI tool, we can execute the :exec-fn from the alias configuration by using the -X command line option

-X is configured with an arg map with :exec-fn and :exec-args keys, and stored under an alias in deps.edn:

Deps and CLI Reference: Execute a Function

so by running clj -X:test we can run the tests once, and with clj -X:test:watch we can continuously run the tests whenever we save the source code file.

Let's write something to test in core.clj namespace.

(ns core)

(defn multiply [x y]
  (* x y))

And write the tests we saw earlier, plus a test for multiplication in core-test.

(ns core-test
  (:require [clojure.test :as t]
            [core :as sut]))

(t/deftest test-assertion
  (t/testing  "true equals false"
    (t/is (= true false)))
  (t/testing "true is true"
      (t/is (true? true))))

(t/deftest multiply-test
  (t/testing "multiplication works as expected"
    (t/is (= 4 (sut/multiply 2 2)))))

SUT (Software Under Test) I picked up this from Practicalli years ago and it stuck with me.

And now we are ready to test.

By default, Kaocha uses the src and tests paths so we can run the tests without explicit configuration. There are various configuration options with plugins, so check out the docs to see what's possible.

Testing in the REPL

Clojure developers usually use their REPL to evaluate the tests, and in my case, it would be in Emacs with CIDER using one of the cider-test-* commands that I have behind keyboard shortcuts.

The same can be done with VS code and Calva. Read from the docs how to run tests with Calva.

Conclusion

Testing in Clojure is pretty effortless. We do not need much ceremony to start testing; we need a namespace for the tests and a test runner of our choice.

I hope you found this helpful, and thank you for reading.

Feel free to reach out and let me know what you think—social links in the menu.

So, let's get to it.

In Javascript, we'd import the fs-module.

const fs = require('node:fs/promises')

I'll be running all of the Javascript blocks inside a (async () => { ...code... })() closure.

In Clojure, we'd reach out for clojure.java.io

(require '[clojure.java.io :as io])

Node Filehandle and Java File

In Javascript, we can get a reference to a file with a Filehandle.

let directory = await fs.open(".");

console.log(directory)

// FileHandle {
//     close: [Function: close],
//     [Symbol(kHandle)]: FileHandle {},
//     [Symbol(kFd)]: 20,
//     [Symbol(kRefs)]: 1,
//     [Symbol(kClosePromise)]: null
// }

FileHandle can be either a directory or a file.

let directoryStats = await fs.stat(".");
directoryStats.isFile(); // false
directoryStats.isDirectory(); // true

let fileStats = await fs.stat("files/a.txt");
fileStats.isFile()); // true
fileStats.isDirectory(); // false

The basic building block in Clojure is java.io.File.

User interfaces and operating systems use system-dependent pathname strings to name files and directories. This class presents an abstract, system-independent view of hierarchical pathnames.

Docs: java.io.File

(io/file ".")
;; => #object[java.io.File 0x54162701 "."]

Similarly to Filehandle in JS, java.io.File can be either a directory or a file. The File has isDirectory and isFile methods that can be called via Java-interop.

(.isDirectory (io/file "."))
;; => true

(.isFile (io/file "."))
;; => false

(.isDirectory (io/file "deps.edn"))
;; => false

(.isFile (io/file "deps.edn"))
;; => true

Let's see what other properties the java.io.File has with the bean function.

bean

Takes a Java object and returns a read-only implementation of the map abstraction based upon its JavaBean properties.

Source

(bean (io/file "."))
;; => 
{:path ".",
 :freeSpace 146245873664,
 :parent nil,
 :directory true,
 :parentFile nil,
 :name ".",
 :file false,
 :canonicalFile
 #object[java.io.File 0x4f2abe7e "/home/tvaisanen/projects/tmp"],
 :absolute false,
 :absoluteFile
 #object[java.io.File 0x5594224b "/home/tvaisanen/projects/tmp/."],
 :hidden true,
 :class java.io.File,
 :canonicalPath "/home/tvaisanen/projects/tmp",
 :usableSpace 124331769856,
 :totalSpace 429923737600,
 :absolutePath "/home/tvaisanen/projects/tmp/."}

Listing Files

Given we have the following file structure.

❯ tree files
files
├── a.txt
├── b.txt
├── c.txt
└── nest
    └── a.txt

In Javascript, we'd typically write.

let dir = await fs.readdir("files");
console.log(dir);
// [ 'a.txt', 'b.txt', 'c.txt', 'nest' ]

And in Clojure, we'll take the folder we want to list as java.io.File and list all the files with file-seq.

(file-seq (io/file "files"))
;; =>
(#object[java.io.File 0x7e2b40d7 "files"]
 #object[java.io.File 0x22aaad8f "files/a.txt"]
 #object[java.io.File 0x3730fe92 "files/b.txt"]
 #object[java.io.File 0x62aaf1b "files/c.txt"]
 #object[java.io.File 0x22236af1 "files/nest"]
 #object[java.io.File 0x55126def "files/nest/a.txt"])

I'd expect this to return files similarly fs.readdir, but now we have the nested files in the listing. Let's see why that happens and if we could get it to work similarly to the listing in Javascript.

If we take a look at the file-seq source code

(defn file-seq
  "A tree seq on java.io.Files"
  {:added "1.0"
   :static true}
  [dir]
    (tree-seq
     (fn [^java.io.File f] (. f (isDirectory)))
     (fn [^java.io.File d] (seq (. d (listFiles))))
     dir))

The docstring says that the function returns a tree seq, which sounds like a non-flat structure. And if we look deeper into tree-seq function and get the docstring.

Returns a lazy sequence of the nodes in a tree, via a depth-first walk. ...

And here's the explanation of what is happening: "a depth-first walk." This function will go first till the end of the file tree and start coming back towards the root directory. The first argument is a function that returns a boolean value to decide whether the node (read file or directory) needs to be traversed (read looked into). If it returns true, the same file is passed to the second function to list its nodes (read files and subdirectories). Based on this, the tree-seq will do a depth-first search and, therefore, read all the subdirectories.

You might have already noticed that the second function argument calls (. d (listFiles)) to get the files for our directory d. Let's use this to mimic the JS version readdir.

(seq (. (io/file "files") listFiles))
;; =>
(#object[java.io.File 0x356bf4d3 "files/a.txt"]
 #object[java.io.File 0x13c28931 "files/b.txt"]
 #object[java.io.File 0x444349ab "files/c.txt"]
 #object[java.io.File 0x2e64f09c "files/nest"])

At this point, the expected result is almost identical to the Javascript's fs.readdir. We have the files listed in the directory, but instead of file names, we have java.io.File instances. Let's iterate over the files and once again use Java-interop with the getName method.

(for [file (seq (. (io/file "files") listFiles))]
  (.getName file))
;; => ("a.txt" "b.txt" "c.txt" "nest")

The fs.readdir function will throw if called with a filename that is not a directory. If we try to create a file sequence from a java.io.File that is not a directory, we get only the file itself in a list.

(file-seq (io/file "files/a.txt"))
;; => (#object[java.io.File 0x37eeb5e7 "files/a.txt"])

Let's combine what we've learned from the previous examples into a new function read-dir that takes a path, checks that it is not a file, and then lists the files and directories in the folder but not the subpages.

(defn read-dir [d]
  (let [directory (io/file d)]
    (when (. directory isFile)
      (throw (ex-info "Path is not a directory" {:path directory})))
    (doall
     (for [file (seq (. directory listFiles))]
       (.getName file)))))

(read-dir "files")
;; => ("a.txt" "b.txt" "c.txt" "nest")

(read-dir "files/a.txt")
;; Unhandled clojure.lang.ExceptionInfo
;; Path is not a directory
;; {:path #object[java.io.File 0x7c7a2da9 "files/a.txt"]}

That's enough on listing files for now. Next, let's look into reading the files.

Read Files

Let's see the content for a couple of the example files.

❯ cat files/a.txt
AAAAA
❯ cat files/b.txt
BBBB
1111
2222
3333

In Javascript, we'd typically use the fs.readFile.

let content = await fs.readFile("files/a.txt", {encoding: "utf8"});
console.log(content);
// AAAAA

In Clojure, we default to slurp.

(slurp (io/file "files/a.txt"))
;; => "AAAAA\n"

Slurp also works with the filename since if the argument is a string; it's first coerced (read interpreted as) as a URI and second as a local file.

(slurp "files/a.txt")
;; => "AAAAA\n"

Both fs.readdir and slurp read the file into memory at once; therefore, you might want to avoid it in production!

The more responsible way is to read the files as streams line by line.

let file = await fs.open('files/b.txt',);

for await (const line of file.readLines()) {
    console.log(line);
}
// BBBB
// 1111
// 2222
// 3333

In Clojure, we can use the java.io.Reader to create a buffered reader for streaming over the files. slurp uses the same mechanism, but instead of reading a file line by line, it reads it all simultaneously.

(io/reader  "files/a.txt")
;; => #object[java.io.BufferedReader 0x4754cc18 "java.io.BufferedReader@4754cc18"]

We can open this stream (read file) to create a lazy line sequence that can be processed one line at a time.

(with-open [reader (io/reader  "files/b.txt")]
  (doall (for [line (line-seq reader)]
           (str "read: " line))))
;; => ("read: BBBB" "read: 1111" "read: 2222" "read: 3333")

Next, to writing files.

Writing Files

With Javascript, we'd typically do this with fs.writeFile or use fs.createFileStream for extra control.

await fs.writeFile("files/write-with-js.txt", "writing from JS");

let content_01 = await fs.readFile("files/write-with-js.txt", 
                                   {encoding: "utf8"});
console.log(content); 
// writing from JS

In Clojure, similarly to slurp we have spit to write data.

(spit "files/new.txt" "New content here")
;; => nil
(slurp "files/new.txt")
;; => "New content here"

By default, spit overrides the file with the given content. To add to the file, use arguments :append true.

(spit "files/new.txt" "Append to file")
;; => nil
(slurp "files/new.txt")
;; => "Append to file"
(spit "files/new.txt" " Try again" :append true)
;; => nil
(slurp "files/new.txt")
;; => "Append to file Try again"

spit wraps the java.io.writer, which can be used directly. Similarly to slurp java.io.writer replaces the file content if :append true is not passed as an option.

(with-open [writer (io/writer "files/other.txt")]
  (.write writer "text here"))

(slurp "files/other.txt")
;; => "text here"

(with-open [writer (io/writer "files/other.txt" :append true)]
  (.write writer " more text"))
;; => nil

(slurp "files/other.txt")
;; => "text here more text"

Let's go through a couple of more use cases.

Renaming Files

In Javascript, we rename files with fs.rename.

await fs.rename("files/c.txt", "files/d.txt");

In Clojure, we use the renameTo method.

(.renameTo (io/file "files/new.txt")
           (io/file "files/newer.txt"))
;; => true
(.isFile (io/file "files/new.txt"))
;; => false
(.isFile (io/file "files/newer.txt"))
;; => true

Deleting Files

In Javascript, we delete files with fs.unlink

 await fs.unlink("files/d.txt");

In Clojure, we have clojure.java.io/delete-file for the same task.

(.isFile (io/file "files/other.txt"))
;; => true
(io/delete-file "files/other.txt")
;; => true
(.isFile (io/file "files/other.txt"))
;; => false

Conclusion

Clojure can do the same tasks as Javascript on the filesystem level since clojure.java.io provides utility functions as a layer on top of java.io classes. If there's no function matching your need, you can use Java-interop by, for example, using the java.io.File methods. If you hit a roadblock, check the following resources for more hints.

• ClojureDocs: clojure.java.io

• JavaDocs: java.io.File

Once again, I hope you found this helpful. Feel free to reach out and let me know what pain points you might have encountered when learning Clojure and what type of posts would help make the jump—social links in the menu.

Thank you for reading.

TLDR: It is possible to declare types for Clojure functions and make them available to the library users by creating clj-kondo configs (type exports). The configs can be created with Malli by collecting function schemas and then transforming these into clj-kondo types in a config file. If the config files are committed into resources, they can be imported by the host project when used as a dependency. Having Malli schemas has the added benefit of enabling the run-time checks via Malli instrumentation.

I like imagining an ecosystem where there'd be a clj-kondo config available for the most used libraries. I thought I would chime in my two cents on spreading the know-how to spread the word on how to do this. I'm by no means an expert on the topic, but I've learned a thing or two that might also be helpful to others.

Suppose you are unfamiliar with Malli and clj-kondo and how they can help you with static code analysis. In that case, I recommend first reading Data Validation in Clojure and Typescript Like Intellisense for Clojure Functions With Malli for background.

This time, I want to dig deeper into how these tools can improve the developer experience by making the types available to our code's human and CI consumers. I don't know if this is already yesterday's news, but this trick is worth knowing.

Let's explore the topic by creating a library, publishing it to Github, then using it as a dependency for another project, and making the library types available to benefit from the extended clj-kondo features.

Publish Library With Types

The only required dependency is Malli, which will be used to generate the clj-kondo types.

Note that in a "real" project Malli would be added as an extra dependency for an alias when creating the types if the library itself is not depending on Malli. If this is something that doesn't make sense to you let me know in the comments.

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        metosin/malli {:mvn/version "0.13.0"}}}

My library will have two functions: adding (x,y) coordinate points and rendering points as a string. If you need help defining the schemas for your use case, check out Malli function schema docs for reference.

(ns tvaisanen.types-export-sample.core
  "Example on how to create and export type clj-kondo
  type definitions that can be installed in a similar way
  that @types/lib-name is used in Typescript.")

(def Point [:map
            [:x :int]
            [:y :int]])

(defn add-points
  {:malli/schema [:=> [:cat Point Point] Point]}
  [p1 p2]
  (merge-with + p1 p2))

(defn render-point
  {:malli/schema [:=> [:cat Point] :string]}
  [p]
  (str p))

The only difference from what you'd typically write is the added metadata key pair. This will be enough for us to get the benefits in later stages when we want to provide the type hints for the library users. Let's take a look at how to make that happen next.

Export Configuration

Exporting is the process of creating the library's clj-kondo configuration under the resources folded as the documentation instructs. To do this, I created a naive exporting function that:

1. Also collects the malli/schema schemas from the functions,

2. generates the type definitions under the Malli clj-kondo-types,

3. copies the config.edn files into resources,

4. and cleans the clj-kondo-types cache.

(require '[malli.dev :as dev]
         '[clojure.java.io. :as io])

(defn export-types []
  ;; collect schemas and start instrumentation
  (dev/start!)

  ;; create export file
  (def export-file
    (io/file "resources/clj-kondo/clj-kondo.exports/tvaisanen/export-types-sample/config.edn"))

  ;; make parents if not exist
  (io/make-parents export-file)

  ;; copy the configs
  (io/copy
   (io/file ".clj-kondo/metosin/malli-types-clj/config.edn")
   export-file)

  ;; clear the cache and stop instrumentation
  (dev/stop!))

If we inspect the .clj-kondo cache folder, we can see that there's a config for malli-types-clj which is used to store the temporary cache while the Malli instrumentation is active (dev/start!) and it's cleared on (dev/stop!).

❯ tree .clj-kondo
.clj-kondo
└── metosin
    └── malli-types-clj
        └── config.edn

What we just did with the export function was copy this config.edn file from the clj-kondo cache into our libs resources for persistence.

❯ tree resources
resources
└── clj-kondo
    └── clj-kondo.exports
        └── tvaisanen
            └── export-types-sample
                └── config.edn

5 directories, 1 file

This is the folder structure that clj-kondo expects to see when it's looking for available configurations to be imported. In my case the config.edn looks like this.

{:linters
 {:unresolved-symbol {:exclude [(malli.core/=>)]},
  :type-mismatch
  {:namespaces
   {export-types-sample.core

    {render-point
     {:arities
      {1 {:args [{:op  :keys,
                  :req {:x :int,
                        :y :int}}],
          :ret  :string}}},

     add-points
     {:arities
      {2 {:args [{:op  :keys,
                  :req {:x :int,
                        :y :int}}
                 {:op  :keys,
                  :req {:x :int,
                        :y :int}}],
          :ret  {:op  :keys,
                 :req {:x :int, :y :int}}}}}}}}}}

From here, we can see that argument and return types are defined for two functions render-point and add-points that have. This tells clj-kondo how the function arguments and return value should look when it's doing the static code analysis.

We are almost done with the first step: creating our library with typehints. Next, commit and push the changes to your Git repo. Here's mine for a reference.

Load the Published Library as a Dependency

Now that we have "published" a library with clj-kondo configs, we can use this as a dependency for another project.

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}

        metosin/malli {:mvn/version "0.13.0"}

        tvaisanen/export-types-sample
        {:git/sha "29f9b259975bde304300e4ca69c70d61622cabab"
         :git/url "https://github.com/tvaisanen/export-types-sample.git"}}}

If you didn't use Github see the available gitlib config options from the docs.

When we fetch the source code for the dependencies, we receive the resources (read configs) in our local cache and the malli function schemas within the source code. We don't need to install additional libs or types to access them. Next, we must collect these configs to our project's .clj-kondo cache. Clj-kondo docs provide the steps on how to do this. Remember to create the .clj-kondo folder if it doesn't exist before copying the configs.

First, we copy the configs.

❯ clj-kondo --lint "$(clojure -Spath)" --copy-configs --skip-lint

Imported config to .clj-kondo/tvaisanen/export-types-sample. 
To activate, add "tvaisanen/export-types-sample" to 
:config-paths in .clj-kondo/config.edn.

Then update the .clj-kondo/config.edn (you might need to create the file) as instructed by clj-kondo output.

{:config-paths ["tvaisanen/export-types-sample"]}

And I am then populating the cache with.

$ clj-kondo --lint $(clojure -Spath) --dependencies --parallel

After this, we should be good to jump into the editor and see if everything worked as expected.

Benefits of Types in the Editor

I created a core namespace for the new project where I've required the library I set up. You can see that when add-points is called with an invalid sample/Point it tells us that we are missing a required key :y. Without the extra type config, we should have an issue with "duplicate key :x", so it looks like the lib configs trump the default linting issues in priority.

As we extend the code, we can see that add-points return a value that the render-point is okay with according to clj-kondo, but if we pass a map that is not a sample/Point we get another linting error on the screen.

These red squiggly lines can save you a lot of headaches by letting you know where the types don't match the expected value. But this is just the first benefit that comes from using the types. So far, we haven't looked into how Malli expands on this.

Benefits of Types in the REPL

If a library has Malli schema function definitions, we can start instrumentation (monitoring with what values the functions are called and what they return) and get run-time type checking. In practice, this means we get a report on "invalid function call" in the REPL every time a function is called with unexpected arguments.

Let me demonstrate this.

We can use this to our advantage since we created the library with the malli schemas instead of writing the type hints manually into the clj-kondo config. Let's require malli.dev in our namespace and run dev/start! as we did when we first exported the types. Now, try running the sample/render-point with an invalid point argument.

When we evaluate line 19 while the malli instrumentation is active, we can find something new in the REPL, a schema error nicely documenting what is happening. The REPL provides helpful information about what went wrong with the function call.

Of course, we must remember that if there are a lot of functions being called, having the instrumentation on all the time can have a performance penalty.

Just keep in mind when running dev/start! the types are created into .clj-kondo/metosin/malli-types-clj/config.edn and when the dev/stop! is run. These types will be cleaned up.

Benefit of Types in the Terminal / CI

Last but least, we can also use the generated types outside our editor by using clj-kondo from the command line.

❯ clj-kondo --lint "src"
src/core.clj:6:2: error: Missing required key: :y
src/core.clj:6:8: error: duplicate key :x
src/core.clj:13:22: error: Missing required key: :x
src/core.clj:13:22: error: Missing required key: :y
linting took 5ms, errors: 4, warnings: 0

Adding this step can help keep many unnecessary bugs out of production when added to the continuous integration checks.

Where to Contribute

If you find that your particular use case is not supported for some reason. Providing patches is encouraged. Malli defines the malli->clj-kondo transformations in malli.clj-kondo namespace, which is where you want to look if you want to learn how the transformation process works or if you're going to extend the functionality. On the clj-kondo side, read the status of types and this source code file listing the available type mappings.

Conclusion

That was probably a lot to digest in one go. It took me some time to understand the relationship between clj-kondo and Malli, but slowly and surely, it's making more sense. I wanted to write about this topic because I think if more people knew how to add the types to their projects, it would improve the developer experience across the whole ecosystem (at least, this is what I'd like to imagine).

This is somewhat what happened with Javascript and Typescript. Little by little, the majority of the libs started offering type definitions. I don't know about you, but I'd appreciate a little extra help from my editor in debugging the expected types for a given function and having the change occasionally to turn on the Malli instrumentation to provide some run-time info on what's going wrong.

Some tools can provide the observability to run-time values like Flowstrom or just good old clojure/tools.trace. However, these tools do not tell me what type of data the original author had intended to receive.

Once again, I hope you found this helpful. Feel free to reach out and let me know what you think—social links in the menu.

Thank you for reading.

I prefer configuring deployments in declarative Terraform files over clicking through dashboards. This helps me to get back on track when I return to my abandoned side projects.

It's worth mentioning that I will be using a local terraform state, although I don't recommend it. Lately, I have been using Terraform Cloud for my projects. If you want to learn how to set up a Github CI with Terraform Cloud for this project, please let me know in the comments.

Most platforms like AWS, GCP, Azure, Linode, Fly, and Digitalocean support Terraform. The big three, AWS, GCP, and Azure, feel excessive for hobby projects. Additionally, managing virtual servers or utilizing Kubernetes is not something I want to deal with, so that crosses Linode off the list, leaving me with Fly and Digitalocean as the options. While I haven't tested Fly yet, DigitalOcean's app platform has become my go-to choice out of habit and because the development experience feels suitable.

Requirements

We need Terraform, Doctl, Docker, and Docker Compose installed on our development machine and a DigitalOcean account with an API token with write permissions that we can use to authenticate ourselves from the command line.

You should be able to follow up on the examples without prior knowledge of Terraform. Still, if you're interested in learning more, the official website is an excellent place to start.

Application

To get started, create the following folder structure for the project. Clojure-related code and required Docker files are in the api folder, where docker-compose.yml is for running the application locally with a Postgres database, and the DigitalOcean-related Terraform code will be in the terraform folder.

project-root
├── api
│   ├── deps.edn
│   ├── docker-compose.yml
│   ├── Dockerfile
│   └── src
│       └── main.clj
└── terraform
    ├── main.tf
    ├── output.tf
    └── setup.tf

Clojure

First, let's start by defining the Clojure dependencies for the application in the api/deps.edn. We need ring dependencies for serving the API and next.jdbc and postgresql for the database access.

{:paths ["src"]

 :deps {org.clojure/clojure {:mvn/version "1.11.0"}
        ring/ring-core {:mvn/version "1.6.3"}
        ring/ring-jetty-adapter {:mvn/version "1.6.3"}
        com.github.seancorfield/next.jdbc {:mvn/version "1.2.659"}
        org.postgresql/postgresql {:mvn/version "42.2.10"}}

 :aliases {:run {:main-opts ["-m" "main"]
                 :exec-fn   main/run!}}}

Next, create the code to serve the API, connect to the database, and make a database query in api/src/main.clj.

(ns main
  (:require [ring.adapter.jetty :as jetty]
            [next.jdbc :as jdbc])
  (:gen-class))

(def port (Integer/parseInt (System/getenv "PORT")))

(def db {:dbtype "postgres"
         :jdbcUrl (System/getenv "JDBC_DATABASE_URL")})

(def ds (jdbc/get-datasource db))

(defn app [_request]
  (let [db-version (jdbc/execute! ds ["SELECT version()"])]
    {:status  200
     :headers {"Content-Type" "application/edn"}
     :body    (str db-version)}))

(defn run! [& _args]
  (jetty/run-jetty #'app {:port port}))

This is all of the application code. The next step is to create a Dockerfile and set up the local development environment to validate that the database connection and the API are working as expected when provided with the expected environment variables.

Dockerfile

Let's keep the api/Dockerfile as simple as possible for now.

FROM clojure:openjdk-17-tools-deps-buster

WORKDIR app
COPY . .

RUN clojure -P

CMD clj -X:run

In general it's better to create the Dockerfile in stages to avoid unnecessary build delay and minimize the image size. If you're interested in learning more about that let me know in the comments. You can also read more Docker: Multi-Stage Builds for further information.

Finally, let's set up a api/docker-compose.yml file for mimicking our deployment environment to test that the API is working as expected.

version: '3.1'

services:

  api:
    build: .
    environment:
      JDBC_DATABASE_URL: "jdbc:postgresql://postgres:5432/db?user=user&password=password"
      PORT: 8000
    ports:
      - 8000:8000

  postgres:
    image: postgres
    environment:
      POSTGRES_DB: db
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    ports:
      - 5432:5432

Validate Application Code

Start the API and the database with docker-compose up and HTTP GET localhost:8000.

❯ http localhost:8000
HTTP/1.1 200 OK
Content-Type: application/edn
Date: Mon, 30 Oct 2023 16:50:53 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

[{:version "PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1) 
            on x86_64-pc-linux-gnu, compiled by gcc 
            (Debian 10.2.1-6) 10.2.1 20210110, 64-bit"}]

If you did get something similar to the above, we are good to continue forward. The next step is to set up the application to run in DigitalOcean.

In the case you're wondering what's the http command. It is httpie.

DigitalOcean and Terraform

The first thing we must do is to set up the DigitalOcean Terraform provider. This is analogous to adding a Clojure dependency in a deps.edn file. Open terraform/setup.tf and add the following content to it.

terraform {
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}

provider "digitalocean" {}

When a new provider is added, Terraform needs to be explicitly initialized with terraform init. After running the command, you should see the following message with additional output.

Terraform has been successfully initialized!

So far, we have downloaded the provider but have not yet configured the authentication. We'll get there in a moment.

Project

Now it's time to set up the project to which we'll assign all our resources. Add the following content to terraform/main.tf and feel free to name your project however you wish.

resource "digitalocean_project" "project" {
  name        = "my-sample-project"
  description = "description"
  environment = "development"
  purpose     = "template-app"
}

If we try to apply the changes before configuring the API token, we should see something like this:

❯ terraform apply

...

digitalocean_project.project: Creating...
╷
│ Error: Error creating Project: POST https://api.digitalocean.com/v2/projects: 
|        401 (request "f2774cfc-66fc-4f34-98d9-0935f9dcd33d") 
|        Unable to authenticate you with digitalocean_project.project,
│        on main.tf line 1, in resource "digitalocean_project" "project":
│        1: resource "digitalocean_project" "project" {
│
╵

To apply the changes, we must authenticate ourselves with a token. There are a couple of different options on how to do this.

token - (Required) This is the DO API token. Alternatively, this can also be specified using environment variables ordered by precedence:

• DIGITALOCEAN_TOKEN

• DIGITALOCEAN_ACCESS_TOKEN

DigitalOcean Terraform Provider

I'll do this step by exporting my API token in an environment variable in the terminal.

❯ export DIGITALOCEAN_TOKEN=${YOUR_API_TOKEN}

After this, we should be able to create the project by running terraform apply.

❯ terraform apply

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # digitalocean_project.project will be created
  + resource "digitalocean_project" "project" {
      + created_at  = (known after apply)
      + description = "description"
      + environment = "development"
      + id          = (known after apply)
      + is_default  = false
      + name        = "my-sample-project"
      + owner_id    = (known after apply)
      + owner_uuid  = (known after apply)
      + purpose     = "template-app"
      + resources   = (known after apply)
      + updated_at  = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

digitalocean_project.project: Creating...
digitalocean_project.project: Creation complete after 1s [id=22276173-77de-4e41-ab47-fae4a86ec414]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

And confirm that the project was created by logging in to the DigitalOcean dashboard.

Container Registry

The next step is to set up the container registry to publish the application image. Add the following to terraform/main.tf.

 resource "digitalocean_container_registry" "app_registry" {
  name                   = "clojure-sample-app"
  subscription_tier_slug = "starter"
}

After applying the configuration, confirm that the registry was created. Let's do it with the doctl command line tool this time around.

❯ doctl registry get clojure-sample-app
Name                  Endpoint                                        Region slug
clojure-sample-app    registry.digitalocean.com/clojure-sample-app    blr1

Here, we need to pick up the endpoint value clojure-sample-app and use that to tag the application image. Add the build section to the api/docker-compose.yml file and run docker-compose build.

services:
  api:
    build:
      context: .
      tags:
        - "registry.digitalocean.com/clojure-sample-app/dev"
    environment:
      JDBC_DATABASE_URL: "jdbc:postgresql://postgres:5432/db?user=user&password=password"
      PORT: 8000
    ports:
      - 8000:8000

After building, confirm that the image exists with the expected name.

❯ docker image ls | grep clojure-sample-app
registry.digitalocean.com/clojure-sample-app/dev   latest 20266c1dfcda   12 seconds ago   715MB

It looks like the build produced the expected outcome. Next, we must log in to the DigitalOcean container registry to publish the Docker image.

❯ doctl registry login
Logging Docker in to registry.digitalocean.com

And then push the image to the registry.

❯ docker push registry.digitalocean.com/clojure-sample-app/dev
...
latest: digest: sha256:d8a5c01cd95ab5c0981c454866ec52203feba529f169e2da93cb6a99f3af5d88 size: 2840

Let's confirm with doctl that the image is available in DigitalOcean.

❯ doctl registry repository list-v2
Name    Latest Manifest                                                            Latest Tag    Tag Count    Manifest Count    Updated At
dev     sha256:d8a5c01cd95ab5c0981c454866ec52203feba529f169e2da93cb6a99f3af5d88    latest        1            1                 2023-11-01 16:07:59 +0000 UTC

At this point, we have created everything required for the app deployment. Let's do that next.

Application

Create a digitalocean_app resource and update the digitalocean_project from before by adding the app to the resources. This way, DigitalOcean can group the resources (in this case, just the app) under the project.

Let's use the least expensive configuration for the dev environment by utilizing a basic-xss instance with a development database. You can modify these based on your needs. See the rest of the available configuration options from the docs.

resource "digitalocean_project" "project" {
  ...
  # Add this line
  resources   = [digitalocean_app.app.urn]
}

resource "digitalocean_app" "app" {
  spec {
    name   = "sample-app"
    region = "ams"

    alert {
      rule = "DEPLOYMENT_FAILED"
    }

    service {
      name               = "api"
      instance_count     = 1
      instance_size_slug = "basic-xxs"

      image {
        registry_type = "DOCR"
        repository    = "dev"
        tag           = "latest"
        deploy_on_push {
          enabled = true
        }
      }

      env {
        key   = "JDBC_DATABASE_URL"
        value = "$${starter-db.JDBC_DATABASE_URL}"
      }

      source_dir = "api/"
      http_port  = 8000

      run_command = "clj -X:run"
    }

    database {
      name       = "starter-db"
      engine     = "PG"
      production = false
    }
  }
}

You probably noticed that we don't have a PORT variable for the application. This is because, in this case, we had configured it. Digitalocean would override it with the http_port variable.

The image block of the configuration ties the container registry we created earlier to the application. After, whenever we push a new image with the dev tag, the application will be redeployed.

Validate

Finally, it's time to check if we did everything correctly. Let's fetch the live URL for our new application. We can do this with doctl apps list.

❯ doctl apps list -o json | jq ".[0].live_url"
"https://sample-app-xeoo8.ondigitalocean.app"

However, I prefer using the resource available attributes via Terraform outputs. These are usually declared in the output.tf file, but I'll skip this step for the example and leave it in terraform/main.tf. Add the following to the file and run terraform apply once more.

output "app_url" {
  value = digitalocean_app.app.live_url
}

Afterwards, the outputs can be viewed with terraform output

❯ terraform output
app_url = "https://sample-app-xeoo8.ondigitalocean.app"

Now, we are ready for the last step. Validate that we get the Postgres version as a result, as we did with the local setup.

❯ http https://sample-app-xeoo8.ondigitalocean.app

HTTP/1.1 200 OK
CF-Cache-Status: MISS
CF-RAY: 81f5801c8898d92e-HEL
Connection: keep-alive
Content-Encoding: br
Content-Type: application/edn
Date: Wed, 01 Nov 2023 16:26:37 GMT
Last-Modified: Wed, 01 Nov 2023 16:26:36 GMT
Server: cloudflare
Set-Cookie: __cf_bm=SHREkz9ddLOs1EoTzuNj7DSPYfazEy_bpdi7y_Kb9BE-1698855996-0-AVgy3TNvUDITYonrci23K+dW1BHozbNCLeNX0FMHMpI3miRJBq8I4HlQE5nMA5YjoZ4dEXLatcef+AE+gjq4xeQ=; path=/; expires=Wed, 01-Nov-23 16:56:36 GMT; domain=.ondigitalocean.app; HttpOnly; Secure; SameSite=None
Transfer-Encoding: chunked
Vary: Accept-Encoding
cache-control: private
x-do-app-origin: 35eedf3b-e0a9-4376-863e-c5ba59ef5d6a
x-do-orig-status: 200

[{:version "PostgreSQL 12.16 on x86_64-pc-linux-gnu, 
            compiled by gcc, a 47816671df p 0b9793bd75, 64-bit"}]

Looks like everything is working as expected.

Destroy

If you don't want to keep the application running, remember to destroy all of the created resources with terraform destroy. The expected monthly cost with this setup with the basic-xxs instance size and development database is around $12 per month, but the starter subscription tier of DOCR is free of charge.

You can read more about the pricing models from the DigitalOcean docs: App platform, DOCR pricing.

Conclusion

I think that DigitalOcean is an excellent lightweight alternative for smaller projects. It does have all the expected features for general app development, and it can be extended with the add-ons found in the marketplace. I have not yet used it for long-term projects since my side projects don't last long enough, but so far, I've been happy with it from the tinkerer's perspective. I would rather not spend my leisure coding time dealing with infrastructure.

Once again, thanks for reading, and I hope you found this helpful. Here's the accompanying GitHub repository.

I figured there must be an easy way to see the translation result, and I asked around what's the default way to do this in the REPL. As far as I know, I might as well have been the first one asking about this since there was no quick answer available (usually there is).

Since Clojurescript is compiled into Javascript, I thought the most straightforward way to do this would be to use the plain JS interop.

Plain JS Interop

The toString() method of Function instances returns a string representing the source code of this function.

MDN Docs

Knowing that toString method prints out the source code for functions. We can view the compiled source by simply calling the .toString. In Javascript, it'd be as simple as:

>> function piece_of_code(){ 
>>  console.log("hello")
>> };
undefined
>> piece_of_code.toString()
'function piece_of_code(){ 
  console.log("hello")
}'

The same approach works in the CLJS context, and we can make the result read better by just printing out the value.

(.toString piece-of-code)
"function cljs$user$piece_of_code(){\nreturn console.log(\"hello\");\n}"

(defn print-src [f]
  (println (str f)))

(print-src piece-of-code)
;; => 
;; function app$core$piece_of_code(){
;;   return console.log("hello");
;; }

CLJS Compiler Option

But after some digging, I found the fantastic video: "Mike Fikes explains the ClojureScript Compiler" and learned that a dynamic CLJS compiler variable *print-fn-bodies* does the same thing without the extra work. Later, I discovered that Mike also has a blog post explaining how to do this.

(set! *print-fn-bodies* true)

(do piece-of-code)
;; => #object[app$core$piece_of_code "function app$core$piece_of_code(){
;;   return console.log("hello");
;; }"]

With this approach, it is enough to set the variable, and you're good to go.

Conclusion

I think it is good to know how to do this when you're working with CLJS. You might not need to drop into the JS level that often, but knowing how to do it quickly without using external tools makes all the difference in the dev flow, and even if you don't need to, it's good to know how to do it just for curiosity's sake.

Ultimately, the code we are dealing with is just JavaScript, and we can use the same functions we'd be using on the JS development. On top of that, if you're working in a browser context, it's also possible to view and interact with the compiled code in the browser console.

Thanks for reading. I hope you found this helpful. I recommend taking a look at Mike's blog to learn more about Clojurescript.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8000/. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.

If you're dealing with the above error with your Clojure web app you've come to the right place. This is a common problem when fetching data from a different origin (read URL) for single-page web applications (SPA). A common scenario is that an SPA is served from AWS S3 and it is fetching data from different servers.

How to Fix "The Same Origin Policy disallows reading the remote resource"

A common fix is to add a backend middleware to deal with the CORS headers such as ring-cors and this is what this post is about. Alternatively, you can serve the web app from the API's origin if possible or write your logic (i.e. middleware) to handle the CORS requests or if you're dealing with a third-party API there is likely an option to configure your allowed web origins.

Configure CORS Middleware

First, add the latest version of ring-cors to your dependencies.

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        ring/ring-core {:mvn/version "1.6.3"}
        ring/ring-jetty-adapter {:mvn/version "1.6.3"}
        ring-cors/ring-cors {:mvn/version "0.1.13"}}}

Require the dependency and wrap the application handler with ring.middleware.cors/wrap.cors.

(ns tvaisanen.cors
  (:require [ring.middleware.cors :refer [wrap-cors]]
            [ring.adapter.jetty :as jetty]))

(defn app [_request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "OK"})

(defonce server (atom nil))

(defn start! []
  (reset! server
          (jetty/run-jetty
           (wrap-cors #'app
                      :access-control-allow-origin #"https://tvaisanen.com"
                      :access-control-allow-methods [:get :put :post :delete])
           {:port 8000 :join? false})))

(comment
  (.stop @server)
  (start!))

Let's run the server and validate that it is working as expected.

Verify the Configuration

Verify the configuration by making an HTTP request from the browser console and inspecting the headers from the network tab.

Send the Request

Type this in your browser's dev tools console (from the origin configured for CORS).

fetch("http://localhost:8000").then(console.log)

Request Headers

You should see something similar to this in your request headers when you inspect the network tab.

GET / HTTP/1.1
Host: localhost:8000
Origin: https://tvaisanen.com
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site

Response Headers

And the response headers are similar to this.

HTTP/1.1 200 OK
Date: Thu, 19 Oct 2023 05:53:16 GMT
Content-Type: text/html
Access-Control-Allow-Methods: DELETE, GET, POST, PUT
Access-Control-Allow-Origin: https://tvaisanen.com
Transfer-Encoding: chunked
Server: Jetty(9.2.21.v20170120)

What is important to notice here is that the response header Access-Control-Allow-Origin has the value from the requests Origin header. Having this with the Access-Control-Allow-Methods header tells the browser that it is okay to use the data they are asking for.

Let's double-check that this is indeed what is happening from the command line.

CORS Headers Included

When the origin header is passed and the value matches the one that is configured on the server side we get the expected Access-Control-Headers.

❯ http localhost:8000 'origin:https://tvaisanen.com'

HTTP/1.1 200 OK
Access-Control-Allow-Methods: DELETE, GET, POST, PUT
Access-Control-Allow-Origin: https://tvaisanen.com
Content-Type: text/html
Date: Thu, 19 Oct 2023 06:02:10 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

OK

CORS Headers Missing

If the value of the origin header does not match the configured value the access control headers should not be in the response.

❯ http localhost:8000 'origin:https://not-tvaisanen.com'

HTTP/1.1 200 OK
Content-Type: text/html
Date: Thu, 19 Oct 2023 06:02:18 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

OK

Looks like everything is working as expected!

There's more to cross-origin resource sharing but this should be enough to get you over the initial problem. I recommend reading related MDN Docs to learn more about the topic.

• Cross-Origin Resource Sharing (CORS)

• CORS Errors

Thanks again for reading, I hope you found this useful.