Ray McDermott - Seeking the Open Grail

REPtiLe - coming out of its shell

Fri, 28 Sep 2018 00:00:00 +0000

This is the moment on Apropos that I saw REPtiLe become something useful.

I saw that a shared REPL in the browser was something useful: four other developers were using it to code and collaborate in real time.

It still has plenty of UX issues and its fair share of bugs and so is not quite ready for prime time.

Github

You can check out the GitHub projects at:

Server - Tail - Minus the boiler plate to set up the HTTP and WS, it’s ~100 lines of code.

UI - Tongue - The UI is bigger ~500 lines of code.

I’m not overly concerned with LOC measures, this is just to say that the project is still quite small.

Twitch

Real time media takes a lot less time to produce and manage on my part. With that in mind, I have created a channel on Twitch. I will use that channel to share the fixes I do to the UX and the code in general.

And here is a real reptile coming out of its shell - cute innit?

Numbers as text

Mon, 17 Apr 2017 00:00:00 +0000

Random Beyonce based headlines, as if you care

I was chatting to a friend about job interviews and he mentioned that he always gives a small challenge to the candidates that they can do at home.

The goal is to spell out numbers as sentences, similar to how text might appear on a bankers cheque (or check if you’re in the US and - bonus - it’s still relevant for you).

Here is the behaviour that the program should provide as a minimum:

(num->text 123456789)
=> "One hundred and twenty three million four hundred and fifty six thousand seven hundred and eighty nine"

The challenge is meant to be relatively simple and serve as a worked example to centre discussions around design and development approaches.

I figured that addressing this problem with Clojure could make an interesting blog post for the same reasons. The code is on Github

This post is enhanced using KLIPSE live code snippets meaning you can edit the code and see immediate results

All the single ladies

Hard code the single numbers you say. No way. Hard code the names at least you say. Again no way. Beyonce would not approve.

Let use key words, sequences and combinations to get straight into it the good FP stuff right from the start…

(def x1 [:one :two :three :four :five :six :seven :eight :nine
         :ten :eleven :twelve :thirteen :fourteen :fifteen :sixteen :seventeen :eighteen :nineteen])
(def singles (zipmap (range 1 20) x1))
; simple interactive test, try it yourself...
(get singles 7)

Clojure make it easy to generating numbers (range) and :key-words are easy to test.

zipmap is a simple way to create a map from two sequences - using the first as keys and the second as values.

So we have the computer doing it’s share of the work already. Neat!

Now put your hands up

Next we deal with the other fixed numbers >= 20 and < 100


(def x10 [:twenty :thirty :forty :fifty :sixty :seventy :eighty :ninety])
(def tens (zipmap (range 20 99 10) x10))
(get tens (rand-nth (keys tens)))

Here we use another form of the range function again to produce a range that starts at 20, ends at 99 and is stepped by 10.

The interactive sample introduces a little randomisation for the LOLs and we will use more of this as we go along.

The refresh on the randomisation is a little magic from Klipse that I use to set a loop every 2 seconds in the HTML mark up.

I’m doing my own little thing

My solution is composed from three logical parts: numbers to 100, numbers between 100 and 1000 and numbers above 1000.


(defn nums-lt-100
  [num]
  {:pre [(pos? num)]}
  (if-let [answer (or (get singles num)
                      (get tens num))]
    [answer]
    (let [ten-part  (quot num 10)
          ten-whole (* 10 ten-part)]
      [(get tens ten-whole) (nums-lt-100 (- num ten-whole))])))

; Generate a random sample of numbers to test it out
(map (fn [n] [n (nums-lt-100 n)])
     (take 2 (random-sample 0.05 (range 1 100))))

You can tweak the above code to adjust the number of samples (take N). By the way, I only take a small sample so that the page doesn’t reflow every two seconds ;-)

The first part of the code (if-let) tests if the answer is present in the singles or tens collection - there is no need to take the risk of testing for a sentinel value. Such values can easily get out of sync with the size of the vector or require maintenance. Also, because it’s a map no need to test for length - just look it up and find it or move along.

If we don’t find it, we take the modulus and look up the answer in one or both sets.

Finally, we use a some Clojure syntax sugar to coerce results into a vector by surrounding the responses with square brackets [ ]

Building the answer as a sequence is a huge simplification compared to the other solutions I have seen out there. This will become more evident as the solution is further composed.

You decided to dip and now you wanna trip

The second part deals with the hundreds…


(defn nums-gt-100-lt-1000
  [num]
  {:pre [(>= num 100) (< num 1000)]}
  (let [hundreds  (quot num 100)
        remainder (- num (* hundreds 100))
        result    [(get singles hundreds) :hundred
                   (if-not (zero? remainder)
                     (nums-lt-100 remainder))]]
    (remove nil? result)))

(map #(nums-gt-100-lt-1000 %)
     (take 3 (filter odd? (random-sample 0.5 (range 921 1000)))))

The main function does some trivial maths to separate out the hundreds and its remainder. The hundreds component is added to the vector and then we use the earlier function on any remainder.

There is a small annoyance to my eye with this solution … I have to use (quot num 100) rather than (/ num 100) as the default answer in Clojure is a rational number. You can coerce it an integer but I’m not a fan of using such type trickery unless it’s absolutely unavoidable.

I got gloss on my lips, a man on my hips

And now I’m ready to deal with the big boys: thousands, millions and so on… to quattuordecillion. We can go higher with a longer list but this makes the samples too awkward. [You can find a longer list up to millinillion on the GitHub repo.]

This is the third part of the solution and is also based on sequences of generated numbers.

(defn units [start step count]
  (reduce (fn [a b] (conj a (* (last a) step)))
          [start] (range count)))

(units 1 2 4)

Got me tighter than my Dereon jeans

The units function provides a way to generate lists of numbers at certain boundaries and we can will use that to establish the various ‘large number’ magnitudes.


(def large-numbers-text [:thousand :million :billion :trillion
                         :quadrillion :quintillion :sextillion
                         :septillion :octillion :nonillion
                         :decillion :undecillion :duodecillion
                         :tredecillion :quattuordecillion])

(def unitable (units 1000N 1000N (dec (count large-numbers-text))))

(def large-number-map (zipmap unitable large-numbers-text))
(def inverse-large-numbers-map (into {} (map (fn [[a b]] [b a])) large-number-map))

(def unit-boundaries (map (fn [u] [u (* u 1000N)]) unitable))

(let [rand (rand-nth (keys large-number-map))]
  [rand (get large-number-map rand)])

We use the same zipmap pattern for large number texts as before…

In addition to the previous pattern, we also generate an inverse look up map that we will need later.

The interesting part of this approach is that we generate all of the boundary conditions for all supported units.

Acting up, drink in my cup

(defn which-bounds? [num unit-bounds]
  (remove nil? (map (fn [[lower upper]]
                      (if (and (>= num lower)
                               (< num upper))
                        lower)) unit-bounds)))

(defn which-unit? [num]
  (let [unit (which-bounds? num unit-boundaries)]
    (get large-number-map (first unit))))

(def various (units 1000 7 7))

(map #(which-bounds? % unit-boundaries) various)

The code to test for the boundaries, and to map it into the correct unit, is a straightforward mapping check over the boundaries.

We use the units function to generate a sequence to test it out. You can edit the code above to see the test data and validate it.

I can care less what you think

With this in place we can present the final form that composes it together.

(defn inject-and
  [num-vec]
  (if (<= (count num-vec) 2)
    num-vec
    (let [check-map (assoc inverse-large-numbers-map :hundred 100)]
      (cond
        (= :and (last (drop-last 1 num-vec))) num-vec
        (get check-map (last num-vec)) num-vec
        (get check-map (last (drop-last 1 num-vec)))
            (concat (drop-last 1 num-vec) [:and] (take-last 1 num-vec))
        (get check-map (last (drop-last 2 num-vec)))
            (concat (drop-last 2 num-vec) [:and] (take-last 2 num-vec))
        :else num-vec))))

(defn num-representation
  [num]
  (inject-and
    (flatten
      (cond
        (< num 100) (nums-lt-100 num)
        (< num 1000) (nums-gt-100-lt-1000 num)
        :else (let [unit           (which-unit? num)
                    divisor        (unit inverse-large-numbers-map)
                    first-part     (quot num divisor)
                    remainder      (- num (* divisor first-part))
                    representation (if (zero? remainder)
                                     [(num-representation first-part) unit]
                                     [(num-representation first-part)
                                      unit
                                      (num-representation remainder)])]
                representation)))))

(num-representation 100101)

inject-and finds the right spot in the vector to inject the and. Since the call site is recursive, it applies properly across all units. Such baroqueness is not needed in the US version so this function could be easily removed.

The trick in the num-representation function is to always have the larger numbers feed back through the function so that each part is processed - from left to right - into the appropriate single, ten or 100 multiplier.

I need no permission, did I mention?

There is one final part at the edge to morph the data into strings…

(defn num-vec->text
  [num-vec]
  (clojure.string/capitalize
    (apply str (interpose " " (map name num-vec)))))

(defn num->text
  [num]
  (num-vec->text (num-representation num)))

(num->text 56789)

If you liked it, then you should have put a ring on it

That’s just the beginning though. This problem is one where the range of answers is so huge that we cannot possibly test all of them. Moreover tests on strings presume a human checker.

So we can do better than that. Next time I want to go into using spec on this problem but I will just finish with a few data samples for tests that I have concocted. You can see more on the GitHub repo.

Oh, oh, oh

CONSISTENCY check: Do all of the first numbers, across all units start with :one ?

(let [sample (units 1000N 1000N (dec (count large-numbers-text)))]
  (distinct (map #(first (num-representation %)) sample)))

Oh, oh, oh, oh, oh, oh

CONSISTENCY check: Is the answer for all of the first numbers, across all units, two words ?

(let [sample (units 1000N 1000N (dec (count large-numbers-text)))]
  (distinct(map #(count (num-representation %)) sample)))

Oh, oh, oh

What does it mean?

Stay with data - it’s easier to work with and to test
Layer on infra - to enable testing on different layers
Use the latest possible transforms at the edge of the system
Use randomness early and often in your feedback loop
Use properties and constraints to properly test your system

Developing with data, sequences and composing functions to manipulate them is how we will finally get to leave the sand pit.

Oh, oh, oh

US readers please note: in the US the grammar for expressing these numbers is different and simpler: there’s no ‘and’.

Providing this facility for all cultures and languages is not trivial and it reminds us that even such a small and simple example like this - like most any thing - is most often a subset of a much more complex domain.

‘Cause if you liked it, then you should have put a ring on it

Disclaimer: Me != maths

Please let me know if you are a maths whizz and know a simpler way to solve this.

Also, if you have another functional way, in Clojure, JS or Haskell or Scala or Java 8.

This has been fun but if there is a ‘right way’, I would be happy to learn it!

There is a right way …

Ha ha … and to prove this point Alex Miller (aka @puredanger) informed me on Twitter that Clojure has something very close to this already - and soooooo much more - in the form of cl-format. This is an implementation of output formats from Common Lisp. Who knew? Wow - it’s astounding and it was honestly worth this effort to discover that it exists! This is a whole new rabbit hole people!


(clojure.pprint/cl-format true "~r" 100101)

Nil points

Tue, 21 Feb 2017 17:00:00 +0000

_{Eurovision memories from back in the day} _{(image source bbc.co.uk)}

Visualising Clojure

I recently visited a splendid blog by Joseph Wilk that explains Clojure sequence functions by visualising their effects.

After being impressed by the visuals I was struck by a couple of the functions (nthnext and nthrest) that are shown to have the same effect on a non-empty sequence.

This post is supported by live code snippets and that you can live edit too:

(nthnext [0 1 2 3] 2)

(nthrest [0 1 2 3] 2)

I was not familiar with them so I started digging a bit deeper. They are based, possibly obviously, on next and rest. The only difference between next and rest (and their derivatives) is their return value when the sequence is nil or empty.

(let [nil-rest-gives (rest nil)
      empty-rest-gives (rest [])]
  [nil-rest-gives empty-rest-gives])

(let [nil-next-gives (next nil)
      empty-next-gives (next [])]
  [nil-next-gives empty-next-gives])

Seq and ye shall find

In turn the docs say that both rest and next are based on seq.

Looking at the code for next and rest we see how this is achieved: they are wrapped in method calls to native code.

Here are the Clojure definitions, stripped of most metadata for brevity:

(def
 ^{:doc "Returns a seq of the items after the first. Calls seq on its
  argument.  If there are no more items, returns nil."}  
 next (fn ^:static next [x] (. clojure.lang.RT (next x))))

(def
 ^{:doc "Returns a possibly empty seq of the items after the first.
 Calls seq on its argument."}  
 rest (fn ^:static rest [x] (. clojure.lang.RT (more x))))

Swan dive

Diving deeper, next and more are written in Java, so let’s sneak a view behind the curtain:

public static ISeq next(Object x) {
  if(x instanceof ISeq) {
    return ((ISeq)x).next();
  } else {
    ISeq seq = seq(x);
    return seq == null? null : seq.next();
  }
}

public static ISeq more(Object x) {
  if(x instanceof ISeq) {
    return ((ISeq)x).more();
  } else {
    ISeq seq = seq(x);
    return (ISeq)(seq == null? PersistentList.EMPTY : seq.more());
  }
}

The significant difference is that when rest calls into more, it will return PersistentList.EMPTY if the call to seq is a Java null. The next implementation instead returns the null.

Well, that’s cool: having drilled right the way down we can see precisely why the behaviour differs.

There maybe reason

So what does it mean to Clojure programmer? Having a variant that returns nil might seem strange in the FP world where Tony Hoare’s nillion dollar mistake has been, ahem, fixed.

In other languages there are no more accidental nils once we have the monadic finery afforded by types like Option, Maybe, Coulda, Shoulda and Woulda. Yes, I did invent those last three to see if you’re still with me.

Functionil Streaming

We don’t have Maybe or Option (or other static) types in Clojure so let’s see how this works in practice.

Let’s start with a trivial - even silly - chain of map / filter over a sequence.

(map inc (nthnext (filter even? (range 200)) 90))

Now lets write a function that does not do well with nils

(defn next-nil-fail [x] (if (= 0 x) 0 (/ x x)))
; (next-nil-fail 10)
(next-nil-fail nil)

Now, let’s throw that function in a working chain and see what happens

(defn next-nil-fail [x] (if (= 0 x) 0 (/ x x)))
(map next-nil-fail (nthnext (filter even? (range 200)) 90))

Boring but OK, now let’s see what happens when we start throwing nils

(defn next-nil-fail [x] (if (= 0 x) 0 (/ x x)))
; filter returns 100 values so 101 > 100 and nthnext will return a nil
(map next-nil-fail (nthnext (filter even? (range 200)) 101))

Nothing will come of nothing

Say what? How in the Lear didn’t it generate an error?

(defn next-nil-fail [x] (if (= 0 x) 0 (/ x x)))
(map next-nil-fail [nil])

We can see that nil-fail will fail if it is passed a nil or a collection of nils.

So something funny going is on here! Let’s look at how map works:

; relevant code snippet from map
([f coll]
 (lazy-seq
  (when-let [s (seq coll)]
    (if (chunked-seq? s)
      (let [c (chunk-first s)
            size (int (count c))
            b (chunk-buffer size)]
        (dotimes [i size]
            (chunk-append b (f (.nth c i))))
        (chunk-cons (chunk b) (map f (chunk-rest s))))
      (cons (f (first s)) (map f (rest s)))))))

And now we see: the last line shows that the map implementation uses rest to avoid producing nils on the stream when the seq is empty or nil.

Nils desperandum

But map does not inspect the content of each element so cannot stop us from hitting all fail cases.

(defn next-nil-fail [x] (if (= 0 x) 0 (/ x x)))
(map next-nil-fail (interpose nil [1 2 3]))

[NB: This throws a null pointer exception for CLJ on the JVM but CLJS is more forgiving!)]

The idiomatic route is to fix up your code:

(defn div-nil-ok [x]
  (cond (nil? x) x
        (= 0 x) 0
        :else (/ x x))) ; it's just a toy, don't make me write all the cases!
(div-nil-ok nil)
(map next-nil-fail (interpose nil [1 2 3]))

Monadic puns

Is nil a seq in Clojure?

(seq? nil)

No, the implementation of map uses a monadic approach to ensure that nils are not propagated through the chain.

Other seq functions take the same approach to ensure that functions which generate nil instead of empty lists are treated equally.

This is also called nil-punning. Oh and one small niggle with that excellent article: Eric claims

first has nothing to return if the seq is empty, and so it returns nil. Because nil is a seq, first works on nil, and returns nil. rest works on nil as well, because nil is a seq.

Naughty Eric ;-) We see that nil is not a seq but instead, the operations on sequences work to provide that effect.

But but but next

Ah, yes in the case of next and its derivatives, the sequence functions make no such concessions.

So why would such functions exist?

Functionil Recursion

Nil can be used to good effect as a terminal condition when writing recursive programs.

(loop [nums (range 10)
       results []]
  (if nums
    (recur (nthnext nums 2) (conj results (first nums)))
    results))

Do not change nthnext for nthrest ;-)

Conclusion

Clojure has made working with nil pain free in general due to the design of the sequence operations.

There are however still some edge cases that need to be addressed where nils can be inserted into sequences.

There are also many options (or enough rope) for adventurous programmers to use them where they have value.

Follow up

I am reserving this space to explain in more detail why nil is treated differently between CLJ and CLJS.

Comments / links to help me are welcome!

Credits

This post was written using the power of KLIPSE to enable the interactive code blocks.

Expressivity: Imperative considered, harmful?

Thu, 15 Sep 2016 18:59:38 +0000

Expressivity in code

In the initial post I gave an example of how we see expressivity increase in maths as we progress from words to symbolic arithmetic to algebra.

I introduced a woolly model to test expressivity - does it enable us to say more in our programs and yet remain precise by being open to a transparent and predictable form of substitution?

I am going to look back at some of the imperative style code that I used to write back in the 90s and run it through the test. It’s a style that’s very much alive today so this is not a pure nostalgia trip.

Imperative

Taking C as a canonical example, the standard C function for copying strings is strcpy

char *strcpy(char *dest, const char *src)

dest - pointer to the destination array where the content is to be copied.
src - The string to be copied.
returns - a pointer to the destination string dest.

Expressive, not

This small example shows how confused the C library authors were about what was the correct thing to do even for the trivial task of copying strings.

There are some good things here though: firstly that the src string is immutable so strcpy cannot change it. Secondly it returns a pointer with the copied value.

Well that would have been nice, but the copied value is dest, so there is never any need to use that return value. It would have been more honest to write:

void strcpy(char *dest, const char *src) /* Honest */

And for my taste, if we have to do it this way, I would prefer src come first and dest second so that it reads more normally (at least for an English reader)

void strcpy(const char *src, char *dest) /* More 'obvious'? */

Playing it out in client code

Here is a simple example using the standard library method:

include <string.h>

int main() {
   char src[40];
   char dest[100];
   memset(dest, '\0', sizeof(dest));

   strcpy(src, "This is an example");
   strcpy(dest, src);
}

The programmer has to abide by the unwritten rules of strcpy, or maybe it’s better said, the conventions of C. Amongst these are:

ensure that the memory for dest array has been allocated
ensure that the size of the dest array is >= the source array, otherwise you will get less than you gave!
the memory must be freed and a proper program rather than this toy would have to do that too

Imperative code like this is awkward because it combines in detail what needs to be done and how exactly to do it.

Improved C

A tweak here and there and we can have simpler and more expressive function:

char *strcpy(const char *src) /* Not good C, but definitely more ergonomic */

include <string.h>

int main()
{
   char *dest = strcpy("This is an example"); /* Not valid C - why? */
}

It’s simpler because there are fewer names to grok and there is no confusion between the input and the output. These small gains in expressivity add up.

Or they would but this does not work in C for one simple reason: C does not have a garbage collector.

If strcpy allocated memory for the return value, there would be no way for strcpy to clean it up and the client code couldn’t either. So we would have a memory leak / hydrant. Not having a GC (or any form of automatic memory management) hobbles our range of expressivity by forcing the developer to allocate and free memory by hand. As they say these days, like an animal.

Modern GC implementations, like those on the Java Virtual Machine, are often as fast if not faster than hand tuned solutions and much less prone to error.

Functions do not a functional language make

Working with a function like strcpy is a good example why most of the C standard library is not functional in its design. In a functional language the goal is a pure function - an input takes parameters and returns a result without any side effects. The one side effect, that memory is allocated in both cases, is managed by the language in a functional environment.

Looping in C

Another area where imperative programming makes us work hard for a small reward is iteration. Here is a simple for loop in C

#include <stdio.h>
 
int main()
{
    const char nums[] = "01234"
    const int max = strlen(nums); /* strlen is a pure function */
    int i;
    for(i = 0; i < max; i++)
        printf("%d ", nums[i]);     // 0 1 2 3 4
}

One small simplifying thing here: the max and nums values are constant or immutable so there’s no way they will change during the loop. Optimising C compilers, and of course those in higher level languages too, can simply substitute the values directly where they are used.

That i however is going to be changing through the use of i++. FYI this is the C equivalent to i = i + 1.

High cost of small defects

If you look carefully at the C code you will notice a small defect. I’m not gonna call it a bug but it’s not 100% correct either. The last iteration of the for loop outputs an extraneous space. It’s not a biggie here but it really does require an additional piece of logic to obtain a more correct solution.

#include <stdio.h>

int main()
{
    const char nums[] = "01234"
    const int max = strlen(nums); /* strlen is good :) too */
    int i;
    for(i = 0; i < max; i++)
        if (i == (max - 1))
            printf("%d\n", nums[i]);
        else
            printf("%d ", nums[i]);
}

I reckon more than one developer, like me, did not immediately know which iteration of i is the correct i. The moment where the i++ operation take place is not obvious in the syntax. In fact, it is after the loop has executed but before the next test. So the i++ is effectively moved to be the final expression of the loop no matter which branch is taken. Or if you prefer, it is moved to be the expression before the test, except for the first time around the loop. And would it make a difference, if you wrote ++i?

To be honest, I have not ran this version of the code so please send a comment to correct the it if you know better and help me to make my point!

Hard work

This is hard work: we need two variables and a loop to print out the values of an array, with the numbers joined by spaces. In a more expressive language we would be able to state our intention and allow the machinery of the language implementation to relieve us from this book-keeping.

Like this perhaps:

(println (string/join " " "1234"))
1 2 3 4

Trust me, there is no trailing space :)

I’m going to go into more depth about getting rid of loops in the Functional Programming post but I think even the most die in the wool C coder would agree that this one liner is more expressive.

Is it harmful?

It makes us work harder and we can cut ourselves - or more importantly our users - more easily. For most common programming problems, yes I would consider C harmful.

The Swift language designers recently decided to remove support for the ++ and -- operators from of the language (v3 and onwards). They argue that it is a bad hangover from the C heritage because they are the source of too many errors :thumbsup:

Manual memory allocation is also a common form of security exploit, for example when it is not allocated or deallocated properly..

C has its historic place in device drivers and other low level system code where OO and functional programming have traditionally struggled to add value. That crown may be at risk however as we continue to see the rise of non-C system level programming languages such as Rust, Go and even JavaScript (see the 64Kb Kinoma) that are less imperative in style.

Next OO

The next post will show how OO - in Java at least - imposes limits on expression, by design.

Expressivity: Functional Programming - a New Hope

Sun, 11 Sep 2016 18:59:38 +0000

Expressivity in code

In the initial post I gave an example of how we see expressivity increase in maths as we progress from words to symbolic arithmetic to algebra

I introduced a woolly model to test expressivity - does it enable us to say more in our programs and yet remain precise by being open to a transparent and predictable form of substitution?

#FP

This is being lazily evaluated. The stream will respond.

Maybe.

Expressivity: Object-oriented composting

Sun, 11 Sep 2016 18:59:38 +0000

Expressivity in code

In the initial post I gave an example of how we see expressivity increase in maths as we progress from words to symbolic arithmetic to algebra

I introduced a woolly model to test expressivity - does it enable us to say more in our programs and yet remain precise by being open to a transparent and predictable form of substitution?

#Object oriented

Will be here soon after the Class hierarchy that we are working on is in its final state.

Expressivity ascending

Sun, 11 Sep 2016 18:59:38 +0000

Expressive

What does it mean to be more expressive? How can I convince you that one or other form of programming is more expressive than others?

Going back to first principles, consider expressivity in general. in English we can write:

One plus one equals two.

That works but it’s more expressive (to programmers at least!) in a mathematical form:

1 + 1 = 2

The mathematical form is more concise than the long hand so there is a win in productivity but that’s not what makes it more expressive for this context.

The increase in expressivity comes from the meaning behind the mathematical syntax. English grammar is open whereas the grammar of mathematical expressions is closed or at least constrained to some well defined semantics.

Parsing the mathematical expression is easier than the English version if you are familiar with the syntax and semantics but otherwise it’s just a pile of symbolic gibberish. The need to understand this extra level of symbology has a price - so it has to be worth paying. And not everybody, as we recall from school, wanted to pay it. Whenever we increase expressivity - be it in maths or poetry or music we lose some people along the way.

Algebra

And so, talking of losing people, let’s take the maths up a level (to about as far as I can safely go to be honest) and compare arithmetic against algebra

1 + 1 = 2  ; arithmetic

a + a = b  ; algebra

The algebraic form and the arithmetic forms are equally concise so we don’t gain anything in the effort required to write the expression. That feels like a productivity bummer.

In fact, we lose something of the directness of the arithmetic form when we compare it to the algebraic form. The algebraic form requires yet another set of semantics - a further abstraction - that sits over the semantics of arithmetic. So we lose some more people at this stage.

Plug and play

We have asserted a truth: in this expression a + a will always equal b.

Algebra buys us the ability to assert expressions and then play with them. By playing, I mean we have introduced the power of inference and deductive logic. For example we can deduce that a and b cannot be equal unless both are 0. Further we can assert that b is always greater than a, again unless they are 0. [ Did you notice how 0 is quite annoying - maybe it’s the mathematical equivalent of null … but that’s for another day)

Many more things can be deduced from this increase in abstraction. And any abstraction that gives us new ways to think is a genuine improvement in expressivity.

Finally, and more prosaically, we can plug in some numbers and play out the arithmetic.

What a change! Arithmetic, once upon a time so magical, is demoted to the role of servant to the algebra.

The Challenge

I want to finish with a woolly model to test expressivity - does it enable us to think in a materially different way and yet remain precise by being open to a transparent and predictable form of substitution?

This challenge is the basis on which we should judge functional programming.

There is a considerable body of software in the world, decent amounts of which appear to work. So do I say that we have been failed by other categories of programming languages? No, I don’t. Like arithmetic, other forms of programming have value. But like algebra, we should strive to use the most powerful form of expression we can harness.

Some will be able to fly to degrees of abstraction that I cannot follow and they will lose me along the way. But I can do the basics and am willing to learn more. If you read this far, I think you are too.

The associated posts will form a short, biased survey of facts and opinions that have been carefully curated chosen to support my arguments.

Imperative code considered, harmful?
Object-oriented composting
Functional programming as a New Hope

Event Stream Processing - Managing time series with core.async

Thu, 11 Feb 2016 18:59:38 +0000

Introduction

In my last post I went through the code for a data flow using a product stock level tracker by combining channels and maintaining state around the go-loop

In this post I want to show how to segregate data from a stream into time series.

In this use case we want to notify the supplier if we have a spike in demand. This spike example is fairly trivial but we will see that core.async can handle many millions of events per minute and provide fine grained time series with very little code.

Data first

As usual in this series we will outline the simple data we use in our model:

; Order
{:item-id A123 :description "Windscreen wiper" :quantity 2 :customer-id C234}

Timing - thinking out of the box

core.async does not provide any models for managing time in the library.

The addition of this support however is relatively simple and surprisingly concise. Well, maybe not that surprising by now.

The development of this model was prompted by a short conversation with @jgdavey in the core.async channel on Slack

If you were to implement yourself, you could make a go that simply pulls from a channel, and adds to another as a [timestamp item] pair, then finally pushes into an unbounded chan that has a transducer that filters based on age of that timestamp.

– Joshua Davey

I couldn’t get my head around the suggestion at first but I decided to give it a try - what could possibly go wrong? In the end mine was a different take on Joshua’s idea but it served as inspiration.

Oh, and Joshua has his own implementation which is interesting in its own right.

Timing orders

(defn gen-timed-orders [frequency-millis coll]
  (let [out (chan)]
    (go-loop []
      (do
        (>! out [(t/now) (rand-nth coll)])
        (<! (timeout frequency-millis))
        (recur)))
    out))

We saw this function in a previous example to generate infinite streams of data with time stamps so this satisfies the first part of generating pairs of data with a time / data tuple.

Timing windows

(defn create-time-series-windows
  "Create a window for X seconds that refresh every Y seconds. 1 <= Y <= X"
  ([open-duration slide-interval]
   {:pre [(> open-duration 0) (> slide-interval 0) (>= open-duration slide-interval)]}
   (let [out (chan)]
     (go-loop [start-time (t/now)]
       (let [window (gen-window start-time open-duration)]
         (do
           (>! out window)
           (<! (timeout (* 1000 slide-interval)))
           (if (= open-duration slide-interval)
             (recur (:to window))
             (recur (t/now))))))
     out))

  ([open-duration]
   {:pre [(> open-duration 0)]}
   (create-time-series-windows open-duration open-duration)))

(finite-printer (stop-after-n-seconds 4) (create-time-series-windows 2))
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x36b824c4
        "clojure.core.async.impl.channels.ManyToManyChannel@36b824c4"]
{:from
 #object[org.joda.time.DateTime 0x6b838f36 "2016-02-14T14:55:19.807Z"],
 :to
 #object[org.joda.time.DateTime 0x595b82e4 "2016-02-14T14:55:21.807Z"],
 :closed false,
 :items []}
{:from
 #object[org.joda.time.DateTime 0x595b82e4 "2016-02-14T14:55:21.807Z"],
 :to
 #object[org.joda.time.DateTime 0x18e43e29 "2016-02-14T14:55:23.807Z"],
 :closed false,
 :items []}
:stop

We have a multi-arity function for creating time windows. With one argument it will produce contiguous windows of X seconds. With two arguments the windows will overlap.

We use state on the go-loop to maintain the interval contiguousness (I checked, yes that is the right word)

We also have a few contractual checks using {pre: to ensure that the arguments are somewhat sane.

Time in Windows

There are many ways to skin this particular cat. But before I go on I just want to say that no cats were actually skinned.

I went away from Joshua’s concept and decided to place the data into the window by means of a vector of items.

We will see that this enables further aggregations based on the data accumulated in each window, which is a reasonably common requirement.

To complete the code, here is the function to generate the windows

(defn- gen-window [start-time open-duration]
  {:pre [(> open-duration 0)]}
  (let [to-time (t/plus start-time (t/seconds open-duration))]
    (assoc {} :from start-time :to to-time :closed false :items [])))

Merging time and data

(defn data-in-timed-series
  "Add timed data from item-ch to the time series windows produced in the window-ch;
   emit the window once only, after it is closed"
  [item-ch window-ch]
  (let [out-ch (chan 1 (comp (map (fn [windows]
                                    (filter #(:closed %) windows)))
                             (filter (fn [windows] (> (count windows) 0)))))]
    (go-loop [active-windows ()]
      (if-let [[data chan] (alts! [item-ch window-ch])]
        (condp = chan
          window-ch (if-let [windows (maintain-active-windows (conj active-windows data))]
                      (do
                        (>! out-ch windows)
                        (recur windows)))
          item-ch (recur (add-timed-item-to-windows data active-windows)))
        (close! out-ch)))
    out-ch))

This function creates a go-loop that combines data from channels on which the windows and time-stamped data are emitted.

When the data is incoming from the item-ch the items are added to appropriate window(s) using add-timed-item-to-windows and we recur on the result.

When the data comes in from the window-ch the new window is added to the list. We will show the maintain-active-windows function shortly.

Finally, the transducer limits the output to closed, non-empty windows.

Time management

Again, no cats skinned but there are choices. In this case I chose to maintain the windows by tracking a boolean and then reaping the windows after a certain time threshold. Here is the code abstracted out into a small function:

(defn- maintain-active-windows [windows]
  (let [now (t/now)
        retention-period (t/millis 500)
        retention-boundary (t/minus now retention-period)
        retained (filter #(t/after? (:to %) retention-boundary) windows)
        to-be-closed (filter #(and (t/before? (:to %) now) (false? (:closed %))) windows)
        closing (map #(assoc % :closed true) to-be-closed)]
    (concat retained closing)))

I chose a retention period of 500ms on the basis that 1000ms is the minimum window size in this design. This way I am always guaranteed to clean up on each new time window.

It also means that there is some small additional time for catching stragglers although that is the limit of any effort to deal with that particular problem. In general stragglers are silently dropped.

Data management

The code for adding items to the windows is fairly boiler plate but presented here so that you can have a more complete view:

(defn- within-interval? [from to time]
  {:pre [(t/before? from to)]}
  "Check whether a time is within an interval"
  (let [interval (t/interval from to)]
    (t/within? interval time)))

(defn- add-timed-item-to-windows [timed-item windows]
  "Add an item to the windows where the time intervals match"
  (if-let [[time item] timed-item]
    (let [matching-windows (filter #(within-interval? (:from %) (:to %) time) windows)
          updated-windows (map #(assoc % :items (conj (:items %) item)) matching-windows)]
      updated-windows)))

The thing I like about this code is that we have avoided creating global state.

Demo time

(finite-printer (stop-after-n-seconds 10) (data-in-timed-series (gen-timed-orders 1000 parts) (create-time-series-windows 4)))
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x2b730646
        "clojure.core.async.impl.channels.ManyToManyChannel@2b730646"]
({:from
  #object[org.joda.time.DateTime 0x6a21c46e "2016-02-14T20:31:28.404Z"],
  :to
  #object[org.joda.time.DateTime 0x51d383fa "2016-02-14T20:31:32.404Z"],
  :closed true,
  :items
  [{:id G__12317, :description "Door control module9"}
   {:id G__12077, :description "Decklid9"}
   {:id G__12382, :description "Fuel tank (or fuel filler) door4"}
   {:id G__12092, :description "Fender (wing or mudguard)4"}]})
({:from
  #object[org.joda.time.DateTime 0x51d383fa "2016-02-14T20:31:32.404Z"],
  :to
  #object[org.joda.time.DateTime 0x66a36cd7 "2016-02-14T20:31:36.404Z"],
  :closed true,
  :items
  [{:id G__12171, :description "Roof rack3"}
   {:id G__12055, :description "Exposed bumper7"}
   {:id G__12428, :description "Window regulator0"}
   {:id G__12167, :description "Rocker9"}]})
:stop

Aggregations

Here is a small general purpose aggegrator that take a function to operate on each window:

(defn interval-aggregator [aggregator in]
  "Execute the provided aggregator against the :items property from the map on the channel"
  (let [out (chan)]
    (go-loop []
      (if-let [window (first (<! in))]
        (do
          (let [results (aggregator (:items window))]
            (>! out results))
          (recur))
        (close! out)))
    out))


(finite-printer (stop-after-n-seconds 10) (interval-aggregator count (data-in-timed-series (gen-timed-orders 1000 parts) (create-time-series-windows 4))))
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x77069406
        "clojure.core.async.impl.channels.ManyToManyChannel@77069406"]
3
4
:stop

Summary

So that’s our final experiment with aggregating over time series.

The amount of code is pleasantly small and we can see many possibilities for playing with time series data.

The code is all on my GitHub.

Conclusions

This really has been me scraping the surface of core.async by trying to scratch a few itches. I found the model quite straightforward to use and extremely powerful. I will continue to experiment with the library and write up some samples as further revelations unfold!

And finally - Thanks!

Thanks for making it through, especially of you ploughed through the series!

I have a better understanding of core.async after writing this and I hope that’s true for you too!

Zing me or ping me if this was useful via Twitter or in the comments below.

Event Stream Processing - Managing state with core.async

Tue, 09 Feb 2016 18:59:38 +0000

Introduction

In my last post I went through the code for a data flow using a product stock level tracker. It had a simple filter to ensure that the notifications occurred when a specific business rule was fired.

In that post I showed the basic mechanisms for creating and connecting channels. In this post I want to show how to combine data from multiple streams. To demonstrate this we will implement the processes around the Stock Management Tracker.

In the tracker we will join the two streams to emit a running total of stock on hand vs orders at any point in time.

Data first

Let us just quickly outline the data in our model:

; Order
{:item-id A123 :description "Windscreen wiper" :quantity 2 :customer-id C234}
; Delivery
{:item-id A123 :description "Windscreen wiper" :quantity 40 :supplier-id S234}

Once again this is certainly a simple model but is sufficient to demonstrate the power of the solution.

To help with the simulation we will throw together some sample data and create a few small functions to generate the orders and deliveries.

Parts data generation

; Wikipedia sourced list of body and main parts
(def manifest ["Bonnet/hood" "Bonnet/hood latch" "Bumper" "Unexposed bumper" "Exposed bumper"
               "Cowl screen" "Decklid" "Fascia rear and support" "Fender (wing or mudguard)"
               "Front clip" "Front fascia and header panel" "Grille (also called grill)"
               "Pillar and hard trim" "Quarter panel" "Radiator core support"
               "Rocker" "Roof rack" "Spoiler" "Front spoiler (air dam)" "Rear spoiler (wing)" "Rims" "Hubcap"
               "Tire/Tyre" "Trim package" "Trunk/boot/hatch" "Trunk/boot latch" "Valance" "Welded assembly"
               "Outer door handle" "Inner door handle" "Door control module" "Door seal"
               "Door watershield" "Hinge" "Door latch" "Door lock and power door locks" "Center-locking"
               "Fuel tank (or fuel filler) door" "Window glass" "Sunroof" "Sunroof motor" "Window motor"
               "Window regulator" "Windshield (also called windscreen)" "Windshield washer motor"
               "Window seal"])

; magic up 460 items with variants for each part (adjust the range to create more / less)
(def parts (mapcat (fn [part]
                     (map #(assoc {} :id (gensym) :description (str part %)) (range 10)))
                   manifest))

Order and delivery generation

(defn gen-timed-orders [frequency-millis coll]
  (let [out (chan)]
    (go-loop []
      (do
        (>! out [(t/now) (rand-nth coll)])
        (<! (timeout frequency-millis))
        (recur)))
    out))

(defn gen-random-deliveries [max-wait-millis coll]
  (let [out (chan)]
    (go-loop []
      (do
        (>! out [(t/now) (rand-nth coll)])
        (<! (timeout (rand-int max-wait-millis)))
        (recur)))
    out))

Here we generate infinite streams of data with time stamps. That will be used in later examples to aggregate data into time series.

They differ only in the way that orders are consistent and deliveries are random. This is not a perfect model of the real world (yeah, I know) but is good enough for this purpose.

Short demo

(def stock-levels-ch (stock-levels orders deliveries))

(finite-printer (stop-after-n-seconds 2) stock-levels-ch)

=> #'core-async-examples.stock-management-tracker/stock-levels-ch
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x6e923694
        "clojure.core.async.impl.channels.ManyToManyChannel@6e923694"]
{:id G__11226, :count -1}
{:id G__11584, :count 1}
{:id G__11422, :count -1}
{:id G__11408, :count -1}
{:id G__11579, :count 1}
{:id G__11223, :count -1}
{:id G__11288, :count -1}
{:id G__11656, :count -1}
{:id G__11333, :count 1}
{:id G__11425, :count -1}
{:id G__11666, :count -1}
{:id G__11561, :count -1}
{:id G__11642, :count -1}
{:id G__11584, :count 1}
:stop

Blocking without Thread/sleep == SCAAAAALE!

One aspect to note in the above code is the timeout function. It reads from a virtual channel and effects a pause on the generation like Thread/sleep but does not block.

Not blocking means that it is OK to start 100s or 1000s of go blocks with very little CPU or RAM overhead. This is similar in effect to the way that NGINX is architected.

Don’t change that channel

I’m going to show this next function in two forms. The first form shows how to use conditionals based on channel

(defn stock-levels [orders deliveries]
  "Stock levels - also includes demand (negative stock numbers)"
  (let [out (chan)
        dec-stock (partial modify-stock dec)
        inc-stock (partial modify-stock inc)]
    (go-loop [stock {}]
      (if-let [[data chan] (alts! [orders deliveries])]
        (let [[_ item] data
              update-operation (condp = chan
                                 orders dec-stock
                                 deliveries inc-stock)]
          (do
            (>! out (update-operation stock item))          ; does not work - need to manage 'stock'
            (recur stock)))
        (close! out)))
    out))

In this first showing of the code we see the ability to read many channels at once using alts! which receives the data and the channel that was read.

In this example we use condp to run differing code based on the channel that was read. In this case we set the method to adjust stock.

Re-Stating our intentions

The gap we have with the above code is that it will only ever report 0 or 1.

That might might be useful if have an external aggregator but we can also aggregate in the go-loop directly:

(defn stock-levels [orders deliveries]
  "Stock levels - also includes demand (negative stock numbers)"
  (let [out (chan)
        dec-stock (partial modify-stock dec)
        inc-stock (partial modify-stock inc)]
    (go-loop [stock (into #{} (map #(assoc {} :id (:id %) :count 0) parts))]
      (if-let [[data chan] (alts! [orders deliveries])]
        (let [[_ item] data
              update-operation (condp = chan
                                 orders dec-stock
                                 deliveries inc-stock)]
          (if-let [[modified-stock updated-item] (update-operation stock item)]
            (do
              (>! out updated-item)
              (recur modified-stock))))
        (close! out)))
    out))

Using the parameters to the go-loop we can initiate state and then maintain it via recur.

In this case we create a set of parts and set each stock item count to 0 when our function starts. One can imagine more complex initialisations!

In the loop we then pass the stock collection to the modify-stock function which returns a new version of the stock collection.

Here we exploit the fact that Clojure collections are very efficient with respect to minimising the costs of each new version.

This stock list could easily be scaled to tens of thousands of parts without adding any latency costs (barring side effects of swapping although luckily we are no longer limited to 32k, 32Mb or even 32Gb of memory like in the good old days). For more information see this rich visual explanation from Jean Niklas L’orange.

Short Demo Two

(def order-chan (async/mult (gen-timed-orders 10 parts)))  ; use mult to allow many readers

(def deliveries-chan (gen-random-deliveries 1000 parts))

; show what's happening with stock
(def stock-chan (stock-levels (async/tap order-chan (chan)) deliveries-chan))
(def backlogs-chan (detect-backlogs -4 stock-chan))
(finite-printer (stop-after-n-seconds 20) backlogs-chan) ; 20k orders will be processed

=> #'core-async-examples.stock-management-tracker/order-chan
=> #'core-async-examples.stock-management-tracker/deliveries-chan
=> #'core-async-examples.stock-management-tracker/stock-chan
=> #'core-async-examples.stock-management-tracker/backlogs-chan
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x694db469
        "clojure.core.async.impl.channels.ManyToManyChannel@694db469"]
{:id G__11288, :count -5}
{:id G__11338, :count -5}
{:id G__11603, :count -5}
{:id G__11338, :count -5}
:stop

Modify without side effects

To finalise the example here is the code for modify-stock which runs the provided function to obtain the new value using the existing count on the item.

(defn modify-stock [count-adjust-fn stock item]
  (let [current-value (first (clojure.set/select #(= (:id %) (:id item)) stock))
        new-value (assoc current-value :count (count-adjust-fn (:count current-value)))
        updated-stock (conj stock new-value)]
    [updated-stock new-value]))

One nice aspect of using Clojure’s set data structure is that we can use conj as a succinct upsert.

Summary

In this example we saw how to implement a stock level tracker. To achieve this functionality we read multiple channels in one go-loop and distinguished data from those channels. Further we maintained aggregations and state around the go-loop to increase power and simplicity.

The code is all on my GitHub.

Next next next

In the final example of this short series I will show how to track stream data within series of time.

And finally - Thanks!

Thanks for making it through. I have a better understanding of core.async after writing this and I hope that’s true for you too!

Zing me or ping me if this was useful via Twitter or in the comments below.

Event Stream Processing - Coding introduction with core.async

Sat, 06 Feb 2016 18:59:38 +0000

Introduction

In my initial post I outlined a small bare-bones order management system that could act as a motivation for Event Stream Processing.

In this post I want to show some code to implement how we may implement the processes around the Stock Tracker.

Data first

Let us just quickly outline the data in our model:

{:id A123 :supplier S345 :quantity 100}

This is certainly a simple model but not necessarily useless and reduces the clutter around the examples.

Start with a printer

At the REPL we define a generic printer for data…

(defn printer [data-ch]
    (go-loop []
      (if-let [data (<! data-ch)]
        (do
          (clojure.pprint/pprint data)
          (recur)))))
=> #'stock.core/printer

The go-loop construct is to establish a go block with a forever loop. The loop will recur until the if-let condition fails while running a read <! on the channel.

Any attempt to read from a closed channel will result in a nil and cause the read to fail. So once we close off the channel the program will quietly end.

(def data-channel (chan))
=> #'stock.core/data-channel
(printer data-channel)
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x6b17235
        "clojure.core.async.impl.channels.ManyToManyChannel@6b17235"]

Next we create a channel with the chan function and pass that to create a running instance of the printer.

The printer is now sitting in the background waiting on data from that open channel and we have our first core.async program up and running.

Let’s play with it a little at the REPL and then shut it down:

(async/put! data-channel "Hello")
=> true
"Hello"
(async/close! data-channel)
=> nil
(async/put! data-channel "Goodbye")
=> false

put! and take! are used for interacting with the channels outside of a go block or a go-loop.

OK, so far so good. We have a basic example so let’s look at the next program that will do something useful in the system…

Next - the end

The most trivial consumer in the diagram is the Supplier Notifier which is at the end of the stream process flow in this case.

That process listens on the Supplier Order channel and calls the supplier API.

Let’s fake up a call to the service…

(defn supplier-api-call [order]
  (println "Calling REST API with order")
  (clojure.pprint/pprint order))
=> #'stock.core/supplier-api-call

Next let’s get our Supplier Order channel consumer defined so that it can invoke the API…

(defn supplier-notify [order-ch]
    (go-loop []
      (if-let [order (<! order-ch)]
        (do
          (supplier-api-call order)
          (recur)))))

Nothing suprising there. We substitute the printing function with the call to the API.

Let’s get the channels created and hooked up:

=> #'stock.core/supplier-notify
(def supplier-order-ch (chan))
(def supplier-order-ch-mult (async/mult supplier-order-ch))
(supplier-notify (async/tap supplier-order-ch-mult (chan)))

=> #'stock.core/supplier-order-ch
=> #'stock.core/supplier-order-ch-mult
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x2e68af05
        "clojure.core.async.impl.channels.ManyToManyChannel@2e68af05"]

The incredibly observant amongst you will have noticed that Order Reconciliation will also listen to the supplier order channel. So this time we have used mult to create a multiple of the supplier-order-ch.

mult enables several clients to listen to the same channel when they call the tap function on a new channel. tap as the name implies copies the data verbatim from the original channel onto the new channel that we are using for our notifier.

Let’s give that a quick spin:

(put! supplier-order-ch {:id (gensym) :supplier "Acme Supplies" :quantity 100 })

=> true
Calling REST API with order
{:id G__12827, :supplier "Acme Supplies", :quantity 100}

Warm up complete

So that’s the first edge of the system complete.

The interesting general point so far is that we have been able to develop and test this all out in the REPL.

We want to keep that up as we go forward to model the Stock Tracker:

(defn stock-tracker [low-water-mark stock-levels]
  "Show where stock levels fall for an item below a low water mark"
  (let [out (chan 1 (filter (fn [part-level]
                              (let [backlog (:quantity part-level)]
                                (< backlog low-water-mark)))))]
    (go-loop []
      (if-let [stock-level-for-part (<! stock-levels)]
        (do
          (>! out stock-level-for-part)
          (recur))
        (close! out)))
    out))

=> #'stock.core/stock-tracker

In this case see our first transducer - the filter function that is attached to the channel. The filter here acts as a simple barrier on the output channel.

That’s it. Our Stock Tracker is complete. Let’s test it out…

(def stock-tracker-ch (chan))
(def stock-tracker-ch-mult (async/mult stock-tracker-ch))
(def orders-ch (stock-tracker 50 (async/tap stock-tracker-ch-mult (chan))))

(channel-printer orders-ch)

=> #'stock.core/stock-tracker-ch
=> #'stock.core/stock-tracker-ch-mult
=> #'stock.core/orders-ch
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0x7618699
        "clojure.core.async.impl.channels.ManyToManyChannel@7618699"]

Here we set up the channels in the same manner as our last case with the addition of our printer to have eyes on whether the filter is doing its job.

Let’s run a couple of test cases through:

(put! stock-tracker-ch {:id (gensym) :supplier "Acme Supplies" :quantity 10 })

=> true
{:id G__13669, :supplier "Acme Supplies", :quantity 10}
(put! stock-tracker-ch {:id (gensym) :supplier "Acme Supplies" :quantity 100 })

=> true

In the second case our low water mark has not been breached so the data is dropped.

Streaming hot

Finally let’s hook up all of the components and run through the same test cases:

(def stock-tracker-ch (chan))

(def stock-tracker-ch-mult (async/mult stock-tracker-ch))

(def supplier-order-ch (stock-tracker 50 (async/tap stock-tracker-ch-mult (chan))))

(def supplier-order-ch-mult (async/mult supplier-order-ch))

(supplier-notify (async/tap supplier-order-ch-mult (chan)))

=> #'stock.core/stock-tracker-ch
=> #'stock.core/stock-tracker-ch-mult
=> #'stock.core/supplier-order-ch
=> #'stock.core/supplier-order-ch-mult
=>
#object[clojure.core.async.impl.channels.ManyToManyChannel
        0xcc08681
        "clojure.core.async.impl.channels.ManyToManyChannel@cc08681"]

We don’t need the printer this time around since the notify function will report is invocation.

And here we see it flow through:

(put! stock-tracker-ch {:id (gensym) :supplier "Acme Supplies" :quantity 10 })

=> true
Calling REST API with order
{:id G__13741, :supplier "Acme Supplies", :quantity 10}

(put! supplier-order-ch {:id (gensym) :supplier "Acme Supplies" :quantity 100 })

=> true

Summary

We have seen the stock data flow through a tracker which had a simple filter to ensure that the notifications occurred when a specific business rule was fired.

And this all happened in real time with minimal resource usage. Scaling down is often as equally important as scaling up.

The code is all on my GitHub.

Next next next

In the next post I show the Stock Management Tracker to combine data from more than one channel and maintain local state within the go block.

And finally - Thanks!

Thanks for making it through. I have a better understanding of core.async after writing this and I hope that’s true for you too!

Zing me or ping me if this was useful via Twitter or in the comments below.