screwlisp proposes kittens

C++ and embeddable common lisp habitat example ~ base64 encoding

Habitat being jeremy_list’s HaikuOS secure scuttlebutt C++ gossip protocol client. Of course, I use ANSI CL and not C++, and openbsd or maybe debian and not haikuos. So let’s see if we can just make the C++ bits into lisp functions in that magical embeddable common lisp way.

Here we continue directly from making the conventional BSD port of ecl embeddable common lisp --with-cxx. I’m aware that at least the gentoo linux distribution has a C++ ecl port (in gentoo, called an ebuild).

Set up common lisp / eev / emacs

#P"~/ecl-habitat-test/eat-base64-habitat.ecl.lisp" was my working file. This is the adaptation of eev’s M-x eeit generated infrastructure I have found myself using.

#|
• (setq inferior-lisp-program "ecl")
• (slime)
• (setq eepitch-buffer-name "*slime-repl ECL*")

(compile-file "~/ecl-habitat-test/eat-base64-habitat.ecl.lisp" :load t)
|#

compile-file in embeddable common lisp provides a very nice low-level-ffi, sffi into either C or C++ (whatever ecl had been built with (--with-cxx or not)).

NOTE all of ecl’s sffi stuff is compile-file-only. So we have to extract the ffi of this article into a lisp file and call compile-file … :load t as indicated above. The interpreter doesn’t know how to compile C (it only interprets lisp).

Taking C++ base64::encode1 from habitat’s source

I eventually came to terms with the fact that habitat used a nonportable HaikuOS C++ string library. Instead, I pulled out the base64 encoding source C++ namespace fragment. I promoted the enums and integer-like types to ints, though ecl’s sffi directly supports all these common C/C++ types, but they are a bit extraneous to this minimal ecl C++ sffi example.

(Edit: Simple fix mentioned everywhere else right after publishing) Actually, from what I can tell, at least my extracted version of b64 encoding here doesn’t work, but the C++ embeddable common lisp works which is our focus. jeremy_list will take a look once this article is up and point out where I went wrong here.

(ffi:clines "
namespace base64 {

enum Variant {
  STANDARD,
  URL,
};

int
encode1(int byte, int variant)
{
  if (byte <= 25) {
    return byte + 'A';
  } else if (byte <= 51) {
    return byte - 26 + 'a';
  } else if (byte <= 61) {
    return byte - 52 + '0';
  } else if (byte == 62) {
    if (variant == STANDARD)
      return '+';
    else
      return '-';
  } else {
    if (variant == STANDARD)
      return '/';
    else
      return '_';
  }
}

}
")

A lisp defun whose body is that C++ function

ffi:defentry (sffi docs) attaches a C/C++ function as body to a common lisp function. This is just what the form is like.

(ffi:defentry c-encode1
    (:int :int)
  (:int |base64::encode1|))

ffi:defentry’s arguements are

  1. name (so the lisp function will be named c-encode1)
  2. C/C++ argument types (I just promoted them to ints - if you want to you can get the underlying types and enum right)
  3. (return-type c-or-c++-function-name)

A common lisp stab at b64 encoding using that

In one big common lisp loop sorry. I just tried to literally match jeremy_list’s code rather than refer to openbsd’s b64encode myself. So I am not sure what translation mistake has crept in: Anyway, the function ‘works’ for what it is it happens to do.

(defun b64encode (bytes &optional (variant 0))
  (loop
    :with mask6 := #b111111

    :for n :from 0 :by 4
    :for byte-1 :in bytes :by #'cdddr
    :for byte-2 :in (cdr bytes) :by #'cdddr ; I forgot this cdr
    :for byte-3 :in (cddr bytes) :by #'cdddr ; and cddr originally here.

    :for result-1
      := (c-encode1 (ash byte-1 -2) variant)
    :for partial-1
      := (logand (ash byte-1 4)
		 mask6)

    :for result-2
      := (c-encode1 (logior partial-1
			    (ash byte-2 -4))
		    variant)
    :for partial-2
      := (logand (ash byte-2 2)
		 mask6)

    :for result-3
      := (c-encode1 (logior (ash byte-3 -6)
			    partial-2)
		    variant)
    :for result-4
      := (c-encode1 (logand byte-3 mask6)
		    variant)

    :nconcing
    (list
     result-1 result-2 result-3 result-4)
      :into b64s
    :finally
       (return
	 (coerce
	  (mapcar 'code-char b64s)
	  'string))))

Seeing that working (“working”)

#|
• (setq inferior-lisp-program "ecl")
• (slime)
• (setq eepitch-buffer-name "*slime-repl ECL*")
(load "eat-base64-habitat.ecl.lisp")

"abcdefghijkl"
;; "abcdefghijkl"
(coerce * 'list)
;; (#a # #c #d #e # #g #h #i #j #k #l)
(mapcar 'char-code *)
;; (97 98 99 100 101 102 103 104 105 106 107 108)
(b64encode *)
;; YWJjZGVmZ2hpamts ; now works.
;; "YWFhZGRkZ2dnampq" ; When I forgot those cdrs.

Well, this (Edit: Now works, simple typo above) seem[ed] wrong viz openbsd b64encode(1):

• (eepitch-shell)
printf "abcdefghijkl" | b64encode lowercase
;; " | b64encode lowercase
;; begin-base64 644 lowercase
;; YWJjZGVmZ2hpamts
;; ====
|#

Edit: Now it’s working. Forgot three cdrs.

Formerly: Though it seems oddly close doesn’t it. But it’s difficult for me to guess what has gone wrong.

Still, the C++ program code was made available in common lisp. So that has gone right.

Conclusions

We saw C++ embeddable common lisp compile and connect common lisp to jeremy_list’s original C++ namespace using its sffi (which is ecl; not the external libffi - I have met people who just guess only libffi exists), and used it to write a portable lisp version of the base64 encoding rather than a nonportable haikuos C++ one. So we can straightforwardly salvage other projects as starting points; especially C++ programs and libs.

A simple sanity check with openbsd shows something like 2/4 bytes seem to agree with openbsd’s b64encode implementation. Edit: This turned out to be a simple fix.

Threeish future implications being ecl inhabiting a C++ program to serve McCLIM application-frames into the running C++ program ; relatedly, starting a swank server inside a pre-existing C++ program and slime-connecting into the running program from emacs ; lastly, and it’s been on my todo-list a while - using the enzyme library for automatic differentiation and friends with the previous, especially together with my Leonardo system.

Fin.

Talk on the Mastodon as always please.

Oh! One hour from right now 8am Zulu Sunday I’m going to have a peertube live with Kasper Galkowski about common lisp indie gamedev with opengl, which is at least an ffi adjacent topic. Watch the Mastodon, again please.

Edrx has been working on emacs eev-mode kitten and has one for me personally I need to work through - this will shape what these articles are like going forward.

screwlisp proposes kittens