:screwlisps-knowledge/tangle common lisp asdf package markdown tangling and lisp pathnamesHere I exhibit a simple use of cl-seriesâ scan-file and collect-file to tangle a markdown file into a lisp package in a specified lisp system. Further, I add a system stubbing function which uses lisp pathnames, which are a tricky topic.
My motivation was to at least in common lisp reconcile my small-web markdown and lisp systems, by defining my lisp systems as tangles of my small-web markdown. Edrx showed me that something like this could be done by modifying emacs eevâs code-c-d, but, well, I know one hundred percent how what I just wrote here works.
It is a little hard to remember that common-lisp pathname stuff, as Kent fairly frequently says: it was written for back when there was more than one style of filesystem. However, after thinking a bit about it, I am fairly happy with it. the gist is that I have
make-pathname.(:relative :wild)).Wild pathnames can have more advanced uses, but this is not a bad idiom on modern systems anyway I think. The modern string convention is I think, more primitive.
This is a block comment, prior to the required package specification. Elsewhere I have talked about my focus on embeddable common lisp.
#|
;; Allowing, the quicklisp links are for sbcl and sly. Still, it gives an idea.
(find-quicklisp-links "series")
(setq inferior-lisp-program "ecl")
(slime)
(setq eepitch-buffer-name "*slime-repl ECL*")
(require :series)
|#
This must be the first lisp expression in the file (since from our :package-inferred-system useage, experientially files wonât exist but packages do).
(uiop:define-package :screwlisps-knowledge/tangle
(:import-from :series install)
(:mix :asdf :uiop :cl)
(:export :tangle)
(:nicknames :sk/tangle :screwniverse/tangle))
(in-package :sk/tangle)
(eval-when
(:compile-toplevel :execute)
(install))
For those who arenât used to common-lisp, asdf, and uiop:
:package-inferred-system detailsThis is in my view a modern residential source filesystem in the sense that there is no âfileâ - only packages, where packages are inside systems. In order to make this work, each source file created in a system starts with either defpackage or uiop:define-package, and the package generally needs to be named :system/dir1/dir2/dir3/filename (downcased).
Then, in the absense of a (:mix ..), (:mix :cl) is assumed if I recall correctly, however I specifically import install from series.
In the new package=file, at compile-toplevel, I call cl-seriesâ install (which temporarily interns seriesâ exported symbols into the current or specified package).
Letâs do this in a comment in the file. In some sense, it is documentation of what we did. Letâs also use the common lisp formatted output ~@\newline directive which will let us use indented '''s safely.
(with-open-file
(*standard-output* #p"/tmp/test-tangle.md"
:direction :output
:if-does-not-exist :create)
(format *standard-output*
"# An MD file~@
having~@
```~@
(lisp code)~@
```~@
blocks~@
## Such as~@
```~@
;; this.~@
```~@
``` but not this.~@
."))
cl-Series provides working fakes of its static-analysis-generated tight, lazy, efficient functionality. However, the fakes are not pure common lisp. But you can use them to explore interactively in seriesâ declarative style, and see what-youâre-working-with while youâre working on it, as Hairylarry for example mentioned doing.
(defparameter *filein* (scan-file #p"/tmp/test-tangle.md"
#'read-line))
*filein*
Using eev-mode, on my right side I see:
#|
SK/TANGLE> (defparameter *filein* (scan-file #p"/tmp/test-tangle.md"
#'read-line))
*FILEIN*
SK/TANGLE> *filein*
#Z("# An MD file " "having " "```" "(lisp code)" "```" "blocks" "## Such as" "```" ";; this." "```" ".")
SK/TANGLE>
|#
I guess, I would like to know where lines starting with ''' are.
(defparameter *codeblock-toggles*
(#Mstring= (series "```") *filein*))
*codeblock-toggles*
Seriesâ series creates an infinite series of whatever its arguement is, which I compare to the finite series of lines read-lineed from the file. We donât have to personally manage the end of file stuff.
Outputs:
#|
SK/TANGLE> (defparameter *codeblock-toggles*
(#Mstring= (series "```") *filein*))
*CODEBLOCK-TOGGLES*
SK/TANGLE> *codeblock-toggles*
#Z(NIL NIL T NIL T NIL NIL T NIL T NIL)
SK/TANGLE>
|#
Looking good. Now here we come to an annoying-ish point for my personal intuitions about series. Sometimes we need to use a toggle as I am about to, because we-donât-know-what-happens-in-the-future. This is quite different to eagerly evaluated code we may be used to.
(defparameter *blocksp*
(let* ((in-block nil))
(mapping
((maybe-latch *codeblock-toggles*)
(line *filein*))
(if in-block
(if maybe-latch
(setq in-block (not in-block))
line)
(and maybe-latch
(setq in-block (not in-block))
nil)))))
Well, letâs do an assertion.
(assert (equal "(lisp code)"
(collect-nth 3 *blocksp*))
nil
"~a should be "(lisp code)" in the test example."
(collect-nth 3 *blocks*))
I guess we just want the lines, and write-lineed to a file.
(collect-file
"/tmp/test-tangle.lisp"
(choose *blocksp* *blocksp*)
'write-line)
Letâs just trivially collect our test assert case into a function.
(defun tangle
(md-path system
&aux
(fileout (make-pathname
:name (pathname-name md-path)
:type "lisp"
:directory
(pathname-directory
(system-source-directory system)))))
(let* ((filein (scan-file md-path #'read-line))
(codeblock-toggles (#Mstring= (series "```")
filein))
(blocksp
(let ((in-block nil))
(mapping
((maybe-latch codeblock-toggles)
(line filein))
(if in-block
(if maybe-latch
(setq in-block (not in-block))
line)
(and maybe-latch
(setq in-block (not in-block))
nil))))))
(collect-file
fileout
(choose blocksp blocksp)
'write-line)))
I guess one should probably not want this. Still, here it is.
(defun stub-system
(namestring)
(let* ((homedir (user-homedir-pathname))
(dir (make-pathname :directory
`(:relative ,namestring)))
(lispdir (make-pathname :directory
'(:relative "common-lisp")))
(sysdir
(reduce 'merge-pathnames
(list dir lispdir homedir)))
(asd
(make-pathname :name namestring
:type "asd"
:directory '(:relative :wild)))
(system-asd
(make-pathname
:directory (pathname-directory sysdir)
:name (pathname-name asd)
:type (pathname-type asd))))
(ensure-directories-exist sysdir)
(with-open-file
(*standard-output* system-asd
:direction :output
:if-does-not-exist :create
:if-exists :error)
(format t
"(defsystem ~a ~
:class :package-inferred-system)"
(intern namestring :keyword))
(probe-file
(make-pathname
:directory (pathname-directory sysdir)
:name (pathname-name asd)
:type (pathname-type asd))))))
(eval-when
()
(stub-system "foo")
(tangle #p"/tmp/test-tangle.md"
"foo")
)
it will error if #p"~/common-lisp/foo/foo.asd" already exists. It works đș
While my intended focus was the first two, I found I usefully covered three topics here:
asdf :class :package-inferred-system lisp systemscan-file and collect-file from seriesmake-pathname directories.I do think that make-pathname is more sophisticated than primitive string-processing style modern convention pathnames. Hope it helps someone get used to these three topics.
Currently works on my machine!
I guess this article is somehow both very simple and very intricate. I would like to hear your improvements, alternatives, or memories of what-you-would-do on the mastodon thread.
Because our article shows all of
as always, I would appreciate you sharing it in any particular means and manner that occurs to you personally.
Small note: When I used the system on itself, my markdown file was named #p"tangle.page.md" so the lisp file it generated from itself used metacircularly was #p"~/common-lisp/screwlisps-knowledge/tangle.page.lisp" - I didnât think of how I want to deal with other programms âsubtypingâ file types like this, so I just fixed it directly for now. If you donât subtype files inside the filename, I guess it just works.