| link | Last. 20260203T095052303Z |
My goal today was to illustrate using my symbolic deep learning inferencing. Simply writing the condition handling and restarts ended up eating the coding part of my day. But this is pretty interesting in itself and it works as I intended. This article follows directly from A Better Deep Learning Algorithm In Lisp.
CL-USER> (infer 0 0 0 0
:item 'foo
:input '(((foo bar)))
:memories '((((bar)))))
(NOW IN HANDLE-COMPUTE-WINNER)
(NOW IN HANDLE-WINNER)
(NOW IN HANDLE-NEW-VALUE)
New value: (BAR)
(BAR)
CL-USER> (infer 0 0 0 0
:item 'bar
:input '(((foo)))
:memories '((((bar)))))
(NOW IN HANDLE-COMPUTE-WINNER)
(NOW IN HANDLE-WINNER)
(NOW IN HANDLE-NEW-VALUE)
New value: (BAR FOO)
(BAR FOO)
CL-USER> (infer 0 0 0 0
:item 'foo
:input '(((foo bar)))
:memories '((((bar)))
(((foo)))))
(NOW IN HANDLE-COMPUTE-WINNER)
(NOW IN HANDLE-WINNER)
(NOW IN HANDLE-NEW-VALUE)
New value: (FOO FOO BAR)
(FOO FOO BAR)
Condition signalling is a good choice for my deep learning inferencing because there are many different correct seeming behaviours for different contexts.
Particularly, if a whole-matrix breadth first update is being done per time step, the result-writing-restart (remember that a condition handler either declines by returning or else transfers local control to a restart) would need to write results to a new matrix so the earlier inferences do not effect later inferences. But incremental single random updates should probably be in-place. One can also imagine writing the change to a file.
In my code here and earlier, atomic items are tested for membership as the predicate. Here, a handler then pushes or deletes the atomic item from the new matrix cell. But one can easily imagine item being a list of symbols, the predicate being intersection, and the true/false outcomes being union and set-difference. These are not mutually exclusive. Using condition signalling and handler-bind, atomic item handlers would just decline sequences for item, and list item handlers vice versa.
Here I am not very confident about my resignalling control flow of basically:
(prog ((c nil))
start
(restart-case
(if c
(signal c))
(resignal (condition) (setq c condition) (go start))))
on the theory that different handlers will pick up the condition that is being signalled multiple times. I cannot signal the condition from inside the resignal restart, because the resignal restart is out of scope there. So I need to go with the condition to before the restart-case, re-enter the restart-case and signal in order to be able to resignal again.
For clarity, when I said my symbolic hopfield network is an implementation of deep learning, I specifically meant that it is dual to a feed-forward neural network of a single hidden layer with an obscure activation function - Krotov and Hopfield, 2016. It is possible to use a Boltzman distribution (attention mechanism) for the update rule - Hopfield Networks Is All You Need (a parody of Attention Is All You Need).
Symbolic neural networks are a verdant field as Amen Zwa, esq. points out, exhibiting interpretability and closely tied to theorem proving as seen in IBM’s Logical Neural Networks (Riegel et al. 2020). My one’s odd point is that it is dual to - i.e. is an implementation of - a feed-forward neural network of a single hidden layer, which is the standard uninterpretable deep learning inferencing thing also used in large model transformers.
Read Artyom’s whole thread basically: https://merveilles.town/@aartaka/115998271580048333
My broad approach was to keep signalling one condition, and have handlers check slots of that condition when choosing to decline.
(define-condition
dl-inference
()
((item :initarg :item :accessor dl-item)
(input :initarg :input :accessor dl-input)
(memories :initarg :memories :accessor dl-memories)
(keys :initarg :keys :type t :accessor dl-keys)
(winner :type t :initarg :winner :accessor dl-winner)
(old-value :type t :initarg :old-value
:accessor dl-old-value)
(new-value :type t :initarg :new-value
:accessor dl-new-value)))
For when a new-value is present, when there is a winner to produce a new-value, and when a winner still needs to be computed.
I believe each of these themes can be improved by including multiple more specific handlers for each, but these ones are indicative.
(defun handle-new-value (dl-inference)
(unless (slot-boundp dl-inference 'new-value)
(return-from handle-new-value))
(print '(now in handle-new-value))
(let ((r (find-restart 'write-new-value dl-inference)))
(when r
(invoke-restart r (dl-new-value dl-inference)))))
(defun handle-winner (dl-inference)
(unless (slot-boundp dl-inference 'winner)
(return-from handle-winner))
(print '(now in handle-winner))
(setf (dl-new-value dl-inference)
(copy-tree (dl-old-value dl-inference))
(dl-new-value dl-inference)
(if (dl-winner dl-inference)
(push (dl-item dl-inference)
(dl-new-value dl-inference))
(delete (dl-item dl-inference)
(dl-new-value dl-inference))))
(let ((r (find-restart 'resignal dl-inference)))
(when r (invoke-restart r dl-inference))))
(defun handle-compute-winner (dl-inference)
(when (slot-boundp dl-inference 'winner)
(return-from handle-compute-winner))
(print '(now in handle-compute-winner))
(with-slots
(winner old-value)
dl-inference
(multiple-value-setq
(winner old-value)
(apply 'sum-memories
(dl-item dl-inference)
(dl-input dl-inference)
(dl-memories dl-inference)
(dl-keys dl-inference))))
(let ((r (find-restart 'resignal dl-inference)))
(when r (invoke-restart r dl-inference))))
Functions like this should be built out of the available conditions, handlers and restarts.
(defun infer (start-row start-col target-row target-col
&key item input memories (dl-keys (list))
(write-new-value-p
(find-restart 'write-new-value)))
(setf (getf dl-keys :start-row) start-row
(getf dl-keys :start-col) start-col
(getf dl-keys :target-row) target-row
(getf dl-keys :target-col) target-col
(getf dl-keys :rectified-polynomial)
(make-rectified-polynomial 2))
(handler-bind
((dl-inference #'handle-new-value)
(dl-inference #'handle-winner)
(dl-inference #'handle-compute-winner))
(prog ((condition nil))
start
(restart-case
(cond
(write-new-value-p
(if condition
(signal condition)
(signal 'dl-inference
:item item :input input
:memories memories
:keys dl-keys)))
(t
(restart-case
(if condition
(signal condition)
(signal 'dl-inference
:item item :input input
:memories memories
:keys dl-keys))
(write-new-value
(new-value)
(terpri)
(princ "New value: ")
(return (prin1 new-value))))))
(resignal (c) (setq condition c) (go start))))))
(defparameter *c* (make-instance
'dl-inference
:item 'foo
:input '(((foo)))
:memories '((((foo))))
:keys (list :rectified-polynomial
(make-rectified-polynomial 2))))
(setf (getf (dl-keys *c*) :target-row) 0)
(setf (getf (dl-keys *c*) :target-col) 0)
(setf (getf (dl-keys *c*) :start-row) 0)
(setf (getf (dl-keys *c*) :start-col) 0)
(handler-bind
((t #'identity))
(handle-compute-winner *c*))
(dl-winner *c*)
(dl-old-value *c*)
(handler-bind
((t #'identity))
(handle-winner *c*))
(dl-new-value *c*)
(restart-case
(handle-new-value *c*)
(write-new-value
(&rest rest)
(print rest)))
->
CL-USER> (defparameter *c* (make-instance
'dl-inference
:item 'foo
:input '(((foo)))
:memories '((((foo))))
:keys (list :rectified-polynomial
(make-rectified-polynomial 2))))
*C*
CL-USER> (setf (getf (dl-keys *c*) :target-row) 0)
0
CL-USER> (setf (getf (dl-keys *c*) :target-col) 0)
0
CL-USER> (setf (getf (dl-keys *c*) :start-row) 0)
0
CL-USER> (setf (getf (dl-keys *c*) :start-col) 0)
0
CL-USER> (handler-bind
((t #'identity))
(handle-compute-winner *c*))
(NOW IN HANDLE-COMPUTE-WINNER)
NIL
CL-USER> (dl-winner *c*)
T
CL-USER> (dl-old-value *c*)
(FOO)
CL-USER> (handler-bind
((t #'identity))
(handle-winner *c*))
(NOW IN HANDLE-WINNER)
NIL
CL-USER> (dl-new-value *c*)
(FOO FOO)
CL-USER> (restart-case
(handle-new-value *c*)
(write-new-value
(&rest rest)
(print rest)))
(NOW IN HANDLE-NEW-VALUE)
((FOO FOO))
((FOO FOO))
CL-USER>