Archive for July, 2018

Log Reading, Week 4

Tuesday, July 31st, 2018

This week sees 10 new log entries. Again not the expected 14. Hence I'm changing my base organizational approach and will be prioritizing the log reading above Keccak and other projects, to move towards the desired number.

Additionally, I plan to cease these semi-weekly progress reports, and simply read through the logs. (Simply, I'm not seeing much value besides informing me in what I already know: that I have a prioritization and effort-management problem). I'll also add some mechanism to the log summaries page to report statistics on entries created in the last week.

If you object to my approach, please voice your concerns below.

cl-keccak, Week 3

Monday, July 30th, 2018

With three weeks spent on the cl-keccak project, the thing is nearing completion. Currently there exists a (hopefully) working sponge, taking in an arbitrarily long vector of bits, and returning an equally arbitrarily long vector of bits. And after fixing two bugs in last week's post, my state transitions match those of s.mg keccak.

However, cl-keccak does not seem to agree with s.mg keccak on return hashes. For instance, if we place1

Long live the Republic!

in a file, ascii encoded, cl-keccak reports the hash as

18B75F434B28705333D18C57627085596EBF2AF68F52D3CF5796C01EAA682D4D5F259D2F89F9AEE082BC577B4DBEB8882D5F125DCE5FC977F701817487F06078

where the raw binary data (reported by xxd -b) is

00000000: 01001100 01101111 01101110 01100111 00100000 01101100  Long l
00000006: 01101001 01110110 01100101 00100000 01110100 01101000  ive th
0000000c: 01100101 00100000 01110010 01100101 01110000 01110101  e repu
00000012: 01100010 01101100 01101001 01100011 00100001 00001010  blic!.

Whereas s.mg keccak (taken from vtools) reports

dc3c4ed4bcbdc55d0917cbea51272f30b0d0dfa5d029100871e383be39568cec30afd298878a7d7f51ce5bad6bc01d651adf1ac5355652cf623d55b9d7da97dc

for the same file. So, I'm going to put the current state of my work below, and attempt to hash out the details this week in the logs.

As for parameters, cl-keccak uses

  • 6 for the Keccak L parameter
  • 1600 for the width
  • 1344 for the bitrate
  • 512 for the output bits

The primary function, keccak-sponge, takes a bit-vector as input and returns a bit-vector as output. At this level, there are no endianness issues whatsoever.

When interfacing with the disk, cl-keccak uses read-byte, which on my machine reads bits in 8 at a time, returning an integer as an (unsigned-byte 8). cl-keccak then converts this integer into a bit-vector which is 8 bits long, creating a little-endian representation of the integer. Ideally, I would like be able to read from the disk a single bit at the time, but as far as I am aware, this is not possible (and not meaningful on a byte-addressed architecture). Hence cl-keccak relies on the system to consistently interpret stored bytes as the same integer on differing hardware architectures.

Finally for this week, here is the current state of the program2:

package.lisp:

(defpackage "CL-KECCAK"
  (:use "COMMON-LISP" "CL-USER")
  (:export keccak-sponge))

bits.lisp:

(in-package "CL-KECCAK")

(defun bit-chunk (bit-vector chunk-size)
  (assert (= 0 (mod (length bit-vector) chunk-size)))
  (let ((chunks '()))
    (dotimes (c (/ (length bit-vector) chunk-size))
      (setq chunks (append chunks (list (subseq bit-vector
                                                (* c chunk-size)
                                                (* (1+ c) chunk-size))))))
    chunks))

(defun bit-pad-right (bv n)
  (do ((x (coerce bv 'list) (append x '(0))))
      ((>= (length x) n)
       (coerce x 'simple-bit-vector))))

(defun bit-pad-right-and-chunk (bit-vector chunk-size)
  (bit-chunk (bit-pad-right bit-vector
                            (* chunk-size (ceiling (length bit-vector)
                                                   chunk-size)))
             chunk-size))

(defun concatenate-bit-vectors (bit-vector-list)
  (reduce #'(lambda (a b) (concatenate 'simple-bit-vector a b))
          bit-vector-list))

(defun bit-vector-to-integer (bv)
  (reduce #'(lambda (a b) (+ a (* 2 b)))
          bv
          :from-end t))

(defun integer-to-bit-vector (n)
  (labels ((bit-array-iter (n array)
             (if (zerop n)
                 array
                 (multiple-value-bind (q r)
                     (floor n 2)
                   (bit-array-iter q
                                   (append array (list r)))))))
    (bit-pad-right (bit-array-iter n '()) 8)))

(defun file-to-bit-vector (filepath)
  (with-open-file (f filepath :direction :input :element-type 'bit)
    (concatenate-bit-vectors
     (mapcar #'integer-to-bit-vector
             (let ((s (make-sequence 'list (file-length f))))
               (read-sequence s f)
               s)))))

cl-keccak.lisp:

(in-package "CL-KECCAK")

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; configure the keccak_L parameter here

(defconstant +keccak_L+ 6)


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; magic numbers

(defconstant +row-size+ 5)
(defconstant +column-size+ 5)
(defconstant +lane-size+ (expt 2 +keccak_L+))
(defconstant +keccak-width+ (* +row-size+ +column-size+ +lane-size+))
(defconstant +round-quantity+ (+ 12 (* 2 +keccak_L+)))

(defparameter *default-bitrate* 1344)
(defparameter *default-output-bits* 512)


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; lanes

;; definition of a lane
(deftype lane () `(simple-bit-vector ,+lane-size+))

;; instantiation lanes
(defun make-lane ()
  (make-sequence 'lane +lane-size+ :initial-element 0))

(defun copy-lane (lane)
  (make-array `(,+lane-size+) :element-type 'bit
              :initial-contents lane))

;; basic operations on lanes
(defun lane-and (a b)
  (declare (type lane a b))
  (bit-and a b))

(defun lane-xor (a b)
  (declare (type lane a b))
  (bit-xor a b))

(defun lane-not (a)
  (declare (type lane a))
  (bit-not a))

(defun lane-rot (a n)
  (let* ((rtn (make-lane)))
    (dotimes (z +lane-size+)
      (setf (aref rtn (mod (+ z n) +lane-size+))
            (aref a z)))
    rtn))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; magic lanes

(defconstant +round-constants+
  (let ((magic
         '(#*1000000000000000000000000000000000000000000000000000000000000000
           #*0100000100000001000000000000000000000000000000000000000000000000
           #*0101000100000001000000000000000000000000000000000000000000000001
           #*0000000000000001000000000000000100000000000000000000000000000001
           #*1101000100000001000000000000000000000000000000000000000000000000
           #*1000000000000000000000000000000100000000000000000000000000000000
           #*1000000100000001000000000000000100000000000000000000000000000001
           #*1001000000000001000000000000000000000000000000000000000000000001
           #*0101000100000000000000000000000000000000000000000000000000000000
           #*0001000100000000000000000000000000000000000000000000000000000000
           #*1001000000000001000000000000000100000000000000000000000000000000
           #*0101000000000000000000000000000100000000000000000000000000000000
           #*1101000100000001000000000000000100000000000000000000000000000000
           #*1101000100000000000000000000000000000000000000000000000000000001
           #*1001000100000001000000000000000000000000000000000000000000000001
           #*1100000000000001000000000000000000000000000000000000000000000001
           #*0100000000000001000000000000000000000000000000000000000000000001
           #*0000000100000000000000000000000000000000000000000000000000000001
           #*0101000000000001000000000000000000000000000000000000000000000000
           #*0101000000000000000000000000000100000000000000000000000000000001
           #*1000000100000001000000000000000100000000000000000000000000000001
           #*0000000100000001000000000000000000000000000000000000000000000001
           #*1000000000000000000000000000000100000000000000000000000000000000
           #*0001000000000001000000000000000100000000000000000000000000000001)))
    (make-array '(24)
                :element-type 'lane
                :initial-contents
                (mapcar #'(lambda (x) (subseq x 0 +lane-size+))
                        magic))))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; states

(deftype state () `(array lane (,+row-size+ ,+column-size+)))

(defun make-state ()
  (make-array `(,+row-size+ ,+column-size+)
              :element-type 'lane
              :initial-element (make-lane)))

(defun lane (a x y)
  (declare (type state a)
           (type fixnum x y))
  (aref a (mod x +row-size+) (mod y +column-size+)))

(defmethod set-lane (a x y L)
  (setf (aref a (mod x +row-size+) (mod y +column-size+))
        L))

(defsetf lane set-lane)

(defun copy-state (state)
  (let ((s (make-array `(,+row-size+ ,+column-size+)
                       :initial-element (make-lane)
                       :element-type 'lane)))
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (setf (lane s x y)
              (copy-lane (lane state x y)))))
    s))

(defun state-linearize (state &optional
                                (start 0)
                                (end +keccak-width+))
  (let ((r '()))
    (dotimes (y +column-size+)
      (dotimes (x +row-size+)
        (setq r (append r (list (lane state x y))))))
    (subseq (concatenate-bit-vectors r)
            start
            end)))

(defun state-xor (state bit-vector)
  (assert (<= (length bit-vector) +keccak-width+))
  (let ((s (copy-state state))
        (x 0)
        (y 0)
        (chunks (bit-pad-right-and-chunk bit-vector +lane-size+)))
    (dolist (c chunks)
      (setf (lane s x y)
            (lane-xor (lane state x y)
                      c))
      (setf x (mod (1+ x) +row-size+))
      (if (= 0 (mod x +row-size+))
          (setf y (mod (1+ x) +column-size+))))
    s))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; keccak round operations

(defmacro with-return-state (s &rest body)
  `(let ((,(first s) (if (= ,(length s) 2)
                         (copy-state ,(second s))
                         (make-state))))
     (progn ,@body)
     ,(first s)))

(defun theta (a)
  (with-return-state (b)
    (let* ((c (make-sequence '(vector lane) +row-size+
                             :initial-element (make-lane)))
           (d (make-sequence '(vector lane) +row-size+
                             :initial-element (make-lane))))
      (dotimes (x +row-size+)
        (setf (aref c x)
              (lane a x 0))
        (loop for y from 1 below +column-size+
           do (setf (aref c x)
                    (lane-xor (aref c x)
                              (lane a x y)))))
      (dotimes (x +row-size+)
        (setf (aref d x)
              (lane-xor (aref c (mod (- x 1) +row-size+))
                        (lane-rot (aref c (mod (+ x 1) +row-size+))
                                  1)))
        (dotimes (y +column-size+)
          (setf (lane b x y)
                (lane-xor (lane a x y)
                          (aref d x))))))))

(defun rho (a)
  (with-return-state (b)
    (setf (lane b 0 0) (lane a 0 0))
    (let ((x 1) (y 0))
      (dotimes (q 24)
        (setf (lane b x y)
              (lane-rot (lane a x y)
                        (/ (* (+ q 1)
                              (+ q 2))
                           2)))
        (psetq x y
               y (+ (* 2 x)
                    (* 3 y)))))))

(defun k-pi (a)
  (with-return-state (b)
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (setf (lane b y (+ (* 2 x)
                           (* 3 y)))
              (lane a x y))))))

(defun chi (a)
  (with-return-state (b)
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (setf (lane b x y)
              (lane-xor (lane a x y)
                        (lane-and (lane-not (lane a (+ x 1) y))
                                  (lane a (+ x 2) y))))))))

(defun iota (r a)
  (with-return-state (b a)
    (setf (lane b 0 0)
          (lane-xor (lane b 0 0)
                    (aref +round-constants+ r)))))

(defun keccak-permute (a)
  (with-return-state (b a)
    (dotimes (r +round-quantity+)
      (setq b (iota r (chi (k-pi (rho (theta b)))))))))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; sponge

(defun keccak-pad (bit-vector bitrate)
  (let ((bv-tmp (concatenate 'simple-bit-vector bit-vector #*1)))
    (concatenate 'simple-bit-vector
                 (bit-pad-right bv-tmp (1- (* bitrate
                                              (1+ (floor (length bv-tmp)
                                                         bitrate)))))
                 #*1)))

(defun keccak-absorb (bit-vector bitrate)
  (assert (<= bitrate +keccak-width+))
  (with-return-state (s)
    (dolist (c (bit-chunk (keccak-pad bit-vector bitrate) bitrate))
      (setq s (keccak-permute (state-xor s c))))))

(defun keccak-squeeze (state bitrate output-bit-quantity)
  (assert (<= bitrate +keccak-width+))
  (multiple-value-bind (full-blocks remaining-bits)
      (floor output-bit-quantity bitrate)
    (concatenate-bit-vectors
     (mapcar (lambda (x)
               (setq state (keccak-permute state))
               (state-linearize state 0 x))
             (append (make-sequence 'list full-blocks
                                    :initial-element bitrate)
                     (if (zerop remaining-bits)
                         '()
                         (list remaining-bits)))))))

(defun keccak-sponge (input-bit-vector &optional
                                         (bitrate *default-bitrate*)
                                         (output-bits *default-output-bits*))
  (keccak-squeeze (keccak-absorb input-bit-vector bitrate)
                  bitrate
                  output-bits))

(defun keccak-hash-file (filepath)
  (write-to-string (bit-vector-to-integer (keccak-sponge (file-to-bit-vector filepath)
                                                         *default-bitrate*
                                                         *default-output-bits*))
                   :base 16))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; for use as an executable

(defun main ()
  (let ((args #+sbcl (cdr sb-ext:*posix-argv*)
              #+ccl (cdr ccl:*command-line-argument-list*)))
    (princ (keccak-hash-file (first args)))))

cl-keccak-tests.lisp:

(in-package "CL-KECCAK")

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; utility functions for moving between integers and lanes
;; these are not used in cl-keccak proper. rather, they are
;; for testing and generating the round constants


(defun bit-truncate-right (bv n)
  (subseq bv 0 n))

(defun integer-to-lane (n)
  (labels ((bit-array-iter (n array)
             (if (zerop n)
                 array
                 (multiple-value-bind (q r)
                     (floor n 2)
                   (bit-array-iter q
                                   (append array (list r)))))))
    (bit-truncate-right (bit-pad-right (bit-array-iter n '())
                                       +lane-size+)
                        +lane-size+)))

(defun lane-to-integer (bv)
  (reduce #'(lambda (a b) (+ a (* 2 b)))
          bv
          :from-end t))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; code used to generate the magic lanes.
;; this code is retained for user verification.
;; numbers used to generate the lanes can be found here:
;; https://keccak.team/keccak_specs_summary.html#roundConstants

;; (defparameter *magic-constants*
;;   (make-array '(24)
;;               :element-type 'lane
;;               :initial-contents
;;               (mapcar #'integer-to-lane
;;                       '(#x0000000000000001
;;                         #x0000000000008082
;;                         #x800000000000808a
;;                         #x8000000080008000
;;                         #x000000000000808b
;;                         #x0000000080000001
;;                         #x8000000080008081
;;                         #x8000000000008009
;;                         #x000000000000008a
;;                         #x0000000000000088
;;                         #x0000000080008009
;;                         #x000000008000000a
;;                         #x000000008000808b
;;                         #x800000000000008b
;;                         #x8000000000008089
;;                         #x8000000000008003
;;                         #x8000000000008002
;;                         #x8000000000000080
;;                         #x000000000000800a
;;                         #x800000008000000a
;;                         #x8000000080008081
;;                         #x8000000000008080
;;                         #x0000000080000001
;;                         #x8000000080008008))))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; procedures for making states

(defun make-state-using-lane-generator (nullary-lane-generator)
  (with-return-state (a)
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (setf (lane a x y) (funcall nullary-lane-generator))))))

(defun make-ascending-state ()
  (make-state-using-lane-generator
   (let ((b -1))
     (lambda ()
       (integer-to-lane (incf b))))))

(defun make-randomized-state ()
  (make-state-using-lane-generator
   (lambda () (integer-to-lane (random (expt 2 +lane-size+))))))

(defun make-bit-ascending-state ()
  (make-state-using-lane-generator
   (let ((b 0))
     (lambda () (integer-to-lane (if (= b 0) (incf b) (setq b (expt 2 b))))))))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; procedures for testing states

(defun diff-states (state1 state2)
  (let ((diff '()))
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (if (not (equal (lane state1 x y)
                        (lane state2 x y)))
            (setq diff (append diff (list (cons x y)))))))
    diff))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; printing the state

(defun print-state (a &optional (stream t))
  (let ((fmt-str (format nil "~~{~~~d,'0X~~^ ~~}"
                         (max 0 (/ +lane-size+ 4)))))
    (dotimes (y +column-size+)
      (format stream fmt-str (mapcar #'(lambda (x) (lane-to-integer (lane a x y)))
                                     '(0 1 2 3 4)))
      (format stream "~%"))))

(defun print-readable-state (a &optional (stream t))
  (let ((fmt-str (format nil "(~~{#x~~~d,'0X~~^ ~~})"
                         (max 0 (/ +lane-size+ 4)))))
    (format stream "#2A(")
    (dotimes (x +row-size+)
      (format stream fmt-str (mapcar #'(lambda (y) (lane-to-integer (lane a x y)))
                                     '(0 1 2 3 4)))
      (format stream
              (if (= x (1- +row-size+)) ")" "~%    ")))))

(defun read-printed-state (stream)
  (with-return-state (b)
    (let ((a (read stream)))
      (dotimes (x +row-size+)
        (dotimes (y +column-size+)
          (setf (lane b x y)
                (integer-to-lane (aref a x y))))))))

(defun print-keccak-permute (state stream)
  (format stream ";; Initial state:~%")
  (print-state state stream)
  (let ((maps `(("theta" . ,#'theta)
                ("rho" . ,#'rho)
                ("pi" . ,#'k-pi)
                ("chi" . ,#'chi))))
    (dotimes (r +round-quantity+)
      (let ((maps (append maps `(("iota" . ,#'(lambda (a) (iota r a)))))))
        (format stream "~%~%~%;; Round ~d~%~%" r)
        (dolist (m maps)
          (format stream "~%;; After ~a:~%" (car m))
          (print-state (setq state
                             (funcall (cdr m) state))
                       stream)))))
  (format stream "~%~%~%;; Final state:~%")
  (print-state state stream))

(defun test-permute-and-write (state output-file)
  (with-open-file (s output-file :direction :output :if-exists :supersede)
    (print-keccak-permute state s)))

Makefile:

LISP=sbcl

all: ${LISP}

sbcl:
    sbcl --no-sysinit --no-userinit --disable-debugger \
        --load package.lisp \
        --load bits.lisp \
        --load cl-keccak.lisp \
        --eval "(sb-ext:save-lisp-and-die #p\"cl-keccak\" :toplevel #'cl-keccak::main :executable t)"

ccl:
    ccl --no-init \
        --load package.lisp \
        --load bits.lisp \
        --load cl-keccak.lisp
        --eval "(ccl:save-application #P\"cl-keccak\" :toplevel-function #'cl-keccak::main :prepend-kernel t)"
  1. no trailing newline []
  2. I am not placing these as files or signing them on purpose. This weeks release is very likely broken []

Log Reading, Week 3

Wednesday, July 25th, 2018

So this week's post is late. But that lateness, coupled with my continuing inability to keep up with the log reading pace, has started to get some nice lessons through my thick skull!1

This week we are up to 75 entries, 11 more than the last. While I honestly believe the past two weeks were far more busy for fiat esthlos than usual, I'm still planning for the worst. So the new plan is two per day, which should be far more sustainable. And anyways, the plan for my reading to speed up simply didn't happen. As Mircea intuits, the logs are information dense2 , and the brain can only process at such and such a rate. My attempts to read faster simply resulted in confusion.

Next week I'll try giving some highlights in the log reading update. Until then!

  1. Here are a few. (1). It's not enough to vaguely plan things out and hope I get done what needs to be done. I need to lay out everything that needs to be accomplished well before the deadline. (2) Things need to be prioritized. Just because one Emacs keybinding is broken doesn't mean that I need to deal with it now. (3) There must be time allocated to the unanticipated and fuckups in general. (4) Plans and scheduling follow through to the end. If I just plan to finish my weekly log reading on Sunday, then guess what: there won't be time for a post! Truly, it's remarkable that I managed to get this far in life while not understanding these kind of things! []
  2. Though I have read enough at this point to have an opinion on this density, and I think an unfortunate that much of it results from people not considering the minds of others and making unnecessary assumptions on all kinds of things. []

cl-keccak 2 Week Progress Report

Tuesday, July 24th, 2018

Two weeks ago, I determined that the writing of a Republican1 Common Lisp Keccak is the best path forward for esthlos-v. So what has the past two weeks bought us?

Well, it has bought two files. The first contains procedures to define a Keccak state and permute it in the ordained methods:

(defpackage "CL-KECCAK"
  (:use "COMMON-LISP" "CL-USER"))

(in-package "CL-KECCAK")

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; configure the keccak_L parameter here

(defconstant +keccak_L+ 6)


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; magic numbers

(defconstant +row-size+ 5)
(defconstant +column-size+ 5)
(defconstant +lane-size+ (expt 2 +keccak_L+))
(defconstant +round-quantity+ (+ 12 (* 2 +keccak_L+)))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; lanes

(deftype lane () `(simple-bit-vector ,+lane-size+))

(defun make-lane ()
  (make-sequence 'lane +lane-size+ :initial-element 0))

(defun lane-and (a b)
  (declare (type lane a b))
  (bit-and a b))

(defun lane-xor (a b)
  (declare (type lane a b))
  (bit-xor a b))

(defun lane-not (a)
  (declare (type lane a))
  (bit-not a))

(defun lane-rot (a n)
  (let* ((rtn (make-lane)))
    (dotimes (z +lane-size+)
      (setf (aref rtn (mod (+ z n) +lane-size+))
            (aref a z)))
    rtn))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; magic lanes

(defconstant +round-constants+
  (let ((magic
         '(#*1000000000000000000000000000000000000000000000000000000000000000
           #*0100000100000001000000000000000000000000000000000000000000000000
           #*0101000100000001000000000000000000000000000000000000000000000001
           #*0000000000000001000000000000000100000000000000000000000000000001
           #*1101000100000001000000000000000000000000000000000000000000000000
           #*1000000000000000000000000000000100000000000000000000000000000000
           #*1000000100000001000000000000000100000000000000000000000000000001
           #*1001000000000001000000000000000000000000000000000000000000000001
           #*0101000100000000000000000000000000000000000000000000000000000000
           #*0001000100000000000000000000000000000000000000000000000000000000
           #*1001000000000001000000000000000100000000000000000000000000000000
           #*0101000000000000000000000000000100000000000000000000000000000000
           #*1101000100000001000000000000000100000000000000000000000000000000
           #*1101000100000000000000000000000000000000000000000000000000000001
           #*1001000100000001000000000000000000000000000000000000000000000001
           #*1100000000000001000000000000000000000000000000000000000000000001
           #*0100000000000001000000000000000000000000000000000000000000000001
           #*0000000100000000000000000000000000000000000000000000000000000001
           #*0101000000000001000000000000000000000000000000000000000000000000
           #*0101000000000000000000000000000100000000000000000000000000000001
           #*1000000100000001000000000000000100000000000000000000000000000001
           #*0000000100000001000000000000000000000000000000000000000000000001
           #*1000000000000000000000000000000100000000000000000000000000000000
           #*0001000000000001000000000000000100000000000000000000000000000001)))
    (make-array '(24)
                :element-type 'lane
                :initial-contents
                (mapcar #'(lambda (x) (subseq x 0 +lane-size+))
                        magic))))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; states

(deftype state () `(array lane (,+row-size+ ,+column-size+)))

(defun make-state ()
  (make-array `(,+row-size+ ,+column-size+)
              :element-type 'lane
              :initial-element (make-lane)))

(defun lane (a x y)
  (declare (type state a)
           (type fixnum x y))
  (aref a (mod x +row-size+) (mod y +column-size+)))

(defmethod set-lane (a x y L)
  (setf (aref a (mod x +row-size+) (mod y +column-size+))
        L))

(defsetf lane set-lane)


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; keccak round operations

(defmacro with-return-state (s &rest body)
  `(let ((,(first s) ,(if (= (length s) 2)
                          (second s)
                          (make-state))))
     (progn ,@body)
     ,(first s)))

(defun theta (a)
  (with-return-state (b)
    (let* ((c (make-sequence '(vector lane) +row-size+
                             :initial-element (make-lane)))
           (d (make-sequence '(vector lane) +row-size+
                             :initial-element (make-lane))))
      (dotimes (x +row-size+)
        (setf (aref c x)
              (lane a x 0))
        ;;(format t "~a~%" (aref c x))
        (loop for y from 1 below +column-size+
           do (setf (aref c x)
                    (lane-xor (aref c x)
                              (lane a x y)))))
      ;;(format t "~{~a~%~^~}~%" (coerce (copy-seq c) 'list))
      (dotimes (x +row-size+)
        (setf (aref d x)
              (lane-xor (aref c (mod (- x 1) +row-size+))
                        (lane-rot (aref c (mod (+ x 1) +row-size+))
                                  1)))
        (dotimes (y +column-size+)
          (setf (lane b x y)
                (lane-xor (lane a x y)
                          (aref d x))))))))


(defun rho (a)
  (with-return-state (b)
    (let ((x 1) (y 0))
      (dotimes (q 24)
        (setf (lane b x y)
              (lane-rot (lane a x y)
                        (/ (* (+ q 1)
                              (+ q 2))
                           2)))
        (psetq x y
               y (+ (* 2 x)
                    (* 3 y)))))))

(defun consts ()
  (let ((a (make-array '(5 5)))
        (b (make-array '(5 5))))
    (let ((x 1) (y 0))
      (dotimes (q 24)
        (setf (aref a (mod x 5) (mod y 5))
              (/ (* (+ q 1)
                    (+ q 2))
                 2))
        (psetq x y
               y (+ (* 2 x)
                    (* 3 y)))))
    (dotimes (x 5)
      (dotimes (y 5)
        (setf (aref b x y)
              (aref a (mod (+ x 3) 5) (mod (+ y 3) 5)))))
    b))

(defun pi (a)
  (with-return-state (b)
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (setf (lane b x y)
              (lane a y (+ (* 2 x)
                           (* 3 y))))))))

(defun chi (a)
  (with-return-state (b)
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (setf (lane b x y)
              (lane-xor (lane a x y)
                        (lane-and (lane-not (lane a (+ x 1) y))
                                  (lane a (+ x 2) y))))))))

(defun iota (r a)
  (with-return-state (b a)
    (setf (lane b 0 0)
          (lane-xor (lane b 0 0)
                    (aref +round-constants+ r)))))

(defun keccak-permute (a)
  (with-return-state (b a)
    (dotimes (r +round-quantity+)
      (setq b (iota r (chi (pi (rho (theta b)))))))))

You will notice that the code defines lanes as simple-bit-vectors, and there is no interpretation of these lanes as numbers in any sense. Hence endianness troubles are eliminated, at the cost of a linear-time vector rotation2 .

A second file contains a few goodies, including methods for converting integers to lanes (little-endian) and back again, and methods for visualizing the Keccak state:

(in-package "CL-KECCAK")

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; utility functions for moving between integers and lanes
;; these are not used in cl-keccak proper. rather, they are
;; for testing and generating the round constants

(defun bit-pad-right (bv n)
  (do ((x (coerce bv 'list) (append x '(0))))
      ((>= (length x) n)
       (coerce x 'simple-bit-vector))))

(defun bit-truncate-right (bv n)
  (subseq bv 0 n))

(defun integer-to-lane (n)
  (labels ((bit-array-iter (n array)
             (if (zerop n)
                 array
                 (multiple-value-bind (q r)
                     (floor n 2)
                   (bit-array-iter q
                                   (append array (list r)))))))
    (bit-truncate-right (bit-pad-right (bit-array-iter n '())
                                       +lane-size+)
                        +lane-size+)))

(defun lane-to-integer (bv)
  (reduce #'(lambda (a b) (+ a (* 2 b)))
          bv
          :from-end t))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; code used to generate the magic lanes.
;; this code is retained for user verification.
;; numbers used to generate the lanes can be found here:
;; https://keccak.team/keccak_specs_summary.html#roundConstants

;; (defparameter *magic-constants*
;;   (make-array '(24)
;;               :element-type 'lane
;;               :initial-contents
;;               (mapcar #'integer-to-lane
;;                       '(#x0000000000000001
;;                         #x0000000000008082
;;                         #x800000000000808a
;;                         #x8000000080008000
;;                         #x000000000000808b
;;                         #x0000000080000001
;;                         #x8000000080008081
;;                         #x8000000000008009
;;                         #x000000000000008a
;;                         #x0000000000000088
;;                         #x0000000080008009
;;                         #x000000008000000a
;;                         #x000000008000808b
;;                         #x800000000000008b
;;                         #x8000000000008089
;;                         #x8000000000008003
;;                         #x8000000000008002
;;                         #x8000000000000080
;;                         #x000000000000800a
;;                         #x800000008000000a
;;                         #x8000000080008081
;;                         #x8000000000008080
;;                         #x0000000080000001
;;                         #x8000000080008008))))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; procedures for making states

(defun make-state-using-lane-generator (nullary-lane-generator)
  (with-return-state (a)
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (setf (lane a x y) (funcall nullary-lane-generator))))))

(defun make-ascending-state ()
  (make-state-using-lane-generator
   (let ((b -1))
     (lambda ()
       (integer-to-lane (incf b))))))

(defun make-randomized-state ()
  (make-state-using-lane-generator
   (lambda () (integer-to-lane (random (expt 2 +lane-size+))))))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; procedures for testing states

(defun diff-states (state1 state2)
  (let ((diff '()))
    (dotimes (x +row-size+)
      (dotimes (y +column-size+)
        (if (not (equal (lane state1 x y)
                        (lane state2 x y)))
            (setq diff (append diff (list (cons x y)))))))
    diff))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; printing the state

(defun print-state (a &optional (stream t))
  (let ((fmt-str (format nil "~~{~~~d,'0X~~^ ~~}"
                         (max 0 (/ +lane-size+ 4)))))
    (dotimes (x +row-size+)
      (format stream fmt-str (mapcar #'(lambda (y) (lane-to-integer (lane a x y)))
                                     '(0 1 2 3 4)))
      (format stream "~%"))))

(defun print-readable-state (a &optional (stream t))
  (let ((fmt-str (format nil "(~~{#x~~~d,'0X~~^ ~~})"
                         (max 0 (/ +lane-size+ 4)))))
    (format stream "#2A(")
    (dotimes (x +row-size+)
      (format stream fmt-str (mapcar #'(lambda (y) (lane-to-integer (lane a x y)))
                                     '(0 1 2 3 4)))
      (format stream
              (if (= x (1- +row-size+)) ")" "~%    ")))))

(defun read-printed-state (stream)
  (with-return-state (b)
    (let ((a (read stream)))
      (dotimes (x +row-size+)
        (dotimes (y +column-size+)
          (setf (lane b x y)
                (integer-to-lane (aref a x y))))))))

(defun print-keccak-permute (state stream)
  (format stream ";; Initial state:~%")
  (print-readable-state state stream)
  (let ((maps `(("theta" . ,#'theta)
                ("rho" . ,#'rho)
                ("pi" . ,#'pi)
                ("chi" . ,#'chi))))
    (dotimes (r +round-quantity+)
      (let ((maps (append maps `(("iota" . ,#'(lambda (a) (iota r a)))))))
        (format stream "~%~%~%;; Round ~d~%~%" r)
        (dolist (m maps)
          (format stream "~%;; After ~a:~%" (car m))
          (print-readable-state (setq state
                                      (funcall (cdr m) state))
                                stream)))))
  (format stream "~%~%~%;; Final state:~%")
  (print-readable-state state stream))

(with-open-file (s "tests.txt" :direction :output :if-exists :supersede)
  (print-keccak-permute (make-state) s))

Let's have some fun with tests. Here are the first three rounds of operating on the null state:

;; Initial state:
#2A((#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000))


;; Round 0


;; After theta:
#2A((#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000))
;; After rho:
#2A((#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000))
;; After pi:
#2A((#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000))
;; After chi:
#2A((#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000))
;; After iota:
#2A((#x0000000000000001 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000))


;; Round 1


;; After theta:
#2A((#x0000000000000001 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000001 #x0000000000000001 #x0000000000000001 #x0000000000000001 #x0000000000000001)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000002 #x0000000000000002 #x0000000000000002 #x0000000000000002 #x0000000000000002))
;; After rho:
#2A((#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000002 #x0000000000000000 #x0000000000000400 #x0000200000000000 #x0000000000000004)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000)
    (#x0000000010000000 #x0000000000200000 #x0000010000000000 #x0000000000000200 #x0000000000008000))
;; After pi:
#2A((#x0000000000000000 #x0000200000000000 #x0000000000000000 #x0000000000000000 #x0000010000000000)
    (#x0000000000000000 #x0000000000000002 #x0000000000000000 #x0000000000000000 #x0000000000008000)
    (#x0000000000000000 #x0000000000000400 #x0000000000000000 #x0000000000000000 #x0000000000200000)
    (#x0000000000000000 #x0000000000000004 #x0000000000000000 #x0000000000000000 #x0000000000000200)
    (#x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000000000000 #x0000000010000000))
;; After chi:
#2A((#x0000000000000000 #x0000200000000400 #x0000000000000000 #x0000000000000000 #x0000010000200000)
    (#x0000000000000000 #x0000000000000006 #x0000000000000000 #x0000000000000000 #x0000000000008200)
    (#x0000000000000000 #x0000000000000400 #x0000000000000000 #x0000000000000000 #x0000000010200000)
    (#x0000000000000000 #x0000200000000004 #x0000000000000000 #x0000000000000000 #x0000010000000200)
    (#x0000000000000000 #x0000000000000002 #x0000000000000000 #x0000000000000000 #x0000000010008000))
;; After iota:
#2A((#x0000000000008082 #x0000200000000400 #x0000000000000000 #x0000000000000000 #x0000010000200000)
    (#x0000000000000000 #x0000000000000006 #x0000000000000000 #x0000000000000000 #x0000000000008200)
    (#x0000000000000000 #x0000000000000400 #x0000000000000000 #x0000000000000000 #x0000000010200000)
    (#x0000000000000000 #x0000200000000004 #x0000000000000000 #x0000000000000000 #x0000010000000200)
    (#x0000000000000000 #x0000000000000002 #x0000000000000000 #x0000000000000000 #x0000000010008000))


;; Round 2


;; After theta:
#2A((#x000000001001048C #x000020001001800E #x000000001001840E #x000000001001840E #x000001001021840E)
    (#x0000210020608C82 #x0000210020608C84 #x0000210020608C82 #x0000210020608C82 #x0000210020600E82)
    (#x000042000000860E #x000042000000820E #x000042000000860E #x000042000000860E #x000042001020860E)
    (#x0000000030210404 #x0000200030210400 #x0000000030210404 #x0000000030210404 #x0000010030210604)
    (#x0000630000410B00 #x0000630000410B02 #x0000630000410B00 #x0000630000410B00 #x0000630010418B00))
;; After rho:
#2A((#x0000000000000000 #x001800E000020001 #x00000000800C2070 #x03081C0000000020 #x0400408610380000)
    (#x0000420040C11904 #x0000000000000000 #x0084008182320800 #x119040000420040C #x0000840081803A08)
    (#x8000108000002183 #x0010800000208380 #x0430700002100000 #x2100000043070000 #xC0000840020410C1)
    (#x0302104040000000 #x0000001000181082 #x0060420808000000 #x0006042080800000 #x0400000100302106)
    (#x0002085800000318 #x30000410B0200006 #x2085800000318000 #x00630000410B0000 #x18C0041062C00000))
;; After pi:
#2A((#x0000000000000000 #x119040000420040C #x0010800000208380 #x0400000100302106 #x2085800000318000)
    (#x00000000800C2070 #x0000420040C11904 #x2100000043070000 #x0000001000181082 #x18C0041062C00000)
    (#x0400408610380000 #x0084008182320800 #x8000108000002183 #x0006042080800000 #x30000410B0200006)
    (#x001800E000020001 #x0000840081803A08 #x0430700002100000 #x0302104040000000 #x00630000410B0000)
    (#x03081C0000000020 #x0000000000000000 #xC0000840020410C1 #x0060420808000000 #x0002085800000318))
;; After chi:
#2A((#x0400408610300000 #x111440818612040C #x801090800020A203 #x0406042180B02106 #x0085800090118006)
    (#x00180060800E2071 #x0000C60041412B0C #x2530600041170000 #x0300105040181082 #x18A3041023CB0000)
    (#x07005C8610380020 #x0084008182320800 #x400018C000043142 #x0066462888800000 #x30000C48B020031E)
    (#x001800E000020001 #x1190C40085A03E04 #x0420F00002308300 #x0702104140302106 #x20E68000413A8000)
    (#x03081C00800C2050 #x0000020040C11900 #xE1000840410310C1 #x0060421808081080 #x18420C4862C00318))
;; After iota:
#2A((#x840040861030808A #x111440818612040C #x801090800020A203 #x0406042180B02106 #x0085800090118006)
    (#x00180060800E2071 #x0000C60041412B0C #x2530600041170000 #x0300105040181082 #x18A3041023CB0000)
    (#x07005C8610380020 #x0084008182320800 #x400018C000043142 #x0066462888800000 #x30000C48B020031E)
    (#x001800E000020001 #x1190C40085A03E04 #x0420F00002308300 #x0702104140302106 #x20E68000413A8000)
    (#x03081C00800C2050 #x0000020040C11900 #xE1000840410310C1 #x0060421808081080 #x18420C4862C00318))

For comparison, here are the first three rounds of the empty state package with Diana Coman's Keccak:

Same, with lanes as 64-bit words:
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000

 -- Round 0 --

After theta:
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
After rho:
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
After pi:
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
After chi:
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
After iota:
0000000000000001 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000

 -- Round 1 --

After theta:
0000000000000001 0000000000000001 0000000000000000 0000000000000000 0000000000000002
0000000000000000 0000000000000001 0000000000000000 0000000000000000 0000000000000002
0000000000000000 0000000000000001 0000000000000000 0000000000000000 0000000000000002
0000000000000000 0000000000000001 0000000000000000 0000000000000000 0000000000000002
0000000000000000 0000000000000001 0000000000000000 0000000000000000 0000000000000002
After rho:
0000000000000001 0000000000000002 0000000000000000 0000000000000000 0000000010000000
0000000000000000 0000100000000000 0000000000000000 0000000000000000 0000000000200000
0000000000000000 0000000000000400 0000000000000000 0000000000000000 0000010000000000
0000000000000000 0000200000000000 0000000000000000 0000000000000000 0000000000000200
0000000000000000 0000000000000004 0000000000000000 0000000000000000 0000000000008000
After pi:
0000000000000001 0000100000000000 0000000000000000 0000000000000000 0000000000008000
0000000000000000 0000000000200000 0000000000000000 0000200000000000 0000000000000000
0000000000000002 0000000000000000 0000000000000000 0000000000000200 0000000000000000
0000000010000000 0000000000000000 0000000000000400 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000010000000000 0000000000000000 0000000000000004
After chi:
0000000000000001 0000100000000000 0000000000008000 0000000000000001 0000100000008000
0000000000000000 0000200000200000 0000000000000000 0000200000000000 0000000000200000
0000000000000002 0000000000000200 0000000000000000 0000000000000202 0000000000000000
0000000010000400 0000000000000000 0000000000000400 0000000010000000 0000000000000000
0000010000000000 0000000000000000 0000010000000004 0000000000000000 0000000000000004
After iota:
0000000000008083 0000100000000000 0000000000008000 0000000000000001 0000100000008000
0000000000000000 0000200000200000 0000000000000000 0000200000000000 0000000000200000
0000000000000002 0000000000000200 0000000000000000 0000000000000202 0000000000000000
0000000010000400 0000000000000000 0000000000000400 0000000010000000 0000000000000000
0000010000000000 0000000000000000 0000010000000004 0000000000000000 0000000000000004

 -- Round 2 --

After theta:
0000700000600487 0000130010018C89 0000700020208606 000021000041840D 0000320030018B01
0000700000608404 0000230010218C89 0000700020200606 000001000041840C 0000220030210B01
0000700000608406 0000030010018E89 0000700020200606 000021000041860E 0000220030010B01
0000700010608004 0000030010018C89 0000700020200206 000021001041840C 0000220030010B01
0000710000608404 0000030010018C89 0000710020200602 000021000041840C 0000220030010B05
After rho:
0000700000600487 0000260020031912 80001C0008082181 00041840D0000210 01800C5808000190
0608404000070000 18C8900002300102 001C000808018180 06000000800020C2 20030210B0100002
0003800003042030 000C0040063A2400 0030300003800101 0000830C1C000042 0085808000110018
C100080000E00020 3191200000600200 3800101001030000 2002083081800004 00220030010B0100
C400018210100001 00000C0040063224 40000E20040400C0 0C00002100004184 08800C0042C14000
After pi:
0000700000600487 18C8900002300102 0030300003800101 2002083081800004 08800C0042C14000
00041840D0000210 20030210B0100002 0003800003042030 3191200000600200 40000E20040400C0
0000260020031912 001C000808018180 0000830C1C000042 00220030010B0100 C400018210100001
01800C5808000190 0608404000070000 000C0040063A2400 3800101001030000 0C00002100004184
80001C0008082181 06000000800020C2 0085808000110018 C100080000E00020 00000C0040063224
After chi:
0030500001E00486 38CA983082300106 08B0340041C14101 2002783081A00483 10488C0040D14100
00049840D3042220 11932210B0700202 40038E20070020F0 31953040D0600010 60030C30241400C2
0000A50434031950 003E0038090A8080 C400828E0C100043 0022263021081812 C41C018A18108081
01840C580E382590 3E08505001060000 040C0061063A6584 39801C4809030010 0A08402100074184
80859C8008192199 C700080080E020E2 008584804017321C 4100180008E801A1 06000C00C0063266
After iota:
8030500001E0840C 38CA983082300106 08B0340041C14101 2002783081A00483 10488C0040D14100
00049840D3042220 11932210B0700202 40038E20070020F0 31953040D0600010 60030C30241400C2
0000A50434031950 003E0038090A8080 C400828E0C100043 0022263021081812 C41C018A18108081
01840C580E382590 3E08505001060000 040C0061063A6584 39801C4809030010 0A08402100074184
80859C8008192199 C700080080E020E2 008584804017321C 4100180008E801A1 06000C00C0063266

(Note that Diana's states are transposes of mine, as she chose to use the y coordinate as the row index when printing). The careful reader will notice that my implementation is failing on rho of row 1; for instance, the (0,0) coordinate is set from 1 to 0, which is in clear violation of the reference. Well, I haven't figured out where my rho is going wrong. Perhaps you, reader, would lend a hand?

Edit: After discovering the problem in rho (no initial assignment of A[0][0] = a[0][0]) and also discovering a problem with pi (mixing up the X and Y with x and y), cl-keccak now exactly aligns with the Eucrypt test vectors! On to the sponge construction!

  1. meaning that transparency, correctness, and clarity of purpose are held above all else. []
  2. If I'm missing a constant-time vector shift operation, please let me know []

Log Reading, Week 2

Monday, July 16th, 2018

64 entries! The summaries project is 20 entries short for this week! What in the world happened?

Well, I could blame things on by fiat job, but we know where that leads. No: I knew these things were coming up; I knew that I would have to prioritize ruthlessly to make the summary target1 this week; and yet, I didn't.

So this week becomes another instance where I set a difficult yet surmountable bar, didn't bother to organize myself to meet that bar, and fell far short. As to the why, there are all kinds of answers, being the usual crowd of laziness, stupidity, or lack of will, but what does one do about these? I only hope that the more fearful part of the brain gets the message that my reputation is on the line, a lesson ill-learned in the land of socialist safety nets and participation awards2.

  1. all the while, chugging along at a Common Lisp Keccak. Which yes, has made good progress. Be on the lookout for a release within the next two weeks. []
  2. also known as diplomas []

Log Reading, Week 1

Sunday, July 8th, 2018

Last Sunday brought the public launch of the Trilema log summary project. The goal is to read 4 logs per day, to which I've had to add the practical cap of 2 hours per day1. I also have concern that much needed projects will eat into the log reading time. Certainly, all of this is a continuing lesson in priorities and pacing over time.

Currently the summaries exist for 54 dates. The target for this week will be 28 entries, 21 of them of older logs, bringing the project somewhere mid-April 2016, and the total to 82. I'm anticipating the reality of ending up off-target, and reassessing how many logs I can commit to per week over the next month or so.

There's also the matter of an esthlos-bot for reading out entries. The call could be something like

!e 2018 7 6 2

to print out the second entry of the 2018-07-06 log summary (that is, July 6th, 2018). It would be nice to also print a link to the log item in question, which could also facilitate tracking long-term conversations in the logs, with the cost of many additional cross-reference links.

In addition to the bot, a search feature will be needed as the summaries grow. I would expect the search to work through the bot as well as on the page, spitting out a text-paste with results.

The bot and search will have to wait, though, while I explicitly prioritize getting esthlos-v working properly.

esthlos-bot, esthlos-v...I need some better names!

  1. Generally I don't expect the 2 hour cap to be reached, and I also anticipate some speedup in reading as the project progresses. March 2016 has so far been a month of megalogs, coinciding with the the shifts occurring in the Republic at the time. []

Routes to Keccak in esthlos-v

Sunday, July 8th, 2018

For reasons amply put in the logs, esthlos-v needs a hashing mechanism, and the standard is Keccak. How can Keccak be attached to the Common Lisp esthlos-v? Really, it comes down to:

  1. Call out to a pre-existing Keccak implementation; or
  2. Write a Keccak implementation for Common Lisp.

Let's consider the options.

Option 1: Call out to a pre-existing Keccak implementation

The only Republican-signed Keccak implementation is smg-keccak, a piece of the EuCrypt library by Diana Coman. How could the smg-keccak be incorporated into esthlos-v? Well, again, there are two options:

  1. Make a new patch with esthlos-v_genesis and some node of the EuCrypt tree as parents.
  2. Build EuCrypt, rip out the non-Keccak code, and patch this code on to esthlos-v.

To 1: Some trouble with the smg-keccak patching prevents simply selecting a subset of the EuCrypt patchset to build only smg-keccak. Either the majority of EuCrypt must be pulled in, or a new patch must be created off of the smg-keccak branch to correct the trouble. I want esthlos-v to be as small1 and simple2 as possible, so pulling in all of EuCrypt is off the table. Creating a trivial new patch, though, is feasible; let's keep this in the back of our minds for now.

To 2: The trouble with this method is how it drops the history built into the EuCrypt smg-keccak patchset, introducing a heap of problems. For instance, dropping the history renders the best resource for understanding the code, Diana's excellent posts, a pain to match to the code. Basically, this approach smells rotten, and will not be pursued.

Details of smg-keccak incorporation aside, what would follow from said incorporation? smg-keccak is written in Ada, so the user of esthlos-v would need a working Ada build environment. If understanding of the code is desired, a working understanding of Ada would be needed as well. Besides, calling out to a common library is a flavor of dynamic soup, to be avoided when possible.

Outside of Ada, a quick search revealed an already existing Keccak for Common Lisp, written by some dweeb.3. Of course, to be signed, it must be deloused and understood, so while it can serve as a useful reference, it can be used as only that: a reference towards a new implementation.4

Option 2: Write a Keccak implementation for Common Lisp

The pluses of having a Common Lisp Keccak are mostly obvious. The use of a single language simplifies things, cutting down on both the requirements needed to get the thing running5 , and the knowledge needed to see how the thing works.

What about the downsides? Well, really the significant downside is the time and effort required to understand the operation of Keccak and implement it. But if smg-keccak was to be pulled into esthlos-v, then that would required a patch, which would need to be signed,6 and so the effort of understanding is unavoidable.7

So in the large, while the incorporation of smg-keccak into the esthlos-v might be the faster and less wasteful option, neither route to the incorporation outweighs the costs. Destroying the vtree structure creates the the "wtf is this, how does it work, and how did it come to be" problem, and branching off the EuCrypt tree still pulls in Ada as a dependency. The most natural cut, then, appears to be a Common Lisp Keccak.

Comments from the Lordship appreciated.

  1. "The practice is pervaded by the reassuring illusion that programs are just devices like any others, the only difference admitted being that their manufacture might require a new type of craftsmen, viz. programmers. From there it is only a small step to measuring "programmer productivity" in terms of "number of lines of code produced per month". This is a very costly measuring unit because it encourages the writing of insipid code, but today I am less interested in how foolish a unit it is from even a pure business point of view. My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger." -Dijkstra []
  2. fits-in-head is a hard requirement []
  3. Warning: the choice of font may melt eyeballs []
  4. And there are those of us stuck in USGland, where the licensing of the thing is an issue. []
  5. aka the number of things which can break []
  6. An outstanding question of mine is whether signing a vpatch is a statement about the transformation of code outlined in the patch, or the state of the codebase once the patch is applied. []
  7. As usual. []

Trilema Log Reading

Sunday, July 1st, 2018

Reading through the logs of The Most Serene Republic, though rewarding and enjoyable, is not for the faint of heart. I'm often left wondering how anyone can make sense of the conversation, but certainly there are glimmers of real experts under the surface of madness.

Anyway, I've read enough to know that I want to keep reading---not only forwards, but backwards. So I built myself a little infrastructure, to develop the bigger view of what's going on:

TMSR Log Summaries

Not sure if it'll be of use to anyone but myself, but if I get a few "that wasn't about X, it was about Y!" spankings out of it, I'll consider it worthwhile.

My attempted pace will be to read 4 days of logs every day (3 older logs, and the present log). The average log takes me about 30 minutes to read (including following links), and there's a bit over 900 logs to catch up on, so I'll be caught up in 300 days, working 2 hours a day. Starting late is a bitch, ya know.