Fixing Emacs library loading with a virtual filesystem

My previous post about speeding up the library search in Emacs was mostly a hack, but I think I have a more radical solution. So, the key problem with slow library loading in Emacs is that it does lots of separate syscalls for every module it tries to load because it checks every item in load-path.

And I've just remembered that there's support for "virtual filesystems" in Emacs where you can set up a handler for a file path with special prefix like /myfs: and the reading and writing will be handled via user-provided functions. In theory, if I pre-load the filename "index" into memory, I won't have to do even a single syscall apart from opening the final file. No probing or anything like that.

So I decided to check if Emacs would really be able to require libraries from a virtual filesystem:

(setq myfs-files
      '(("/myfs:/lib/test-lib.el" . "(provide 'test-lib)\n(defun hello () (message \"Hello from memory!\"))")))

(defun myfs-handler (operation &rest args)
  (message "Handling %s with %S" operation args)

  (let ((file (car args)))
    (cond
     ;; Provide virtual content for insert-file-contents
     ((and (eq operation 'insert-file-contents)
           (assoc file myfs-files))
      (let ((buf (current-buffer))
            (content (cdr (assoc file myfs-files))))
        (with-current-buffer buf
          (erase-buffer)
          (insert content)
          (setq buffer-file-name file))
        (list file (length content))))

     ((and (eq operation 'file-exists-p)
           (assoc file myfs-files))
      t)

     ((and (eq operation 'file-readable-p)
           (assoc file myfs-files))
      t)

     ((and (eq operation 'file-truename)
           (assoc file myfs-files))
      file)

     (t
      (let ((inhibit-file-name-handlers
             (cons 'myfs-handler
                   (and (eq (car inhibit-file-name-handlers) 'myfs-handler)
                        (cdr inhibit-file-name-handlers))))
            (inhibit-file-name-operation operation))
        (apply operation args))))))


(add-to-list 'file-name-handler-alist '("^/myfs:" . myfs-handler))

(add-to-list 'load-path "/myfs:/lib/")


(require 'test-lib)

And it looks like it can! When I execute the last line, the (hello) function gets loaded from string residing in myfs-files. Now, building upon this it should be possible to create an in-memory index. Stay tuned.