JSON at Rest

As has been mentioned many times, Irmin is fundamentally a key-value store. Thanks to its portability and flexibility both in storage backend and data format, Irmin is not the only means by which to interact with the data.

Storing JSON values

We can instantiate a simple in-memory Irmin store that stores JSON objects.

module Store = Irmin_mem.KV.Make (Irmin.Contents.Json)
let info () = Store.Info.v (Unix.gettimeofday () |> Int64.of_float)

The type of JSON objects is identical to that of the Ezjsonm library. The objects are association lists (lists of pairs where the first pair is a string, like a dictionary in other programming languages).

# #show Irmin.Contents.Json.t;;
val t : Store.contents Repr.ty
type nonrec t = (string * Irmin.Contents.json) list

This is very convenient, and we can quickly get and set values directly in the store using JSON-like OCaml values. The fact that the Ezjsonm representation of JSON values and the Irmin representation are the same is no coincidence, however, there is no strict dependency between the two so they could change in the future.

# let set_json_string_exn s k v =
  match Ezjsonm.value_from_string v with
  | `O assoc -> Store.set_exn ~info s k assoc
  | _ -> Lwt.fail (Failure "Expected a JSON object as a string");;
val set_json_string_exn : Store.t -> Store.path -> string -> unit Lwt.t =
  <fun>

From here we can now add JSON objects directly into the store.

# let config = Irmin_mem.config () in
  let* repo = Store.Repo.v config in
  let* main = Store.main repo in
  let* () = set_json_string_exn main [ "a" ] {|{ "hello": "world" }|} in
  let+ s = Store.get main [ "a" ] in
  print_endline @@ Ezjsonm.value_to_string (`O s);;
{"hello":"world"}
- : unit = ()

Custom Types Stored as JSON

One problem with using Irmin.Contents.Json.t is that we've lost the richness of the OCaml type system to a certain extent. This means it isn't obvious what are store is actually storing. Is it random JSON objects or a serialisation of a more rich OCaml value? If it is the latter, it probably isn't the interface we want.

For example, consider the following simple message datatype.

module type Message = sig
  type t = string [@@deriving irmin]

  include Irmin.Contents.S with type t := t
end

module Message : Message = struct
  type t = string [@@deriving irmin]

  let merge ~old:_ a b =
    match String.compare a b with
    | 0 ->
        if Irmin.Type.(unstage (equal t)) a b then
            Irmin.Merge.ok a
        else
            let msg = "Conflicting entries have the same timestamp but different values" in
            Irmin.Merge.conflict "%s" msg
    | 1 -> Irmin.Merge.ok a
    | _ -> Irmin.Merge.ok b
    
  let merge = Irmin.Merge.(option (v t merge))
end

By default if we create a store with this content type, the data will be stored using the string representation defined in repr. For the most part this is actually quite JSON-like.

# Irmin.Type.to_string Message.t "Hello World";;
- : string = "Hello World"

But there is an actual JSON-backend to the representation.

# Irmin.Type.to_json_string Message.t "Hello World";;
- : string = "\"Hello World\""

In fact for the most part the encoding does use JSON to format the OCaml values. The difference are usually very subtle, for example OCaml strings are just bytes whereas for JSON they must be UTF-8. So we can get very different formats (or as above where the JSON string requires the inverted-commas).

let _no_output_because_utf8 = Irmin.Type.to_string Message.t "\xc3\x28"

Whereas we must convert to a UTF-8 string that we can serialise and deserialise.

# Irmin.Type.to_json_string Message.t "\xc3\x28";;
- : string = "{\"base64\":\"wyg=\"}"

Fortunately, we can override the runtime representation of the type Irmin uses to store the values and keep the richness of the actual type when programming with the Irmin interface, but be serialising the data into JSON values. This is particularly useful, for example, with the Git.FS backend to read and write JSON values in Git stores.

module Message_json : Message = struct
  type t = string [@@deriving irmin]

  let merge ~old:_ a b =
    match String.compare a b with
    | 0 ->
        if Irmin.Type.(unstage (equal t)) a b then
            Irmin.Merge.ok a
        else
            let msg = "Conflicting entries have the same timestamp but different values" in
            Irmin.Merge.conflict "%s" msg
    | 1 -> Irmin.Merge.ok a
    | _ -> Irmin.Merge.ok b

  let t = Irmin.(Type.like ~pp:(Type.pp_json t) ~of_string:(Type.of_json_string t) t)
    
  let merge = Irmin.Merge.(option (v t merge))
end

Care with Serialisation of your Types

One thing to watch out for with serialisation of types is that it won't be applied recursively to all of your types. Consider the following defintion where we make an indirection via a type alise and forget we haven't converted the name_t to use JSON.

module Message_json = struct
  type name = string [@@deriving irmin]
  type t = name list [@@deriving irmin]

  let merge ~old:_ a b =
    match List.compare String.compare a b with
    | 0 ->
        if Irmin.Type.(unstage (equal t)) a b then
            Irmin.Merge.ok a
        else
            let msg = "Conflicting entries have the same timestamp but different values" in
            Irmin.Merge.conflict "%s" msg
    | 1 -> Irmin.Merge.ok a
    | _ -> Irmin.Merge.ok b

  let t = Irmin.(Type.like ~pp:(Type.pp_json t) ~of_string:(Type.of_json_string t) t)
    
  let merge = Irmin.Merge.(option (v t merge))
end

Now if we accidently used Message_json.name_t directly it won't be like Message_json.t.

# let msg = [ "\xc3\x28" ] in
  let v = Fmt.str "%s" Irmin.Type.(to_string Message_json.t msg) in
  let v' = Fmt.str "[%a]" Fmt.(list string) (List.map Irmin.Type.(to_string Message_json.name_t) msg) in
  v = v';;
- : bool = false

This example is pretty contrived, but it is meant to just show the problem rather than an exact real-world example.