Seeking guidance/code review for JSON marshaling

dvdsgl · March 10, 2018, 7:07pm

Hello! I’m writing a Ruby backend for quicktype and here’s my first proof of concept using dry-types and I was hoping for some feedback. Given the pokedex.json sample I generate this Ruby:

module Types
  include Dry::Types.module
  Egg = Types::String.enum("Not in Eggs", "Omanyte Candy", "10 km", "2 km", "5 km")
  Weakness = Types::String.enum("Bug", "Dark", "Dragon", "Electric", "Fairy", "Fighting", "Fire", "Flying", "Ghost", "Grass", "Ground", "Ice", "Poison", "Psychic", "Rock", "Steel", "Water")
end

module Egg
  NotInEggs    = "Not in Eggs"
  OmanyteCandy = "Omanyte Candy"
  The10KM      = "10 km"
  The2KM       = "2 km"
  The5KM       = "5 km"
end

class Evolution < Dry::Struct
  attribute :num,  Types::Strict::String
  attribute :name, Types::Strict::String

  def self.from_dynamic!(d)
    Evolution.new(
      num:  d["num"],
      name: d["name"],
    )
  end

  def self.from_json!(json)
    from_dynamic!(JSON.parse(json))
  end

  def to_dynamic
    {
      "num"  => @num,
      "name" => @name,
    }
  end

  def to_json(options = nil)
    JSON.generate(to_dynamic, options)
  end
end

module Weakness
  Bug      = "Bug"
  Dark     = "Dark"
  Dragon   = "Dragon"
  Electric = "Electric"
  Fairy    = "Fairy"
  Fighting = "Fighting"
  Fire     = "Fire"
  Flying   = "Flying"
  Ghost    = "Ghost"
  Grass    = "Grass"
  Ground   = "Ground"
  Ice      = "Ice"
  Poison   = "Poison"
  Psychic  = "Psychic"
  Rock     = "Rock"
  Steel    = "Steel"
  Water    = "Water"
end

class Pokemon < Dry::Struct
  attribute :id,             Types::Strict::Int
  attribute :num,            Types::Strict::String
  attribute :name,           Types::Strict::String
  attribute :img,            Types::Strict::String
  attribute :type,           Types.Array(Types::Strict::String)
  attribute :height,         Types::Strict::String
  attribute :weight,         Types::Strict::String
  attribute :candy,          Types::Strict::String
  attribute :candy_count,    Types::Strict::Int.optional
  attribute :egg,            Types::Egg
  attribute :spawn_chance,   Types::Strict::Decimal
  attribute :avg_spawns,     Types::Strict::Decimal
  attribute :spawn_time,     Types::Strict::String
  attribute :multipliers,    Types.Array(Types::Strict::Decimal).optional
  attribute :weaknesses,     Types.Array(Types::Weakness)
  attribute :next_evolution, Types.Array(Types.Instance(Evolution)).optional
  attribute :prev_evolution, Types.Array(Types.Instance(Evolution)).optional

  def self.from_dynamic!(d)
    Pokemon.new(
      id:             d["id"],
      num:            d["num"],
      name:           d["name"],
      img:            d["img"],
      type:           d["type"],
      height:         d["height"],
      weight:         d["weight"],
      candy:          d["candy"],
      candy_count:    d["candy_count"],
      egg:            d["egg"],
      spawn_chance:   d["spawn_chance"],
      avg_spawns:     d["avg_spawns"],
      spawn_time:     d["spawn_time"],
      multipliers:    d["multipliers"],
      weaknesses:     d["weaknesses"],
      next_evolution: d["next_evolution"]&.map { |x| Evolution.from_dynamic!(x) },
      prev_evolution: d["prev_evolution"]&.map { |x| Evolution.from_dynamic!(x) },
    )
  end

  def self.from_json!(json)
    from_dynamic!(JSON.parse(json))
  end

  def to_dynamic
    {
      "id"             => @id,
      "num"            => @num,
      "name"           => @name,
      "img"            => @img,
      "type"           => @type,
      "height"         => @height,
      "weight"         => @weight,
      "candy"          => @candy,
      "candy_count"    => @candy_count,
      "egg"            => @egg,
      "spawn_chance"   => @spawn_chance,
      "avg_spawns"     => @avg_spawns,
      "spawn_time"     => @spawn_time,
      "multipliers"    => @multipliers,
      "weaknesses"     => @weaknesses,
      "next_evolution" => @next_evolution&.map { |x| x.to_dynamic },
      "prev_evolution" => @prev_evolution&.map { |x| x.to_dynamic },
    }
  end

  def to_json(options = nil)
    JSON.generate(to_dynamic, options)
  end
end

class Pokedex < Dry::Struct
  attribute :pokemon, Types.Array(Types.Instance(Pokemon))

  def self.from_dynamic!(d)
    Pokedex.new(
      pokemon: d["pokemon"].map { |x| Pokemon.from_dynamic!(x) },
    )
  end

  def self.from_json!(json)
    from_dynamic!(JSON.parse(json))
  end

  def to_dynamic
    {
      "pokemon" => @pokemon.map { |x| x.to_dynamic },
    }
  end

  def to_json(options = nil)
    JSON.generate(to_dynamic, options)
  end
end

Is this the intended usage of dry-types? Am I doing any extra work?

My most immediate question is, is there any way for dry-types to coerce data from JSON.parse that is intended to correspond to nested structs? You can see in from_dynamic I have to map this dynamic value to Dry::Struct subtypes, etc. It seems like there’s enough information here already to make this automatic, but I wasn’t sure how.

Here’s another example. Given this weather data:

{
    "description": {
        "title": "Contiguous U.S., Average Temperature, January-December",
        "units": "Degrees Fahrenheit",
        "base_period": "1901-2000",
        "missing": -9999
    },
    "data": {
        "189512": {
            "value": "50.34",
            "anomaly": "-1.68"
        },
        "189612": {
            "value": "51.99",
            "anomaly": "-0.03"
        },
        "189712": {
            "value": "51.56",
            "anomaly": "-0.46"
        }
  }
}

I generate this Ruby:

module Types
  include Dry::Types.module
end

class Datum < Dry::Struct
  attribute :value,   Types::Strict::String
  attribute :anomaly, Types::Strict::String

  def self.from_dynamic!(d)
    Datum.new(
      value:   d["value"],
      anomaly: d["anomaly"],
    )
  end

  def self.from_json!(json)
    from_dynamic!(JSON.parse(json))
  end

  def to_dynamic
    {
      "value"   => @value,
      "anomaly" => @anomaly,
    }
  end

  def to_json(options = nil)
    JSON.generate(to_dynamic, options)
  end
end

class Description < Dry::Struct
  attribute :title,       Types::Strict::String
  attribute :units,       Types::Strict::String
  attribute :base_period, Types::Strict::String
  attribute :missing,     Types::Strict::Int

  def self.from_dynamic!(d)
    Description.new(
      title:       d["title"],
      units:       d["units"],
      base_period: d["base_period"],
      missing:     d["missing"],
    )
  end

  def self.from_json!(json)
    from_dynamic!(JSON.parse(json))
  end

  def to_dynamic
    {
      "title"       => @title,
      "units"       => @units,
      "base_period" => @base_period,
      "missing"     => @missing,
    }
  end

  def to_json(options = nil)
    JSON.generate(to_dynamic, options)
  end
end

class Weather < Dry::Struct
  attribute :description, Types.Instance(Description)
  attribute :data,        Types::Strict::Hash.meta(of: Types.Instance(Datum))

  def self.from_dynamic!(d)
    Pokedex.new(
      description: Description.from_dynamic!(d["description"]),
      data:        d["data"].map { |k, v| [k, Datum.from_dynamic!(v)] }.to_hash,
    )
  end

  def self.from_json!(json)
    from_dynamic!(JSON.parse(json))
  end

  def to_dynamic
    {
      "description" => @description.to_dynamic,
      "data"        => @data.map { |k, v| [k, v.to_dynamic] }.to_hash,
    }
  end

  def to_json(options = nil)
    JSON.generate(to_dynamic, options)
  end
end

Notice the manual hash marshaling for USTemperatures.data.
Is there a way to declare the :data attribute as a typed hash? You have Types.Array(t) but I could use Types.Hash(Types::String, Types.Instance(Datum).

Thank you for your help and suggestions!

dvdsgl · March 12, 2018, 4:06am

Just to follow up, I’ve published the project that generates this code and made a new post about it: Feedback wanted: autogenerated dry-types from JSON/Schema

flash-gordon · March 12, 2018, 9:22pm

For me serialization/deserialization is a separate concern. I would call .new directly if the data is in the right shape already and for everything else I would use https://github.com/solnic/transproc

Dry::Struct acts as a type and thus you don’t need to wrap it with Types.Instance, attribute :pokemon Types::Strict::Array.of(Pokemon) works just fine. If you pass a nested hash to .new it will create structs recursively, there’s no need for additional steps.

dvdsgl · March 12, 2018, 11:22pm

Wow, that’s great to hear!

What about automatic conversion to Hash<String, T>? JSON maps are all keyed by strings, and I can represent Types.Array.of(Pokemon) and it’s marshaled for me, but is there any way to do something like Types.Hash.of(keys: Types::String, values: Pokemon) for homogenous maps that are automatically marshaled, as Types::Array.of(Pokemon) is?

dvdsgl · March 13, 2018, 2:20am

Alright I think I figured out how to represent a homogeneous JSON-like map type:

module Types
  include Dry::Types.module

  class Map
    def self.of(definition)
      Types::Constructor(Hash) do |values|
        values.map { |k, v| [k.to_s, definition[v]] }.to_h
      end
    end
  end
end

class Things < Dry::Struct
  attribute :my_map, Types::Map.of(Types::String)
end

flash-gordon · March 13, 2018, 8:15pm

We’re working on Map in https://github.com/dry-rb/dry-types/pull/242, it’s pretty much ready but I’m going to cut off some rough edge after merging.

Topic		Replies	Views
Feedback wanted: autogenerated dry-types from JSON/Schema	0	841	March 12, 2018
Weird problem with validating sum types from JSON dry-validation	4	365	December 1, 2022
How to return an object in json to be a hash and not a string hash dry-types	3	2146	March 16, 2018
Customizing Hash dry-types	3	698	February 12, 2021
Suggestion: Simplify Dry::Structs type system	0	413	November 1, 2021

Seeking guidance/code review for JSON marshaling

Related topics