Separating `nil` from value never given in dry-struct

Support dry-types
1

:wave: friends TGIF!

I am wondering if anyone has solved the following problem as I have failed to do so both in dry-* and sorbet. I’d like to name this problem the “PATCH problem” :tm:

Let’s assume we have the following a users table with
name → not nullable string
dob → (date of birth) nullable string
gender → nullable string

In a typical PATCH request the user gets to update whatever the provided in the request body only, therefore we should not update fields that are missing.

The loosely typed nature of hashes has an advantage over types as an input: there is a clear distinction between a missing input and a null input. Let’s see an example.

(1) The following request body is relatively clear, we want to update the two fields.

{"user": {"name":  "Meredith", "dob":  "1968-01-01", "gender": "Female"}}

(2) With this body, we just want to nullify the dob.

{"user": {"dob": null}}

(3) With this body, we just want to update the name

{"user": {"name":  "Dwight"}}

Let’s represent the above as a type

class User < Dry::Struct
  attribute :name, String
  attribute :dob, Types::String.optional
  attribute :gender, Types::String.optional
end

And now we have an issue at hand. Looking at an instance of a user, how do you know dob is null because scenario 2 or 3? You cannot really distinguish.

It might be worth mentioning that I want to craft a domain type or even a typed input from the controller before I pass it in the service / domain layer.

In Haskell or Elm for instance, the above can be represented as

attribute :dob, Unchanged | Deleted | Updated a

I wonder if anyone has come across this before and has a workable solution. For instance, a simple but not great implementation would be

class User < Dry::Struct
  attribute :name, String
  attribute :dob, Types::String.optional
  attribute :dob_given?, Bool
  # ...
  attribute :gender, Types::String.optional
end

Another example is having a specific struct for each possibility(Expressing the type of attribute B as dependent on an arbitrary value of attribute A.) but that can explode relatively quickly.

where we explicitly ask the crafter of the Struct to tell us if the input is actually nil because the user provided nil or never given.

Bonus points: one extra worry I have is how would any solution play with conversion to other types like my_struct.to_hash.

2

I think the fact that you are trying to model this as a User is the whole problem. It’s not a user, it’s a patch to user data. That is an entirely different data object.

A struct is not meant to be instantiated unless its data is validated and correct. It’s not appropriate to build this struct until you are actually patching the data into the integrated whole.

This makes me think it would be nice to have something along the lines of TypeScript’s Partial<T> utility type. It would be useful to pass a schema into something that transforms all required keys into optional.

The notion of having a single, god-like object for the abstract concept of a thing like User is a bad abstraction, mental baggage from ActiveRecord in my case. I have been far more successful designing my data objects as use cases.

3

hey thanks for getting back to me!

I think my post was not clear enough so let me be a bit more concrete in what I am asking. Let’s see an actual example

class UsersController < ApplicationController
  def update
   # Let's call this a command
    Users::Update.call(..)
  end
end

This is a typical action that I want to add 2 layers to:

The first one is something similar to dry-rails as a replacement for strong parameters. This layer validates the shape of the request. Here the request can be in the form of json:api which is a bit more complex than a simple json api where you have {"user": {...}".

Once I am happy with the shape and data I have received, I want to craft something reasonable the Users::Update understands which is input (HTTP, Kafka, etc) agnostic. I don’t want to tie my code to a particular input, passing the raw json:api would make the command unusable. In the controller layer, I can craft a hash based on the params that satisfies my commands expectation. This works fine and this is how my app works right now.

Now I have the following problem however. Since Users::Update receives a hash, I can no longer trust the input and I need a similar contract/schema validation in my service. You can imagine Users::Update can be called from various places and I need to be able to reveal intend to them as to what I expect in the hash and what are valid inputs (like constrains). dry-validations satisfies that very well. it’s what I am actually using on the controller layer.

A dry-struct satisfies the above because it cannot be an invalid object. So once the request has been validated I can convert that into a dry-struct. In my command, I don’t need to add extra validations to double check the shape. And this is how we arrive at my issue. With hashes I could tell if the user wanted to nullify something or not. With a struct is not that easy.

The added benefit is also around code completion and working with an object is less error-prone than hashes.

To be clear since you mentioned a god-like object, that’s not the intention in any way. The type for the example can be UpdateableUser or UserParams or whatever.

I hope this clears things up.

Update: an example of the same problem but completely different environment is protobufs and how google suggest to handle partial updates AIP-134: Standard methods: Update. From the guidelines:

  • If partial resource update is supported, a field mask must be included. It must be of type google.protobuf.FieldMask

in essence, if I understand it correctly, you should describe in this mask field which fields are being updated.

4

Yes, your use-case is substantially more complicated than I originally thought. I agree that json:api structure should not leave the controller layer, you clearly need a patch object of some kind.

I think you can have the best of both worlds with a tiny bit of metaprogramming. User can still be authoritative for the whole object, but you can dynamically construct a less strict subset of keys. This will allow you to tell the difference between a missing key and an intentional null field.

def Patch(klass, attrs)
  parent_keys = klass.schema.keys
  patch_keys  = parent_keys.select { |key| attrs.has_key?(key.name) }

  patch_class = Class.new(Dry::Struct) do
    attributes parent_keys.map { |key| ["#{key.name}?", key.type.optional] }.to_h

    define_method(:__parent) { klass }
    define_method(:__field_mask) { attrs.keys }
    define_singleton_method(:name) { "#{klass.name}::Patch" }
  end

  patch_class.new(attrs)
end

Omittable keys are implemented internally as key.name with a ? suffix, usually you would declare them with attribute?.

Optional keys means the type allows a nil value. When you call #attributes on the struct, explicit nils will be present, but omitted keys will not.

5

that’s a nice implementation indeed of the a patched struct! I think unfortunately it has a sneaky behaviour that could lead to bugs

Screenshot 2022-09-28 at 09.58.21
Screenshot 2022-09-28 at 09.58.211859×947 160 KB

the only way to determine whether something was passed in, is to call a to_hash on it. but it is also reasonable or expected that you might also call the method itself which will return nil.

also I don’t really need this User object, we can define a new PatchedUser dry class that would have a more concrete implementation but it suffers from the same issue.

the more I think about it the more I am convinced this cannot be easily solved without any dangerous side-effects. I was hoping maybe to rely more on types but obviously there is only so much dry can help with this.

thanks for the help!

6

That was my reasoning for including the __field_mask attribute, which tells you which attributes are being patched