Improvement: better memory usage in dry-monitor

Ideas
1

After testing a dry-web-roda app with the gem derailed_benchmarks I found that my app was using more memory than a bigger Rails app.

Actually, this is the first part of the output:

$ bundle exec derailed bundle:mem
TOP: 42.4297 MiB
  dry-web: 13.6484 MiB
    dry/web/container: 13.6484 MiB
      dry/monitor: 13.4023 MiB
        dry/monitor/sql/logger: 12.9414 MiB
          rouge: 12.9414 MiB

That means, that the app is using 42 MiB, and the dry-web component is the biggest one, that is using 13.6 MiB, because of rouge (12.9 MiB).

So, requiring only the needed components of rouge, by changing in lib/dry/monitor/sql/logger.rb the line 2:

require 'rouge'

with:

require 'rouge/util'
require 'rouge/token'
require 'rouge/theme'
require 'rouge/themes/gruvbox'
require 'rouge/formatter'
require 'rouge/formatters/terminal256'
require 'rouge/lexer'
require 'rouge/regex_lexer'
require 'rouge/lexers/sql'

We can get an improvement. The mem test after that returns:

$ bundle exec derailed bundle:mem
TOP: 26.1211 MiB
  rom: 8.0352 MiB
    ...
  dry-web: 1.1211 MiB
    dry/web/container: 1.1211 MiB
      dry/monitor: 0.75 MiB
        dry/monitor/sql/logger: 0.5 MiB

So, the memory used by dry-monitor was reduced from 13.4 MiB to 0.75 Mib. And the impact on the memory usage of a small dry-web-roda app was bigger than that: from 42.4297 MiB to 26.1211 MiB.

I did not analyze how derailed_benchmarks make its maths. But it’s clear that rouge has a lot of cool components that are not used by dry-monitor. Then, Why do not get rid of it?

Please, let me know what you think about this. Thank you.

2 Likes
2

Oh wow, I did not expect that. I’m all for limiting requires to what you actually need. In fact, I’m a strong believer that all gems should be designed in a way that they always provide cherry-pickable components.

Would you mind sending a PR to dry-monitor? :slight_smile:

2 Likes
3

@solnic also, I checked my job project for object allocation and found interesting things:

$ bundle exec derailed bundle:objects

Measuring objects created by gems in groups [:default, "production"]
Total allocated: 91115563 bytes (800628 objects)
Total retained:  15093944 bytes (134684 objects)

allocated memory by gem
-----------------------------------
  10664845  rouge-2.2.1
   8722637  ruby-2.4.2/lib
   8059914  newrelic_rpm-4.2.0.334
   6488996  mime-types-3.1
   4489137  dry-initializer-2.3.0

As you can see, rouge allocate more objects raise than other dependencies. And also, this dependency needs only for color output. Sometimes it can break production (when logger services doesn’t support the color format, for example).

That’s why I have a question: WDYT if we make rouge dependency as optional? I mean if a user wants to use rouge they can install the gem and that’s all. Also, a user gets control on color/not color logs for any environments.

Something like this (check logger file):

begin
  require 'rouge'

  module Dry
    module Monitor

      module SQL
        class Logger
          # use rouge lexer and other stuff
        end
      end
    end
  end
rescue LoadError
  module Dry
    module Monitor

      module SQL
        class Logger
          # logger without color formatter
        end
      end
    end
  end
end
1 Like
4

Yeah, it should be optional. I just wanted to have colors and put it there by default. Logger is configurable so it is very easy to add config setting for disabling colors.

5

Quick update, I reported an issue about it https://github.com/dry-rb/dry-monitor/issues/9 I also discovered that I actually made colorful output configurable already :sweat_smile: