If a line is completely empty, it is not useful to log it at all since
it would be invisible on the created logstream anyway. If we get passed
an empty String from the encoder, we can thus silently ignore it.
If an event's message is completely empty (usually because there was
nothing logged at all), it isn't very useful to output this bare marked
up message at all. We thus ignore it and pass it along unchanged in the
Message encoder.
Note that we still add tags for messages consisting of other whitespace
like e.g. a newline added from a formatted empty message added to a
Logger.
The middleware can be used in a Rack appliction wrap all log messages
emitted to the logger during a single request in a single emitted log
event. This ensures that all data concerning the request, including log
messages as well as additional fields and tags are logged as one single
event.
This ensures that the data is kept as a whole when the log event is
handled by later systems like Logstash. Each request can be analyzed as
a whole without having to group or parse complex multi-line log formats.
With this, the Buffer knows three buffering modes:
* :full - This is the same as the previous buffering-enabled mode. With
this, we buffer everything and never auto-flush
* :none - the previous non-buffering mode. We autoflush everytime there
is a new messase or explicitly added fields. All stored data is
cleared afterwards.
* :data - the new mode. It behaves almost like :none with the notable
exception that we retain fields and tags after the auto flush.
The new mode is especially useful to emulate a regular Logger even when
using per-request buffers. With that, you can add fields once to a
buffer. Each time a message is added, it will be flushed directly
without waiting for the request to be finished. Yet, the flows can still
take advantage of all the previously added fields and tags to properly
format the emitted log event.
The rational implementation will be laoded automatically by Ruby.
Attempting to load rational.rb (in Ruby 2.1) will cause unintended
consequences for the whole running program.
This fizes a Ruby warning generated when running the tests for
Rackstash::Adaptes::Logger. The warning was:
> warning: instance variable @closed not initialized
The Sink was a vehicle to transport some shared state between the logger
instance and the buffers, most notably the default fields and default
tags.
It turns out however, that error handling during merging of the
default_fields and default_tags is non trivial since there, the buffer
is in sime kind of limbo: users won't write to it anymore (and thus
don't expect exceptions there) while the error handling of the
individual flows is not yet reached. Since users can specify procs in
default_fields, they can still raise for whatever user-defined reason.
Luckily, the insertion of default fields and default tags can easily be
done by a filter later anyway, under the protection of the flow's error
handling in Flow#write. This allows us just remove the whole concept of
the sink and just pass the Flows object to the Buffer.
Not having to merge default_fields during event creation significantly
simplifies Buffer#to_event which was rather ugly to begin with but now
turned out quite beatifully.
If messages are removed by the final cut step (but not when applying the
selectors), we insert an ellipsis, by default a Message with the
contents of "[...]\n", at the place where we removed messages. This
helps the investigating human who reads the logs to understand that
there were logs messages removed from the event.
This can be used to achieve similar results as
ActiveSupport::TaggedLogger. With our implementation, the "tags" are
prepended to each line (instead of each message) and we add the same
tags to all message lines.
If required, you can still use ActiveSupport::TaggedLogger with
Rackstash to add tags at the time of adding the log message.
If we have a proc, we can pass it directly to the `select!` method
instead of having to manually call the selector. This gives a
significant speedup for the common case of simple selectors:
require 'benchmark/ips'
Benchmark.ips do |x|
selector = ->(value) { value < 500 }
values = (1..1000).to_a
x.report("call") { values.select { |v| selector.call(v) } }
x.report("block") { values.select(&selector) }
x.compare!
end
Warming up --------------------------------------
call 663.000 i/100ms
block 1.598k i/100ms
Calculating -------------------------------------
call 6.897k (± 4.8%) i/s - 34.476k in 5.011617s
block 16.561k (± 4.2%) i/s - 83.096k in 5.026999s
Comparison:
block: 16560.7 i/s
call: 6896.5 i/s - 2.40x slower
Any public method is part of the public interface of the class and thus
needs to be equally well supported. Only because methods are not usually
called by end-users, we still need to properly document, test, and check
them.
This resolves the following warning:
> spec/rackstash/encoders/logstash_spec.rb:18: warning: ambiguous first
> argument; put parentheses or a space even after `/' operator
This allows to configure the last (and often only) flow on creation of
the Logger without having to manually create the object. We can thus use
the following shortcut to e.g. set a custom encoder:
Rackstash::Logger.new(STDOUT) do
encoder Rackstash::Encoder::Message.new
end
