1
0
mirror of https://github.com/meineerde/rackstash.git synced 2025-10-17 14:01:01 +00:00

170 Commits

Author SHA1 Message Date
40aa74c5c0 Always add a @timestamp field in the logstash encoder, even if it was removed by a filter 2017-08-24 23:07:17 +02:00
7b06d053ca Create an error event in the Flow exactly as a Buffer would
That way, we ensure that the error_flow receives an expected event and
doesn;t have to deal with different data formats which might result in
additional (then quietly hidden) errors.
2017-08-24 23:07:17 +02:00
96aaaa0348 Set Encoder helper methods as private
That way, they don't pollute the public interface of the encoders
including the helpers modules.
2017-08-24 23:07:17 +02:00
3078bccafd Pass the raw Buffer timestamp through the event Hash
This allows filters and finally the encoder to use this without having
to first convert the timestamp (back) to a more suitable format.
2017-08-24 23:07:17 +02:00
99728842aa Pass the raw messages array in the event Hash to the encoder
The encoder is then responsible to format it as it pleases. Commonly,
encoders can use Rackstash::Encoders::Helpers::Message#normalize_message
to create a single combined message string.

If the encoder is not interested in using the message, it can jsut get
rid if it without incuring any overhead.
2017-08-22 00:02:10 +02:00
78466f9439 Set the Buffer's timestamp when setting a tag 2017-08-17 00:49:10 +02:00
021dbc256b Flush silent Buffers by default
With the change to allow ading fields to the Buffer with a Logger
interface in the previous commit, we should also flush those Buffers
with fields set that way by default.
2017-08-17 00:34:47 +02:00
013d0f7d92 Auto-flush the buffer when adding fields using Logger#add
With this interface, the user adds fields the same way they would add
messages. The implicit assumption here is that the buffer is handled the
same way. Thus, if the current buffer is non-buffering, we will
automatically flush it to the sink and clear it, just the same way as we
would have done it for a message.
2017-08-17 00:34:47 +02:00
cca00408e2 With allow_silent, also check the presence of an explicit timestamp for Buffer#pending? 2017-08-16 20:54:55 +02:00
d8300a125e Rename Buffer#allow_empty? to Buffer#allow_silent?
This new name better reflects what this option actually does: it allows
to flush a Buffer even if there were just added fields or tags. A tryely
empty Buffer, i.e., one which has neither added messages, fields, or
tags is never flushed.
2017-08-16 20:46:28 +02:00
0a5e0a4aab Add TruncateMessage filter to restrict the size of the logged message 2017-08-16 00:43:09 +02:00
ee1f5480c4 Remove useless assignments in message_spec 2017-08-15 23:45:54 +02:00
fa174bba9d Freeze the raw object along the wrapper for all fields 2017-08-15 18:36:33 +02:00
57f1197299 Do not alter the message in JSON encoder
If required, users can setup filters for that. We shouldnot assume any
overly specific use-cases here.
2017-08-14 23:25:59 +02:00
0a1fe46ea5 Add filter to remove all ANSI color codes from the event message 2017-08-14 23:18:19 +02:00
10993d2b1a Serialize Rational numbers to a String
With this, we are more compatible with the common JSON encoders,
including the one on RUby core.
2017-08-14 14:10:48 +02:00
ab595d031b Add Logstash encoder
This encoder is very similar to the JSON encoder. It handles a few
additional specifics for Logstash to be suotable for Logstash's json
input.
2017-08-14 13:33:52 +02:00
2fbdf3c1f5 Don't create the @version fields in events by default
This field is only relevant when actually sending JSON to Logstash.
Other formats don't need it. It is thus more desireable to only include
this field in a specific logstash JSON encoder.
2017-08-14 13:27:20 +02:00
536bb9a087 Ensure that the Formatter freezes the strings in all cases
Previously, the returned String would been frozen only in Rubies
understanding the frozen_literals magic comment.
2017-08-12 14:05:20 +02:00
168e62b63c Remove useless newlines 2017-08-12 14:02:03 +02:00
3a1f3686c0 Add Message#strip methods to strip whitespace from messages
These mehods work very similar to their String equivalents.
2017-08-08 22:26:34 +02:00
e999746f1e Add Message#gsub and Message#sub methods
These allow to create new Message objects with an updated message,
similar to `gsub` and `sub` of the core String.
2017-08-08 22:15:21 +02:00
eb93d70468 Add Message#copy_with to create an adjusted copy of an exising message
You can overwrite any (or none) of the existing fields to create a new
frozen copy of a message object.
2017-08-08 22:15:13 +02:00
011249bb75 Add Flows#first and Flows#last 2017-08-07 22:49:39 +02:00
eab3d16ec5 Fix minor code formatting issues 2017-08-05 23:53:01 +02:00
d362b7efaa Make Flow#unshift_filter the primary alias before filter_prepend to be more consistent 2017-08-05 23:29:54 +02:00
581b8b1819 Allow to insert a new filter defined by a symbol or String into a FilterChain 2017-08-05 23:29:54 +02:00
f0c65524a7 Build a new Filter from a String or Symbol 2017-08-05 23:29:54 +02:00
77675f65a6 Add Logger#push_buffer and Logger#pop_buffer for manual Buffer management
Most of the time, these methods should be used. They are only required
for special cases where the Buffer needs to be flushed later than when
its poped. In this case, special precautions need to be made to ensure
that the Buffer is always reliably poped and flushed to avoid loosing
logs.
2017-08-05 00:05:41 +02:00
a987e92250 Add BufferStack#pop to remove the top-most Buffer without flushing it
This can be used for certain advanced constructs where the Buffer needs
to be manually flushed later, after it was already removed from the
stack.
2017-08-04 14:49:10 +02:00
3f8745e32c Move the responsiblity for creating the event hash to the Buffer
With this move, we can also optimize the sinplest case where there are
no fields or tags on the Buffer and no defines default_fields or
default_tags on the Sink. In that case, we don't need to merge them,
avoiding the creation of several unecessary objects on Buffer flush.
2017-08-04 00:53:36 +02:00
4438bf42bc Lazy instanciate tags and fields on a Buffer instance 2017-08-03 22:31:48 +02:00
697641635b Don't use Concurrent::{Hash|Array} objects in Fields
Since the fields are (through the thread-local BufferStack) only ever
accessed from a single Thread, there is no need to accept the additional
locking overhead of the Concurrent raw values.

We can just use simple Hashes and Arrays here for higher performance.
2017-08-03 21:09:50 +02:00
271c547f05 Initialize monitor and copy internal filters list on FilterChain#dup 2017-08-02 17:50:53 +02:00
6c145579ed Add FilterChain#to_a to get a plain Array representation of the chain 2017-08-02 17:37:29 +02:00
597f605e06 Don't stub FilterChain#to_s in test of #inspect
That way, we can also ensure that the nested locking works correctly
without any deadlock.
2017-08-02 17:32:10 +02:00
f6c5930db6 Use __dir__ instead of deriving from __FILE__ where suitable 2017-08-02 16:06:21 +02:00
b228494d14 Raise a custom Rackstash::NotImplementedHereError instead of NotImplementedError
Using the core `NotInmplementedError` is not desireable since its
documentation includes:

> Note that if `fork` raises a `NotImplementedError`, then
> `respond_to?(:fork)` returns `false`.

Since we are responding to the method but still raise an error, our
usage of the exception does not fulfill its documentation.

A custom error instead of a default `NoMethodError` is still desireable
since it significantly helps with debugging. With a different Exception,
we make it clear that a method is expected to be there and just wasn't
implemented by a subclass as opposed to the caller just using an object
wrong and calling entirely unexpected methods on it.
2017-08-01 12:28:58 +02:00
4eb381a733 Move Logger#add_exception to the Buffer and forward from the Logger 2017-07-28 16:58:51 +02:00
65cad497f0 Add complete tests for setting a custom error_flow on Flows 2017-07-28 12:59:46 +02:00
c4f0b9cdeb Use an error_flow to log any exceptions during logging
During normal operation, the Flows will rescue all exceptions and log
them to the special error_flow. By default, we will write JSON logs to
STDERR.

The log location and format can either be change globally by setting (or
changing) the Rackstash.error_flow or for each Flow for a Logger
individually by setting Flow#error_flow.
2017-07-28 02:16:31 +02:00
fb7732ef42 Extract close!, reopen!, and write! methods from Rackstash::Flow
These methods do not rescue any thrown errors. The usual loggers will
always want to use the non-bang methods which rescue errors and attempt
to log them.
2017-07-27 23:15:59 +02:00
be6e3517de Increase sub-second precision of timestamps to 6 decimal places 2017-07-25 23:32:14 +02:00
7a32781c17 Return the pop'ed Buffer from the BufferStack on flush_and_pop 2017-07-25 23:32:14 +02:00
f92bde5af2 Add Logger#[] and Logger#[]= alias methods to get and set fields 2017-07-25 00:03:47 +02:00
12261ef786 Fix Typo in spec description 2017-07-22 00:07:18 +02:00
9656297762 Remove ANSI color codes from the message in JSON encoder 2017-07-22 00:06:35 +02:00
667beb662f Freeze Messages directly after initialization
This ensures that they are actually immutable.
2017-07-21 22:50:44 +02:00
57f200ab35 Check that Logger#tag implements the same interface as Buffer#tag 2017-07-21 18:34:50 +02:00
9536b0473b Forward Logger#timestamp to Buffer#timestamp 2017-07-21 18:34:29 +02:00