1
0
mirror of https://github.com/meineerde/rackstash.git synced 2025-10-17 14:01:01 +00:00

279 Commits

Author SHA1 Message Date
f304f04ded Always run concurrency tests for File adapter with threads
Since fork is not available on some platforms (aka. Windows), we
avoid it and use normal Threads instead. We have fail saves in place to
ensure that tests are still valid even on MRI with its GIL.
2019-09-17 22:26:07 +02:00
c79f2e45d3 Close a tempfile before attemting to rename it in File specs
This is required on Windows which stubornly refused to rename opened
files by default.
2019-09-17 22:26:07 +02:00
68d674668a Explicitly remove temporary files in specs 2019-09-17 22:26:07 +02:00
b6be188dbd Use Logger#capture instead of with_buffer as a more intention revealing name 2019-01-14 11:51:05 +01:00
26a1ce0a68 Log the original event to the error_flow if Flow#write raises an error 2018-12-05 18:54:27 +01:00
0967593010 Perform actions on flows asynchronously by default
Each flow now has an associated executor which performs all actions
(writing events, closing, reopening) asynchronously by default using a
Concurrent::SingleThreadExecutor.

This improves the responsiveness of the application by performing the
(usually) IO-bound task of writing the logs to a background thread.

By creating a flow with `synchronous: true`, all actions are run in the
calling thread as before, making the flow blocking.
2018-12-05 18:51:29 +01:00
8a19dea76c Rename the DropIf filter to Drop and allow it to drop a percentage of events
With this, we also drop the ability to define conditions in the filter
itself. When adding a filter, users can still setup a condition using
the common functionality of all filters.
2018-08-28 10:33:20 +02:00
9152b67df0 Allow to pass initializer arguments to the encoder of a flow 2018-07-18 12:47:55 +02:00
a3f16b42cd Add attribute reader for tagged fields to Encoder::Message 2018-07-18 12:33:10 +02:00
b646ef9ef1 Add Rack::Errors#close to satisfy the expected protocol for a logdev of Ruby's core Logger 2018-07-17 22:53:54 +02:00
f97bcf2334 Add Flow#raise_on_error! as a convenience method 2018-07-17 22:49:32 +02:00
d510280532 Preserve existing field mappings for encoder fields 2018-07-17 21:39:37 +02:00
3ef0d56c2d Bug: Allow Flows#auto_flush if there are only auto_flushing flows 2018-07-12 18:20:34 +02:00
2c7d897889 Allow to set a registered encoder in a Flow by name 2018-07-12 18:02:09 +02:00
a807f99af5 Interpret a Date in the Gelf timestamp field as UTC midnight 2018-06-15 00:12:51 +02:00
916cabd43d Add Gelf encoder to create logs in the Graylog Extended Log Format 2018-06-14 23:54:36 +02:00
fcb1043b6f Correctly format date and time objects in Lograge encoder 2018-06-14 23:51:40 +02:00
5e9c76d6df Allow to rotate log files using a date-based pattern
We now support two different modes of file rotation at the same time:

* auto_reopen can be used to automatically reopen a logfile at the
  original location if the file was moved or deleted from the filesystem
* rotate can be used to write to a rotate file which can be reopened /
  created based on Date pattern.

The user can now decide whether they want to use an external logrotate
command or use internal rotation with Rackstash instead.
2018-06-13 22:33:47 +02:00
a99f182715 Format UNIX timestamps in encoders and preserve pre-formatted Strings 2018-05-16 21:36:24 +02:00
a3d66deb7a Preserve the frozen state when cloning Flows objects 2018-05-16 21:01:10 +02:00
cdc237897f Add a shortcut to enable auto_flush on a Flow with Flow#auto_flush! 2018-05-16 20:46:59 +02:00
d502bece9c Add tests for Flow#auto_flush 2018-05-16 20:44:42 +02:00
3b5ff3a4dc Normalize fields more consistently in encoders 2018-04-19 22:54:16 +02:00
5c626f984b Always add buffering buffers with the Rack middleware 2018-04-10 22:44:25 +02:00
6b4f009107 Introduce auto_flushing Flows to simplify the buffering behavior of Buffers
Instead of defining the specific buffering behavior on a Buffer, we can
now mark individual flows as auto_flushing or now. An auto_flushing Flow
with a buffering Buffer behaves the same as a Buffer with `buffering:
:data` would before.

This allows us to simplify the buffering logic on the Buffer. Also, we
can now use "normal" flows and auto_flushing flows on the same logger in
parallel. Each of them behaves as expected with the same unchanged logger
code.

It is thus easier to define behavior for a development or production
environment of an app since the necessary changes can all be defined on
the logger itself (through the defined flows) without having to adapt
the code which creates suitable Buffers with the Logger#with_buffer
method in any way.
2018-04-10 22:44:25 +02:00
88d50afbf9 Write the raw buffer to the event on flush and use #to_h instead of #to_event to create the event
With that, we can lazy-transform the Buffer to the event hash. B using
the common `.to_h` protocol, we can also support various other objects
here instead of just Buffers (including actual raw Hashes).
2018-03-02 20:10:33 +01:00
5e29b7f912 Enfore RSpec expect syntax 2018-02-15 16:04:34 +01:00
dab393ee21 Disable moneky patching with RSpec 2018-02-15 16:03:13 +01:00
8c9b564f54 Allow to modify a filter chain instance by specifying registered filter names as a Symbol 2018-02-10 21:30:42 +01:00
42d120561b Clarify the error message when trying to fetch invalid specs from a ClassRegistry 2018-02-10 14:12:11 +01:00
2d04ccb6c9 Add ClassRegistry#fetch and change ClassRegistry#[] to return nil if no class was found 2018-02-10 14:08:20 +01:00
21e0c5a228 Rename the filename to paht in file adapter 2018-02-06 20:32:31 +01:00
58b58cfe4a Test sane behavior when trying to set fields or tags causing exception in Rack::Middleware 2018-01-30 18:37:25 +01:00
ce12e26bba Simplify the Rack middleware to write request log immediately
Previously, we would only setup a `::Rack::BodyProxy` to eventually log
the request once the response is deivered to the client. We did this
mainly in order to have a more "accurate" duration in the logs, similar
to what the `::Rack::CommonLogger` middleware does.

This has some significant disadvantages though:

* If an exception is raised by an outer middleware after we returned,
the response is never delivered and the log is lost.
* If a timeout occured (and e.g. a Unicorn worker is killed or a
`Rack::Timeout` stroke) before we return the full response, the log is
lost.
* If the client goes away before the request is finished, the app server
might decide no just throw away the response.

Besides addressing these issues, logging directly after the request
leaves our middleware scope makes the whole process much easier to
reason about.
2018-01-30 18:36:47 +01:00
1c461bf8da Specify exact behavior for @version and @timestamp fields for Logstash encoder 2018-01-27 00:48:51 +01:00
1d16d366a0 Test JSON encoder behavior with a missing timestamp 2018-01-27 00:47:25 +01:00
952ecd5c73 Ensure that the JSON encoder correctly serializes the event timestamp 2018-01-27 00:36:03 +01:00
d6cc062b93 Omit an empty message when encoding an event as JSON 2018-01-27 00:36:03 +01:00
7f6e7e86c6 Add tests to ensure the the Message encoder can deal with uncommon messages 2018-01-27 00:36:03 +01:00
18abd892ba Test that Adapter::File.from_uri only accepts file URIs 2018-01-25 11:14:27 +01:00
6d4106fcb7 Add Select and Remove filters to filter which fields get passed to the encoder 2018-01-25 10:52:42 +01:00
dda72c2f84 Work around deprecations on Ruby 2.5.0 2018-01-24 20:18:50 +01:00
99d2f489dd Update Copyright year 2018-01-24 00:21:04 +01:00
577c065900 Allow to define a File adapter using a file:// URI 2018-01-23 23:54:48 +01:00
39f12047cb Add helper method to parse URI parameter from an adapter URI 2018-01-23 23:53:31 +01:00
c194b00b34 Improve code formatting and fix Rubocop issues 2018-01-23 23:21:58 +01:00
38a5c669a3 Allow to register complex URI schemes for Adapters
Most notably, we want to use the plus character in URI schemes. By only
checking the first character of the scheme registration, we can better
fullfil our contract and better distinguish between schemes and class
names of registered Adapters.
2018-01-22 01:11:27 +01:00
e8bf2c31bd Raise ArgumentError when trying to resolve an unknown adapter URI
If we were to fall through, undefined schemes would always end up as a
file adapter. This is generally undesired since it hides the fact that
we have not found a suitable adapter. Most of the time, this will be a
configuration error which should be reported early.

If a user still wants to create a file adapter with a filename that
looks like a URI, they can create a Rackstash::Adapter::File object
manually.
2018-01-22 01:05:06 +01:00
bdeb0534c9 Allow to set the Flow#raise_on_error setting with the same getter/setter protocol as all the other values 2018-01-12 23:34:17 +01:00
96f102875f Allow to set Flow#raise_on_error? to aid in debugging flows 2018-01-12 21:03:52 +01:00