Skip to main content
Version: 2.6.1

1. Overview

1.1 Plugin Name#

  • Logging-Pulsar Plugin

1.2 Appropriate Scenario#

  • collect http request log to Pulsar, consume Pulsar message to another application and analysis.

1.3 Plugin functionality#

Apache ShenYu The gateway receives requests from the client, forwards them to the server, and returns the server results to the client. The gateway can record the details of each request,
The list includes: request time, request parameters, request path, response result, response status code, time consumption, upstream IP, exception information waiting.
The Logging-Pulsar plugin is a plugin that records access logs and sends them to the Pulsar cluster.

1.4 Plugin code#

  • Core Module shenyu-plugin-logging-pulsar.

  • Core Class org.apache.shenyu.plugin.logging.pulsar.LoggingPulsarPlugin

  • Core Claas org.apache.shenyu.plugin.logging.pulsar.client.PulsarLogCollectClient

1.5 Added Since Which shenyu version#

  • Since ShenYu 2.5.1

1.6 Technical Solutions#

  • Architecture Diagram

  • Full asynchronous collection and delivery of Logging inside the Apache ShenYu gateway

  • Logging platform by consuming the logs in the Pulsar cluster for repository, and then using Grafana, Kibana or other visualization platform to display

2. How to use plugin

2.1 Plugin-use procedure chart#

2.2 Import pom#

  • Add the Logging-Pulsar dependency to the gateway's pom.xml file.
 <!--shenyu logging-pulsar plugin start--><dependency>    <groupId>org.apache.shenyu</groupId>    <artifactId>shenyu-spring-boot-starter-plugin-logging-pulsar</artifactId>    <version>${project.version}</version></dependency><!--shenyu logging-pulsar plugin end-->

2.3 Enable plugin#

  • In shenyu-admin --> Basic Configuration --> Plugin Management --> loggingPulsar, configure the pulsar parameter and set it to on.

2.4 Config plugin#

2.4.1 Open the plugin and configure pulsar, configure it as follows.#

  • The individual configuration items are described as follows:
config-itemtypedescriptionremarks
topicStringMessage Queue Topicsmust
serviceUrlStringMessage queue service addressmust
sampleRateStringSampling rate, range 0~1, 0: off, 0.01: acquisition 1%, 1: acquisition 100%Optional, default 1, all collection
compressAlgStringCompression algorithm, no compression by default, currently supports LZ4 compressionOptional, no compression by default
maxResponseBodyIngeterMaximum response size, above the threshold no response will be collectedOptional, default 512KB
maxRequestBodyIngeterMaximum request body size, above the threshold no request body will be collectedOptional, default 512KB

Except for topic, serviceUrl, all others are optional, in most cases only these 3 items need to be configured. The default group id is "shenyu-access-logging"

2.4.2 Configuring Selectors and Rulers#

In addition, sometimes a large gateway cluster corresponds to multiple applications (business), different applications (business) corresponds to different topics, related to isolation,
then you can configure different topics (optional) and sampling rate (optional) by selector, the meaning of the configuration items as shown in the table above.
The operation is shown below:

2.5 Logging Info#

collect request info as follows

Field NameMeaningDescriptionRemarks
clientIpClient IP
timeLocalRequest time string, format: yyyy-MM-dd HH:mm:ss.SSS
methodrequest method (different rpc type is not the same, http class for: get, post wait, rpc class for the interface name)
requestHeaderRequest header (json format)
responseHeaderResponse header (json format)
queryParamsRequest query parameters
requestBodyRequest Body (body of binary type will not be captured)
requestUriRequest uri
responseBodyResponse body
responseContentLengthResponse body size
rpcTyperpc type
statusresponse status
upstreamIpUpstream (program providing the service) IP
upstreamResponseTimeTime taken by the upstream (program providing the service) to respond to the request (ms ms)
userAgentRequested user agent
hostThe requested host
moduleRequested modules
pathThe requested path
traceIdRequested Link Tracking IDNeed to access link tracking plugins, such as skywalking,zipkin

2.6 Examples#

2.6.1 Collect Http Log by Pulsar#

2.6.1.1 Plugin Configuration#

Open the plugin and configure pulsar, configure it as follows.

2.6.1.2 Selector Configuration#

For detailed configuration of selectors and rules, please refer to: Selector and rule management

In addition, sometimes a large gateway cluster corresponds to multiple applications (business), different applications (business) corresponds to different topics, related to isolation,
then you can configure different topics (optional) and samplingf rate (optional) by selector, the meaning of the configuration items as shown in the table above.
The operation is shown below:

2.6.1.3 Rule Configuration#

2.6.1.4 Request Service#

2.6.1.5 Consumption and display of Logging#

As each logging platform has differences, such as storage available clickhouse, ElasticSearch, etc., visualization has self-developed or open source Grafana, Kibana, etc..
Logging-Pulsar plugin uses Pulsar to decouple production and consumption, while outputting logs in json format, consumption and visualization require users to choose different technology stacks to achieve their own situation.=

3. How to disable plugin

  • In shenyu-admin --> BasicConfig --> Plugin --> loggingPulsar set Status disable.