Uu3mWe

docker-compose安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
version: '3.0'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
- TZ=Asia/Shanghai
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- ./elasticsearch/analysis/synonym.txt:/usr/share/elasticsearch/config/analysis/synonym.txt
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
- ./elasticsearch/config/crack/x-pack-core-7.13.0.jar:/usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.0.jar
- ./elasticsearch/data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
- TZ=Asia/Shanghai
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
- ./elasticsearch/config/crack/x-pack-core-7.13.0.jar:/usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.0.jar
- ./elasticsearch/data02:/usr/share/elasticsearch/data
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
- TZ=Asia/Shanghai
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
- ./elasticsearch/config/crack/x-pack-core-7.13.0.jar:/usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.0.jar
- ./elasticsearch/data03:/usr/share/elasticsearch/data

kibana:
image: docker.elastic.co/kibana/kibana:7.13.0
container_name: kibana
restart: always
ports:
- 5601:5601
volumes:
- ./kibana/config:/usr/share/kibana/config
environment:
I18N_LOCALE: zh-CN
ELASTICSEARCH_URL: https://es01:9200
ELASTICSEARCH_HOSTS: '["https://es01:9200","https://es02:9200","https://es03:9200"]'

ent-search:
image: docker.elastic.co/enterprise-search/enterprise-search:7.13.0
container_name: ent-search
environment:
- "JAVA_OPTS=-Xms2048m -Xmx2048m"
volumes:
- ./enterprise-search/config/enterprise-search.yml:/usr/share/enterprise-search/config/enterprise-search.yml
- ./enterprise-search/config/certs:/usr/share/enterprise-search/config/certs
ports:
- 3002:3002

cerebro:
image: lmenezes/cerebro:0.9.4
container_name: cerebro
restart: always
ports:
- 8900:9000
command:
- -Dhosts.0.host=https://es01:9200
- -Dplay.ws.ssl.loose.acceptAnyCertificate=true

es-head:
image: mobz/elasticsearch-head:5
container_name: es-head
restart: always
ports:
- 9100:9100

networks:
default:
external:
name: dakewe

使用X-Pack设置授权加密

按照 Elasticsearch 的要求,如果我们在 docker 的环境中启动 xpack.security.enabled,我们必须也启动 xpack.security.transport.ssl.enabled。否则,我们将会看到如下的错误信息:

[1]:Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]

接下来,针对7.13.0版本的ElasticSearch配置加密授权访问,下边的步骤是必不可少的,建议认真阅读下去。X-PackElasticSearch 的一个插件,这个插件将提供与ElasticSearch来往的安全性。通过安装这个插件,我们就可以对 ElasticSearch 的集群节点生成证书,配置服务访问密码,以及使用TLS来确保HTTP客户端与集群之间的通信是加密的。

1
docker exec -it es01 bash

进入容器后,前往工作目录下(即/usr/share/elasticsearch),为Elasticearch集群创建一个证书颁发机构。使用elasticsearch-certutil命令输出一个默认名为elastic-stack-ca.p12的PKCS#12密钥存储库文件,它包含CA的公共证书和用于为每个节点签名证书的私钥。

1
2
cd /usr/share/elasticsearch
bin/elasticsearch-certutil ca

如下的命令来生成一个证书

1
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

上面的命令将使用我们的 CA 来生成一个证书 elastic-certificates.p12:
执行exit退出容器,我们把上面的 elastic-certificates.p12 证书移至./elasticsearch/config/certs文件夹。

1
2
3
docker cp es01:/usr/share/elasticsearch/elastic-certificates.p12 ./elasticsearch/config/certs
sudo chmod -R 777 ./elasticsearch/config
docker-compose down

在docker-compose.yaml配置好证书映射
HUW30H
别忘了docker-compose down关掉服务,因为我们要改配置了。

接下来修改config/elasticsearch.yml来使用加密授权。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
cluster.name: "docker-cluster"
network.host: 0.0.0.0

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type


xpack.license.self_generated.type: basic

xpack.security.enabled: true

# 传输层通信:传输协议用于Elasticsearch节点之间的内部通信
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

# HTTP层通信:客户端到Elasticsearch集群的通信
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.verification_mode: certificate

xpack.monitoring.collection.enabled: false

verification_mode 我们选择certificate,这个模式不会去检查证书的CN,只验证证书是否是信任机构签名的即可.如果我们需要验证,并且配置了IP,则需要把这个模式该为full

如果证书是PEM格式,则使用下方配置

1
2
3
4
5
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /home/es/config/node01.key
xpack.security.transport.ssl.certificate: /home/es/config/node01.crt
xpack.security.transport.ssl.certificate_authorities: [ "/home/es/config/ca.crt" ]

设置授权访问的账号和密码

再次启动并进入容器docker exec -it es01 bash,使用elasticsearch-setup-passwords为各个角色创建随机的密码:

1
bin/elasticsearch-setup-passwords auto

也可以使用密码设置来为每个角色设定密码:

1
bin/elasticsearch-setup-passwords interactive

4YqC8I
用interactive参数为每个角色设定密码

访问localhost:9200,输入user elastic的密码,成功获取正确json说明x-pack授权加密已经成功。

为 Elasticsearch 设置认证

内置用户
elastic : 内置的超级用户
kibana_system: 用户Kibana用于连接Elasticsearch并与之通信
logstash_system: Logstash写入监控数据时所需要的ES权限用户
beats_system: Beats写入监控数据时所需要的ES权限用户
apm_system: APM写入监控数据时所需要的ES权限用户
remote_monitoring_user: Metricbeat用户在Elasticsearch中收集和存储监视信息时使用。

使Kibana应用帐号密码

在kibana部分的kibana.yml追加参数:

1
2
elasticsearch.username: "kibana_system"
elasticsearch.password: "XXX"

使logstash应用帐号密码

1
2
3
4
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "https://es0:9200" ]
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "XXX"

然后执行docker-compose up -d kibana 启动服务,等待几分钟,访问localhost:5601,成功出现需要输入密码的界面说明配置成功。
q295bz

到这里,已经完成elasticsearch和kibana的全部部署工作.
pyo0I0

专题目录

ElasticStack-安装篇
ElasticStack-elasticsearch篇
ElasticStack-logstash篇
elasticSearch-mapping相关
elasticSearch-分词器介绍
elasticSearch-分词器实践笔记
elasticSearch-同义词分词器自定义实践
docker-elk集群实践
filebeat与logstash实践
filebeat之pipeline实践
Elasticsearch 7.x 白金级 破解实践
elk的告警调研与实践

出处:英文原文

类转换器的作用是将普通的javascript对象转换成类对象。我们通过api端点或者json文件访问所得的是普通的json文本,一般我们通过JSON.parse把其转换成普通的javascript对象,但是有时候我们想让它变成一个类的对象而不是普通的javascript对象。比如用class-validator来验证从后端api获取的json字符串时,我们就需要自动把json转为待验证类的对象而不是一个js对象。

例如我们现在可以读取远程api的一个users.json的内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[{
"id": 1,
"firstName": "Johny",
"lastName": "Cage",
"age": 27
},
{
"id": 2,
"firstName": "Ismoil",
"lastName": "Somoni",
"age": 50
},
{
"id": 3,
"firstName": "Luke",
"lastName": "Dacascos",
"age": 12
}]

我们有一个User

1
2
3
4
5
6
7
8
9
10
11
12
13
14
export class User {
id: number;
firstName: string;
lastName: string;
age: number;

getName() {
return this.firstName + " " + this.lastName;
}

isAdult() {
return this.age > 36 && this.age < 60;
}
}

然后你想通过user.json来获取User的对象数组

1
2
3
4
5
fetch("users.json").then((users: User[]) => {
// you can use users here, and type hinting also will be available to you,
// but users are not actually instances of User class
// this means that you can't use methods of User class
});

现在你可以获取users[0].firstname但是由于你获取的是普通的js对象而非User类的对象,所以你无法调用users[0].getName()方法,而class-transformer就是为了把普通的js对象按你的需求转换成类对象而生的。

你只要像下面这样就可以创建真正的User[]对象数组了

1
2
3
4
fetch("users.json").then((users: Object[]) => {
const realUsers = plainToClass(User, users);
// now each user in realUsers is instance of User class
});

安装

安装class-transformer:
npm install class-transformer --save
安装reflect-metadata:

安装后在app.ts这种顶层文件你需要import "reflect-metadata";

基础方法

plainToClass

普通对象转换为类对象

1
2
3
import {plainToClass} from "class-transformer";

let users = plainToClass(User, userJson); // to convert user plain object a single user. also supports arrays

plainToClassFromExist

普通对象合并已经创建的类实例

1
2
3
4
const defaultUser = new User();
defaultUser.role = 'user';

let mixedUser = plainToClassFromExist(defaultUser, user); // mixed user should have the value role = user when no value is set otherwise.

classToPlain

类实例转换为普通对象

转换后可以使用JSON.stringify再转成普通的json文本

1
2
import {classToPlain} from "class-transformer";
let photo = classToPlain(photo);

classToClass

克隆类实例

1
2
import {classToClass} from "class-transformer";
let photo = classToClass(photo);

可以使用ignoreDecorators选项去除所有原实例中的装饰器

serialize

直接把类实例转换为json文本,是不是数组都可以转换

1
2
import {serialize} from "class-transformer";
let photo = serialize(photo);

deserialize 和 deserializeArray

直接把json文本转换为类对象

1
2
import {deserialize} from "class-transformer";
let photo = deserialize(Photo, photo);

如果json文本是个对象数组请使用deserializeArray方法

1
2
import {deserializeArray} from "class-transformer";
let photos = deserializeArray(Photo, photos);

强制类型安全

plainToClass会把所有的被转换对象的属性全部类实例的属性,即时类中并不存在某些属性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import {plainToClass} from "class-transformer";

class User {
id: number
firstName: string
lastName: string
}

const fromPlainUser = {
unkownProp: 'hello there',
firstName: 'Umed',
lastName: 'Khudoiberdiev',
}

console.log(plainToClass(User, fromPlainUser))

// User {
// unkownProp: 'hello there',
// firstName: 'Umed',
// lastName: 'Khudoiberdiev',
// }

你可以使用excludeExtraneousValues选项结合Expose装饰器来指定需要公开的属性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import {Expose, plainToClass} from "class-transformer";

class User {
@Expose() id: number;
@Expose() firstName: string;
@Expose() lastName: string;
}

const fromPlainUser = {
unkownProp: 'hello there',
firstName: 'Umed',
lastName: 'Khudoiberdiev',
}

console.log(plainToClass(User, fromPlainUser, { excludeExtraneousValues: true }))

// User {
// id: undefined,
// firstName: 'Umed',
// lastName: 'Khudoiberdiev'
// }

子类型转换

嵌套对象

由于现在Typescript对反射还没有非常好的支持,所以你需要使用@Type装饰器来隐式地指定属性所属的类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import {Type, plainToClass} from "class-transformer";

export class Album {

id: number;

name: string;

@Type(() => Photo)
photos: Photo[];
}

export class Photo {
id: number;
filename: string;
}

let album = plainToClass(Album, albumJson);
// now album is Album object with Photo objects inside

多类型选项

一个嵌套的子类型也可以匹配多个类型,这可以通过判断器实现。判断器需要指定一个 property,而被转换js对象中的嵌套对象的也必须拥有与property相同的一个字段,并把值设置为需要转换的子类型的名称。判断器还需要指定所有的子类型值以及其名称,具体示例如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import {Type, plainToClass} from "class-transformer";

const albumJson = {
"id": 1,
"name": "foo",
"topPhoto": {
"id": 9,
"filename": "cool_wale.jpg",
"depth": 1245,
"__type": "underwater"
}
}

export abstract class Photo {
id: number;
filename: string;
}

export class Landscape extends Photo {
panorama: boolean;
}

export class Portrait extends Photo {
person: Person;
}

export class UnderWater extends Photo {
depth: number;
}

export class Album {

id: number;
name: string;

@Type(() => Photo, {
discriminator: {
property: "__type",
subTypes: [
{ value: Landscape, name: "landscape" },
{ value: Portrait, name: "portrait" },
{ value: UnderWater, name: "underwater" }
]
}
})
topPhoto: Landscape | Portrait | UnderWater;

}

let album = plainToClass(Album, albumJson);
// now album is Album object with a UnderWater object without `__type` property.

此外可以设置keepDiscriminatorProperty: true,这样可以把判断器的属性也包含在转换后的对象中

排除与公开

公开方法的返回值

添加@Expose装饰器即可公开getter和方法的返回值

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import {Expose} from "class-transformer";

export class User {

id: number;
firstName: string;
lastName: string;
password: string;

@Expose()
get name() {
return this.firstName + " " + this.lastName;
}

@Expose()
getFullName() {
return this.firstName + " " + this.lastName;
}
}

公开属性为不同名称

如果要使用其他名称公开某些属性,可以通过为@Expose装饰器指定name选项来实现:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import {Expose} from "class-transformer";

export class User {

@Expose({ name: "uid" })
id: number;

firstName: string;

lastName: string;

@Expose({ name: "secretKey" })
password: string;

@Expose({ name: "fullName" })
getFullName() {
return this.firstName + " " + this.lastName;
}
}

跳过指定属性

有时您想在转换过程中跳过一些属性。这可以使用@Exclude装饰器完成:

1
2
3
4
5
6
7
8
9
10
11
import {Exclude} from "class-transformer";

export class User {

id: number;

email: string;

@Exclude()
password: string;
}

现在,当您转换用户时,password属性将被跳过,并且不包含在转换结果中。

根据操作决定跳过

我们可以通过toClassOnly或者toPlainOnly来控制一个属性在哪些操作中需要排除

1
2
3
4
5
6
7
8
9
10
11
import {Exclude} from "class-transformer";

export class User {

id: number;

email: string;

@Exclude({ toPlainOnly: true })
password: string;
}

现在password属性将会在classToPlain操作中排除,相反的可以使用toClassOnly

跳过类的所有属性

你可以通过在类上添加@Exclude装饰器并且在需要公开的属性上添加@Expose装饰器来只公开指定的属性

1
2
3
4
5
6
7
8
9
10
11
12
13
import {Exclude, Expose} from "class-transformer";

@Exclude()
export class User {

@Expose()
id: number;

@Expose()
email: string;

password: string;
}

另外,您可以在转换期间设置排除策略:

1
2
import {classToPlain} from "class-transformer";
let photo = classToPlain(photo, { strategy: "excludeAll" });

这时你不需要在添加@Exclude装饰器了

跳过私有属性或某些前缀属性

我们可以排除公开具有指定前缀的属性以及私有属性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import {Expose} from "class-transformer";

export class User {

id: number;
private _firstName: string;
private _lastName: string;
_password: string;

setName(firstName: string, lastName: string) {
this._firstName = firstName;
this._lastName = lastName;
}

@Expose()
get name() {
return this.firstName + " " + this.lastName;
}

}

const user = new User();
user.id = 1;
user.setName("Johny", "Cage");
user._password = 123;

const plainUser = classToPlain(user, { excludePrefixes: ["_"] });
// here plainUser will be equal to
// { id: 1, name: "Johny Cage" }

使用组来控制排除的属性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import {Exclude, Expose} from "class-transformer";

@Exclude()
export class User {

id: number;

name: string;

@Expose({ groups: ["user", "admin"] }) // this means that this data will be exposed only to users and admins
email: string;

@Expose({ groups: ["user"] }) // this means that this data will be exposed only to users
password: string;
}

let user1 = classToPlain(user, { groups: ["user"] }); // will contain id, name, email and password
let user2 = classToPlain(user, { groups: ["admin"] }); // will contain id, name and email

使用版本范围来控制公开和排除的属性

如果要构建具有不同版本的API,则class-transformer具有非常有用的工具。您可以控制应在哪个版本中公开或排除模型的哪些属性。示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import {Exclude, Expose} from "class-transformer";

@Exclude()
export class User {

id: number;

name: string;

@Expose({ since: 0.7, until: 1 }) // this means that this property will be exposed for version starting from 0.7 until 1
email: string;

@Expose({ since: 2.1 }) // this means that this property will be exposed for version starting from 2.1
password: string;
}

let user1 = classToPlain(user, { version: 0.5 }); // will contain id and name
let user2 = classToPlain(user, { version: 0.7 }); // will contain id, name and email
let user3 = classToPlain(user, { version: 1 }); // will contain id and name
let user4 = classToPlain(user, { version: 2 }); // will contain id and name
let user5 = classToPlain(user, { version: 2.1 }); // will contain id, name nad password

特殊处理

将日期字符串转换为Date对象

有时,您的JavaScript对象中有一个以字符串格式接收的Date。您想从中创建一个真正的javascript Date对象。您只需将Date对象传递给@Type装饰器即可完成此操作:

当从类对象反向转换为普通对象时registrationDate将会被转回为字符串

1
2
3
4
5
6
7
8
9
10
11
12
13
import {Type} from "class-transformer";

export class User {

id: number;

email: string;

password: string;

@Type(() => Date)
registrationDate: Date;
}

当您想将值转换为Number, String, Boolean 类型时也是这样做

数组处理

当你想转换数组时,你必须使用@Type装饰器指定数组项的类型也可以使用自定义的数组类型

Set和Map也是一样

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import {Type} from "class-transformer";

export class AlbumCollection extends Array<Album> {
// custom array functions ...
}

export class Photo {

id: number;

name: string;

@Type(() => Album)
albums: Album[];
// albums: AlbumCollection; 使用自定义类型
}

export class Skill {
name: string;
}

export class Weapon {
name: string;
range: number;
}

export class Player {
name: string;

@Type(() => Skill)
skills: Set<Skill>;

@Type(() => Weapon)
weapons: Map<string, Weapon>;
}

自定义转换

基本使用

你可以使用@Transform添加额外的数据转换,例如当你想把通过普通对象中的字符串日期转换后的date对象继续转换变成moment库的对象:

1
2
3
4
5
6
7
8
9
10
11
12
import {Transform} from "class-transformer";
import * as moment from "moment";
import {Moment} from "moment";

export class Photo {

id: number;

@Type(() => Date)
@Transform(value => moment(value), { toClassOnly: true })
date: Moment;
}

现在当执行plainToClass转换后的对象中的date属性将是一个Moment对象。@Transform同样支持组和版本。

高级用法

@Transform有更多的参数给你创建自定义的转换逻辑

@Transform((value, obj, type) => value)
参数 描述
value 自定义转换执行前的属性值
obj 转换源对象
type 转换的类型

其他装饰器

签名 示例
@TransformClassToPlain @TransformClassToPlain({ groups: [“user”] })
@TransformClassToClass @TransformClassToClass({ groups: [“user”] })
@TransformPlainToClas @TransformPlainToClass(User, { groups: [“user”] })
上述装饰器接受一个可选参数:ClassTransformOptions-转换选项,例如groups, version, name,示例:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
@Exclude()
class User {

id: number;

@Expose()
firstName: string;

@Expose()
lastName: string;

@Expose({ groups: ['user.email'] })
email: string;

password: string;
}

class UserController {

@TransformClassToPlain({ groups: ['user.email'] })
getUser() {
const user = new User();
user.firstName = "Snir";
user.lastName = "Segal";
user.password = "imnosuperman";

return user;
}
}

const controller = new UserController();
const user = controller.getUser();
user对象将包含firstname,latstname和email

使用泛型

由于目前Typescript对反射的支持还没有完善,所以只能使用其它替代方案,具体可以查看这个例子

隐式类型转换

你如果将class-validator与class-transformer一起使用,则可能不想启用此功能。

根据Typescript提供的类型信息,启用内置类型之间的自动转换。默认禁用。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import { IsString } from 'class-validator'

class MyPayload {

@IsString()
prop: string
}


const result1 = plainToClass(MyPayload, { prop: 1234 }, { enableImplicitConversion: true });
const result2 = plainToClass(MyPayload, { prop: 1234 }, { enableImplicitConversion: false });

/**
* result1 will be `{ prop: "1234" }` - notice how the prop value has been converted to string.
* result2 will be `{ prop: 1234 }` - default behaviour
*/

循环引用

如果User包含一个Photo类型的photos数组属性,而Photo又包含一个属性链接到User,则转换过程中此属性会被忽略,除了classToClass操作。

概述

nestjs的装饰器很好用,于是想着自定义装饰器来实践下。
想到2个场景非常适合自定义装饰器。
一个是通过@User装饰器获取token下当前用户,
一个是通过@Permissions,进行角色权限的守卫校验。
那么实践一下呗

@User

对@user装饰器的实现,主要作用是能快速从token中快速拿到用户信息
user.decorator.ts

1
2
3
4
5
6
7
8
9
import { createParamDecorator, ExecutionContext } from '@nestjs/common';

export const User = createParamDecorator(
(data: string, ctx: ExecutionContext) => {
const request = ctx.switchToHttp().getRequest();
const user = request.user;
return data ? user && user[data] : user;
},
);

使用

1
2
3
4
@Get()
async findOne(@User() user: UserEntity) {
console.log(user);
}

1
2
3
4
@Get()
async findOne(@User('firstName') firstName: string) {
console.log(`Hello ${firstName}`);
}

@Permissions

permissions.decorator.ts

基于角色权限的装饰器实现,主要作用是进行增删改查操作的权限校验

1
2
3
4
5
6
7
8
9
10
11
// import { applyDecorators, SetMetadata } from '@nestjs/common';
// export const Permissions = (permissions: string) => SetMetadata('permissions', permissions);


import { applyDecorators, SetMetadata, UseGuards } from "@nestjs/common";
export function Permissions(permissions: string): Function {
// 可定义‘组合装饰器’
return applyDecorators(
SetMetadata('permissions', permissions)
)
}

拿个例子说下
获取用户列表v1/users接口增加@Permissions('sys:user:list')

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import { Body, Controller, Delete, Get, Param, ParseIntPipe, Post, Put, Query, UseGuards, UsePipes, ValidationPipe } from '@nestjs/common';
import { AuthGuard } from '@nestjs/passport';
import { ApiBearerAuth, ApiCreatedResponse, ApiOkResponse, ApiOperation, ApiParam, ApiTags } from '@nestjs/swagger';
import { JwtAuthGuard } from 'src/common/guards/jwt-auth.guard';
import { RolesGuard } from 'src/common/guards/roles.guard';
import { Result } from 'src/common/utils/result';
import { QueryUserDto } from './dto/query.dto';
import { UserService } from './user.service';
import { Permissions } from 'src/common/decorator/permissions.decorator'

@ApiTags('用户相关')
@Controller('v1/users')
@ApiBearerAuth()
@UseGuards(JwtAuthGuard, RolesGuard)
export class UserController {
constructor(
private readonly userService: UserService
) { }

@Get()
@ApiOperation({ summary: '查询用户列表' })
@Permissions('sys:user:list')
async list(@Query() dto: QueryUserDto): Promise<Result> {
console.log(dto)
const res = await this.userService.page(dto)
return Result.ok(res)
// throw new ForbiddenException()
}
}

通过RolesGuard进行角色权限校验
roles.guard.ts

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import { Injectable, CanActivate, ExecutionContext, ForbiddenException } from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import { Observable } from 'rxjs';

@Injectable()
export class RolesGuard implements CanActivate {
constructor(private reflector: Reflector) { }
async canActivate(
context: ExecutionContext,
): Promise<boolean> {
const request = context.switchToHttp().getRequest();
const user = request.user;
console.log('当前用户', user)
// 当前请求所需权限
const currentPerm = this.reflector.get<string>('permissions', context.getHandler());
console.log('当前所需权限:', currentPerm)
// 标识不需要权限
if (!currentPerm) {
return true;
}
// 根据用户id 查询所拥有的权限
// const permList = await this.permSerivce.findUserPerms(user.id)
// const perms: string[] = []
// for (let i = 0, len = permList.length; i < len; i++) {
// permList[i]['m_perms'].indexOf(',') > -1 ? perms.push(...permList[i]['m_perms'].split(',')) : perms.push(permList[i]['m_perms'])
// }
// 匹配权限
// if (perms.includes(currentPerm)) return true
// throw new ForbiddenException()
}
}

概述

最近在用nestjs重新做RBAC构架。
本次完整实践nestjs的注册登陆,到jwt授权认证,到guard守卫拦截验证,到strategy的JWT策略。
和Spring Boot的Shiro对比下来,还是nestjs的写着舒服。

实践

auth.controller.ts
注册登陆的controller实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import { Body, Controller, Post } from '@nestjs/common';
import { ApiOkResponse, ApiOperation, ApiTags } from '@nestjs/swagger';
import { Result } from 'src/common/utils/result';
import { CreateUserDto } from '../user/dto/create.dto';
import { LoginUserDto } from '../user/dto/login.dto';
import { UserEntity } from '../user/user.entity';
import { UserService } from '../user/user.service';
import { AuthService } from './auth.service';

@ApiTags('登录注册')
@Controller('v1/auth')
export class AuthController {
constructor(
private readonly userService: UserService,
private readonly authService: AuthService
) { }

@Post('register')
@ApiOperation({ summary: '用户注册' })
@ApiOkResponse({ type: UserEntity })
async create(@Body() user: CreateUserDto): Promise<Result> {
console.log('user', user)
const res = await this.userService.create(user)
return Result.ok(res)
}

@Post('login')
@ApiOperation({ summary: '登录' })
async login(@Body() dto: LoginUserDto): Promise<Result> {
const res = await this.userService.login(dto.account, dto.password)
return Result.ok(res)
}
}

user.service.ts
注册登陆及JWT的service实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
import { HttpException, HttpStatus, Injectable, NotFoundException } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Like, Repository, UpdateResult } from 'typeorm';
import { UserEntity } from './user.entity';
import { Result } from 'src/common/utils/result';
import { CreateUserDto } from './dto/create.dto';
import { QueryUserDto } from './dto/query.dto';
import { classToPlain, plainToClass } from 'class-transformer';
import { RedisService } from 'nestjs-redis';
import { ConfigService } from '@nestjs/config';
import { genSalt, hash, compare } from 'bcrypt'
import { JwtService } from '@nestjs/jwt';
import { UpdateUserDto } from './dto/update.dto';
@Injectable()
export class UserService {
constructor(
@InjectRepository(UserEntity)
private readonly userRep: Repository<UserEntity>,
private readonly redisService: RedisService,
private readonly config: ConfigService,
private readonly jwtService: JwtService,
) { }

async create(dto: CreateUserDto): Promise<UserEntity | Result> {
console.log(dto)
const existing = await this.findByUsername(dto.username)
if (existing) throw new HttpException('账号已存在,请调整后重新注册!', HttpStatus.NOT_ACCEPTABLE);
const salt = await genSalt()
dto.password = await hash(dto.password, salt)
const user = plainToClass(UserEntity, { salt, ...dto }, { ignoreDecorators: true })
console.log('user', user)
const res = await this.userRep.save(user)
return res
}

// 登录
async login(account: string, password: string): Promise<object | Result> {
const user = await this.findByUsername(account)
console.log("user", user)
if (!user) throw new HttpException('账号或密码错误', HttpStatus.NOT_FOUND);
console.log('账号', account)
console.log('密码', password)
console.log('加密的密码', user.password)
const checkPassword = await compare(password, user.password)
console.log('是否一致', checkPassword)
if (!checkPassword) throw new HttpException('账号或密码错误', HttpStatus.NOT_FOUND);
// 生成 token
const data = this.genToken({ id: user.id })
return data
}

// 根据ID查找
async findById(id: number): Promise<UserEntity> {
const res = await this.userRep.findOne(id)
if (!res) {
throw new NotFoundException()
}
return res
}

// 根据用户名查找
async findByUsername(username: string): Promise<UserEntity> {
return await this.userRep.findOne({ username })
}

// 生成 token
genToken(payload: { id: number }): Record<string, unknown> {
const accessToken = `Bearer ${this.jwtService.sign(payload)}`
const refreshToken = this.jwtService.sign(payload, { expiresIn: this.config.get('jwt.refreshExpiresIn') })
return { accessToken, refreshToken }
}

// 刷新 token
refreshToken(id: number): string {
return this.jwtService.sign({ id })
}

// 校验 token
verifyToken(token: string): number {
try {
if (!token) return 0
const id = this.jwtService.verify(token.replace('Bearer ', ''))
return id
} catch (error) {
return 0
}
}

// 根据JWT解析的ID校验用户
async validateUserByJwt(payload: { id: number }): Promise<UserEntity> {
return await this.findById(payload.id)
}
}

jwt-auth.guard.ts
守卫实现,拦截accessToken和refreshToken,过期进行续签。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
import { Injectable, CanActivate, ExecutionContext, UnauthorizedException } from '@nestjs/common';
import { AuthGuard } from '@nestjs/passport';
import { Observable } from 'rxjs';
import { UserService } from 'src/modules/user/user.service';
@Injectable()
export class JwtAuthGuard extends AuthGuard('jwt') {
constructor(
private readonly userService: UserService,
) {
super()
}
async canActivate(
context: ExecutionContext,
): Promise<boolean> {
const req = context.switchToHttp().getRequest()
const res = context.switchToHttp().getResponse()
try {
const accessToken = req.get('Authorization')
if (!accessToken) throw new UnauthorizedException('请先登录')

const atUserId = this.userService.verifyToken(accessToken)
if (atUserId) return this.activate(context)
console.log(req.user)
const refreshToken = req.get('RefreshToken')
const rtUserId = this.userService.verifyToken(refreshToken)
if (!rtUserId) throw new UnauthorizedException('当前登录已过期,请重新登录')
const user = await this.userService.findById(rtUserId)
if (user) {
const tokens = this.userService.genToken({ id: rtUserId })
// request headers 对象 prop 属性全自动转成小写,
// 所以 获取 request.headers['authorization'] 或 request.get('Authorization')
// 重置属性 request.headers[authorization] = value
req.headers['authorization'] = tokens.accessToken
req.headers['refreshtoken'] = tokens.refreshToken
// 在响应头中加入新的token,客户端判断响应头有无 Authorization 字段,有则重置
res.header('Authorization', tokens.accessToken)
res.header('RefreshToken', tokens.refreshToken)
// 将当前请求交给下一级
return this.activate(context)
} else {
throw new UnauthorizedException('用户不存在')
}
} catch (error) {
// Logger
return false
}
}

async activate(context: ExecutionContext): Promise<boolean> {
return super.canActivate(context) as Promise<boolean>
}
}

jwt.strategy.ts
JWT解析后进行校验

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import { Injectable, UnauthorizedException } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { PassportStrategy } from '@nestjs/passport';
import { ExtractJwt, Strategy } from 'passport-jwt';
import { UserService } from 'src/modules/user/user.service';

@Injectable()
export class JwtStrategy extends PassportStrategy(Strategy) {
constructor(
private readonly userService: UserService,
private readonly config: ConfigService
) {
super({
jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(),
secretOrKey: config.get('jwt.secretkey'),
})
}

async validate(payload: any) {
const user = await this.userService.validateUserByJwt(payload)
// 如果有用户信息,代表 token 没有过期,没有则 token 已失效
if (!user) throw new UnauthorizedException()
return user
}
}

概述

根据 ElasticStack-安装篇 安装好logstash,我们开始进行配置和同步数据。
本次实践通过logstash同步mysql到es。

配置

进入logstash容器进行安装logstash-input-jdbc

1
docker exec -it docker_logstash bash

安装logstash-input-jdbc

1
./bin/logstash-plugin install logstash-input-jdbc

下载 mysql-connector-java-8.0.24.jar
放入/data/dockers/logstash/config/mysql

在文件目录/data/dockers/logstash/config/conf.d下创建jdbc.conf文件,进行mysql数据到es的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
input{
jdbc{
# 连接数据库
jdbc_connection_string => "jdbc:mysql://47.119.168.111:3306/fob?serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false"
jdbc_user => "root"
jdbc_password => "XXXXX"
# 连接数据库的驱动包
jdbc_driver_library => "/usr/share/logstash/config/mysql/mysql-connector-java-8.0.24.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
codec => plain { charset => "UTF-8" }

# 数据追踪
# 追踪的字段
tracking_column => "updated_at"
# 上次追踪的元数据存放位置
last_run_metadata_path => "/usr/share/logstash/config/lastrun/logstash_jdbc_last_run"
# 设置时区
jdbc_default_timezone => "Asia/Shanghai"
# sql 文件地址
# statement_filepath => ""
# sql
statement => "SELECT g.id AS id,g.product_name AS product_name,g.shop_id AS shop_id,g.category_id AS category_id,g.keyword AS keyword,g.status AS status FROM fg_product g WHERE g.updated_at > :sql_last_value"
# 是否清除 last_run_metadata_path 的记录,如果为真那么每次都相当于从头开始查询所有的数据库记录
clean_run =>false
# 这是控制定时的,重复执行导入任务的时间间隔,第一位是分钟 不设置就是1分钟执行一次
schedule => "* * * * *"
}
}
output{
elasticsearch{
# 要导入到的Elasticsearch所在的主机
hosts => "47.119.168.111:9200"
# 要导入到的Elasticsearch的索引的名称
index => "fob_index"
# 类型名称(类似数据库表名)
document_type => "fg_product"
# 主键名称(类似数据库表名)
document_id => "%{id}"
}

stdout{
# JSON 格式输出
codec => json_lines
}
}


查看mysql数据是否进入到es
k0CrP0

专题目录

ElasticStack-安装篇
ElasticStack-elasticsearch篇
ElasticStack-logstash篇
elasticSearch-mapping相关
elasticSearch-分词器介绍
elasticSearch-分词器实践笔记
elasticSearch-同义词分词器自定义实践
docker-elk集群实践
filebeat与logstash实践
filebeat之pipeline实践
Elasticsearch 7.x 白金级 破解实践
elk的告警调研与实践

概述

主要汇总下基础的restful api

host:ip:9200

基础查询

获取es信息 GET host

request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"name": "5f04c6c0a818",
"cluster_name": "elasticsearch",
"cluster_uuid": "vUPjZeMvSGqO_lrtmhqlmw",
"version": {
"number": "7.12.0",
"build_flavor": "default",
"build_type": "docker",
"build_hash": "78722783c38caa25a70982b5b042074cde5d3b3a",
"build_date": "2021-03-18T06:17:15.410153305Z",
"build_snapshot": false,
"lucene_version": "8.8.0",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}

获取索引 GET host/_cat/indices?v

request:

1
2
3
4
5
6
7
8
9
10
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open bank ktIRbx9ZTwyDrEHlxiZpqw 1 1 1000 0 379.3kb 379.3kb
green open .kibana_task_manager_7.12.0_001 Yi3dSuiVSYWYaFidZxHSJg 1 0 9 26006 2.6mb 2.6mb
green open .apm-custom-link DLOQIooBRiWD4O237c8tBA 1 0 0 0 208b 208b
green open .apm-agent-configuration zUC91G_oRw63OY54rV2orw 1 0 0 0 208b 208b
green open .async-search NkCazI4YQDirjCrnvlEv2Q 1 0 0 24 794.5kb 794.5kb
green open .kibana_7.12.0_001 z4vax_yNTq2cOP7JxBphOQ 1 0 63 10 2.1mb 2.1mb
green open .kibana-event-log-7.12.0-000001 CYz7KhrsRa-dino1NEPDog 1 0 7 0 32.9kb 32.9kb
green open .tasks 6wSie0zSRIW17VNQuxHgNQ 1 0 8 0 42.4kb 42.4kb

批量创建数据 POST host/bank/account/_bulk

request:

1
2
3
4
5
6
{"index":{"_id":"1"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
{"index":{"_id":"6"}}
{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"}
{"index":{"_id":"13"}}
{"account_number":13,"balance":32838,"firstname":"Nanette","lastname":"Bates","age":28,"gender":"F","address":"789 Madison Street","employer":"Quility","email":"nanettebates@quility.com","city":"Nogal","state":"VA"}

GET查询 GET host/bank/_search?q=age:31

response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 61,
"relation": "eq"
},
"max_score": 1.0,
"hits": [
{
"_index": "bank",
"_type": "account",
"_id": "51",
"_score": 1.0,
"_source": {
"account_number": 51,
"balance": 14097,
"firstname": "Burton",
"lastname": "Meyers",
"age": 31,
"gender": "F",
"address": "334 River Street",
"employer": "Bezal",
"email": "burtonmeyers@bezal.com",
"city": "Jacksonburg",
"state": "MO"
}
}
]
}
}

POST查询 POST host/bank/_search

request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
{
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"range": {
"balance": {
"gte": 20000,
"lte": 30000
}
}
}
}
},
"sort": {
"age": {
"order": "asc"
}
},
"_source": [
"account_number",
"balance",
"address"
],
"from": 1,
"size": 10
}

response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 217,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": "bank",
"_type": "account",
"_id": "292",
"_score": null,
"_source": {
"account_number": 292,
"address": "691 Nassau Street",
"balance": 26679
},
"sort": [
20
]
}
]
}
}

聚合查询

分组 tags 后再分组source_ip

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
GET vpn-log-*/_search
{
"query": {
"bool": {
"must": {
"match": {
"_index": "<vpn-log-{now/d{YYYY-MM-dd}}>"
}
},
"filter": [
{
"terms": {
"tags": [
"QN"
]
}
}
]
}
},
"aggs": {
"topn": {
"terms": {
"field": "tags"
},
"aggs": {
"source_ip_topn": {
"terms": {
"field": "source_ip"
}
}
}
}
}
}

专题目录

ElasticStack-安装篇
ElasticStack-elasticsearch篇
ElasticStack-logstash篇
elasticSearch-mapping相关
elasticSearch-分词器介绍
elasticSearch-分词器实践笔记
elasticSearch-同义词分词器自定义实践
docker-elk集群实践
filebeat与logstash实践
filebeat之pipeline实践
Elasticsearch 7.x 白金级 破解实践
elk的告警调研与实践

背景

NWd1w3

需要使用elasticSearch进行商品搜索及广告统计,对Elastic Stack进行调研.
这用nodejs客户端进行演示记录

安装

由于Elastic stack的所有Component都要互通,
要先设定一个network让所有的Container吃同一个网路如下:

docker network create elastic_stack

es安装

docker安装

1
2
3
4
5
6
7
8
9
10
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
-e ES_JAVA_OPS="-Xms256m -Xmx256m" \
-e "discovery.type=single-node" \
--network elastic_stack \
-d docker.elastic.co/elasticsearch/elasticsearch:7.12.0

// 把配置文件拿出来
docker cp elasticsearch:/usr/share/elasticsearch/config /data/dockers/es/config
docker cp elasticsearch:/usr/share/elasticsearch/data /data/dockers/es/data
docker cp elasticsearch:/usr/share/elasticsearch/plugins /data/dockers/es/plugins

chmod -R 777 /data/dockers/es

1
2
3
4
5
6
7
8
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
-e ES_JAVA_OPS="-Xms256m -Xmx256m" \
-e "discovery.type=single-node" \
--network elastic_stack \
-v /data/dockers/es/config:/usr/share/elasticsearch/config \
-v /data/dockers/es/data:/usr/share/elasticsearch/data \
-v /data/dockers/es/plugins:/usr/share/elasticsearch/plugins \
-d docker.elastic.co/elasticsearch/elasticsearch:7.12.0

验证是否正常:http://localhost:9200/

Kibana安装

1
2
3
docker run --name kibana \
--network elastic_stack \
-p 5601:5601 -d docker.elastic.co/kibana/kibana:7.12.0

验证是否正常:http://localhost:5601/

logstash安装

1. 首先创建一个容器 用来获取它的配置文件

1
2
docker run --name logstash \
-d docker.elastic.co/logstash/logstash:7.12.0

查看日志信息 是否启动成功

1
docker logs -f logstash

2. 创建挂载文件

拷贝数据

1
2
3
4
5
docker cp logstash:/usr/share/logstash/config /data/dockers/logstash/config

docker cp logstash:/usr/share/logstash/data /data/dockers/logstash/data

docker cp logstash:/usr/share/logstash/pipeline /data/dockers/logstash/pipeline

创建logstash配置文件路径

1
mkdir -p /data/dockers/logstash/config/conf.d

修改配置
logstash.yml

1
2
3
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
path.config: /usr/share/logstash/config/conf.d/*.conf //配置mysql同步es配置使用
1
chmod -R 777 /data/dockers/logstash/

重新创建新的容器

1
2
3
4
5
6
7
8
9
10
11
docker run \
--name logstash \
--restart=always \
--network elastic_stack \
-p 5044:5044 \
-p 9600:9600 \
-e ES_JAVA_OPTS="-Duser.timezone=Asia/Shanghai" \
-v /data/dockers/logstash/config:/usr/share/logstash/config \
-v /data/dockers/logstash/data:/usr/share/logstash/data \
-v /data/dockers/logstash/pipeline:/usr/share/logstash/pipeline \
-d logstash:7.12.0

详细配置见ElasticStack-logstash篇

专题目录

ElasticStack-安装篇
ElasticStack-elasticsearch篇
ElasticStack-logstash篇
elasticSearch-mapping相关
elasticSearch-分词器介绍
elasticSearch-分词器实践笔记
elasticSearch-同义词分词器自定义实践
docker-elk集群实践
filebeat与logstash实践
filebeat之pipeline实践
Elasticsearch 7.x 白金级 破解实践
elk的告警调研与实践

背景

为什么多阶段构建

  • Docker镜像是分层的,Dockerfile中的每个指令都会创建一个新的镜像层,镜像层可以被复用和缓存。当Dockerfile的指令修改了,复制的文件变化了,或者构建镜像时指定的变量不同了,对应的镜像层缓存就会失效,某一层的镜像缓存失效之后,它之后的镜像层缓存都会失效。

  • 因此我们还可以将RUN指令合并,但是需要记住的是,我们只能将变化频率一致的指令合并。

  • 我们应该把变化最少的部分放在Dockerfile的前面,这样可以充分利用镜像缓存。

  • 通过最小化镜像层的数量,我们可以得到更小的镜像。

实践

因为最近都在看nestjs,所以直接以nestjs来构建镜像

1
2
yarn global add @nestjs/cli
nest new nestjs-admin

稍等一会你会得到这样一个目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
./nestjs-admin
├── .gitignore
├── .prettierrc
├── README.md
├── nest-cli.json
├── package.json
├── src
│ ├── app.controller.spec.ts
│ ├── app.controller.ts
│ ├── app.module.ts
│ ├── app.service.ts
│ └── main.ts
├── test
│ ├── app.e2e-spec.ts
│ └── jest-e2e.json
├── tsconfig.build.json
├── tsconfig.json
├── tslint.json
└── yarn.lock

我们在根目录下创建一个 .dockerignore 文件,内容如下

.dockerignore

1
2
3
4
5
node_modules  
.git
.idea
.vscode
/coverage

然后我们在根目录继续创建一个 Dockerfile 文件,内容如下

Dockerfile

为了避免镜像中打包冗余的文件,我们使用多阶段构建镜像

大幅减小镜像体积的最简单和最快的方法是选择一个小得多的基本镜像。Alpine是一个很小的Linux发行版,可以完成这项工作。只要选择Node.js的Alpine版本,就会有很大的改进。

1
2
3
4
5
6
7
8
9
10
11
FROM node:12-alpine AS dependencies  
WORKDIR /usr/src/app
COPY package.json yarn.lock ./
RUN yarn install --production

FROM node:12-alpine
WORKDIR /usr/src/app
COPY package.json dist ./
COPY --from=dependencies /usr/src/app/node_modules ./node_modules
EXPOSE 3000
CMD [ "node", "dist/main" ]

package增加命令

1
"build:docker": "yarn build && docker build -t bulolo/$npm_package_name:latest . && docker push bulolo/$npm_package_name:latest",

运行docker打包镜像
yarn build:docker

背景

最近在做一套RBAC的权限管理系统,当在设计部门及查询的时候,需要维护一个部门结构。结构包含 部门id,上级部门pid,当查询一个部门的所有子级部门时候,需要根据pid进行递归查询,层级越多,查询次数越多。那么怎么通过一个简单的数据库设计,满足查询某个部门的所有子级部门。

QAAAig

技巧

对于上图的部门层级结构,给出对应的数据库设计

4tKPL0

原理:
添加一个辅助的varchar字段pids,字段的逻辑是多个部门的id使用,来连接,假设首层使用0表示,每一个层级使用上一层的pids拼接上,再拼接上级部门id来表示,我们也可以通俗的理解为这个是部门层级pids为该部门的所有上级路径

举例:
技术部(id:1)的上级部门是长沙分公司(id:2),
长沙分公司的上级是集团(id:5)
所以技术部的pids为:0,5,2,
长沙分公司的pids为:0,5

因此要查询长沙分公司的子部门,只需要用 0,5,2%去查询即可,即找到当前部门记录的pids(0,5), 拼上,再拼上当前部门ID(2),再拼个%做后缀模糊匹配

1
pids like {pids},{id}%

命令创建项目结构

module、controller、service创建

--no-spec不创建测试用例

1
2
3
nest g mo user
nest g controller user
nest g service user --no-spec

typeorm-model-generator

数据库表自动生成实体类

package.json的scripts配置数据库生成实体类命令:

1
"db": "rm -rf src/entities & npx typeorm-model-generator -h localhost -d race -p 3306 -u root -x hong -e mysql -o src/entities --noConfig --relationIds --lazy --ce none --cp none"

npx typeorm-model-generator –help 查看帮助

1
2
3
4
5
6
7
8
9
10
11
12
13
14

-h dbhost
-d dbname
-p port
-u username
-x password
-e dbtype

-a 生成实体类 extends BaseEntity 可使用基础的CURD
-o 默认输出项目根目录下 output 文件夹下 可以自定义输出目标目录,例:-o ./src/demo/entities
--cf,将文件名转换为指定的大小写 [可选值:"pascal","camel","none"] [默认值 none"]
--ce pascal 生成实体类名采用人名式格式 [可选值:"pascal","camel","none"] [默认值 none"]
--cp camel 生成实体类属性采用驼峰 [可选值:"pascal","camel","none"] [默认值 none”]
--noConfig 不产生相关配置文件

使用方法:
1、将数据库所有表转成实体类

1
npm run db

2.单独将数据库的sys_user表生成实体类

1
npm run db -- --tables sys_user,sys_role

装饰器

装饰器 介绍
@Req() 获取Request对象(express)
@Res() 获取Response对象(express)
@Session() 获取req.session
@Param(key?: string) req.params / req.params[key],param是路由中的参数
@Body(key?: string) req.body / req.body[key]
@Query(key?: string) req.query / req.query[key],query是?后的参数
@Headers(name?: string) req.headers / req.headers[name]
@HttpCode(201) 指定response中的返回值

Modules

属性 介绍
providers 被Nest injector实例化的providers
controllers 被Nest injector实例化的controllers
imports 引入这些moludes export 的providers
exports providers的子集,被其他模块可以import

请求生命周期

周期 xm1jf3

顺序:

  • 中间件
  • 卫兵
  • 拦截器(在操纵流之前)
  • 管道
  • 服务端路由函数
  • 拦截器(在操纵流之后)
  • 异常过滤器(如果捕获到任何异常)

中间件

请求生命周期的第一个流程是经过中间件,我们来实现一个日志中间件

如何实现

中间件必须实现NestMiddleware

./common/middleware/logger.middleware.ts

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import { Injectable, NestMiddleware } from '@nestjs/common';
import { Request, Response } from 'express';

@Injectable()
export class LoggerMiddleware implements NestMiddleware {
use(req: Request, res: Response, next: Function) {
const logFormat = ` >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Request original url: ${req.originalUrl}
Method: ${req.method}
IP: ${req.ip}
Status code: ${res.statusCode}
Parmas: ${JSON.stringify(req.params)}
Query: ${JSON.stringify(req.query)}
Body: ${JSON.stringify(req.body)} \n >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
`
console.log(logFormat);
next();
}
}

如何注册中间件

  • 全局中间件
    在main.ts中进行全局注册
1
2
import { LoggerMiddleware } from './common/middleware/logger.middleware'
app.use(new LoggerMiddleware().use)
0%