2021-12-01 00:00:07.115 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 50-9A-4C-27-F9-D3, the source IP address is fe80::e8d3:8281:e69e:afda, the destination IP address is ff02::1:3. The number of broadcast packets is equal to or larger than 32 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:00:07.115 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 50-9A-4C-27-F9-D3, the source IP address is 192.168.9.103, the destination IP address is 224.0.0.252. The number of broadcast packets is equal to or larger than 32 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:01:34.923 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 50-9A-4C-27-F9-D3, the source IP address is 192.168.9.103, the destination IP address is 224.0.0.251. The number of broadcast packets is equal to or larger than 40 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:01:34.923 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 50-9A-4C-27-F9-D3, the source IP address is fe80::e8d3:8281:e69e:afda, the destination IP address is ff02::fb. The number of broadcast packets is equal to or larger than 40 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:03:48.133 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 48-4D-7E-BE-B0-87, the source IP address is 192.168.9.21, the destination IP address is 224.0.0.251. The number of broadcast packets is equal to or larger than 52 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:03:48.133 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 48-4D-7E-BE-B0-87, the source IP address is fe80::c129:65df:e7de:f745, the destination IP address is ff02::fb. The number of broadcast packets is equal to or larger than 52 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:03:48.133 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 50-9A-4C-27-F9-D3, the source IP address is 192.168.9.103, the destination IP address is 224.0.0.251. The number of broadcast packets is equal to or larger than 60 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:03:48.133 [HUB "hub_dkwbj"] Session "SID-BRIDGE-5": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is 50-9A-4C-27-F9-D3, the source IP address is fe80::e8d3:8281:e69e:afda, the destination IP address is ff02::fb. The number of broadcast packets is equal to or larger than 60 items per 1 second (note this information is the result of mechanical analysis of part of the packets and could be incorrect).
2021-12-01 00:11:07.141 On the TCP Listener (Port 5555), a Client (IP address 167.248.133.58, Host name "scanner-09.ch1.censys-scanner.com", Port number 40418) has connected.
2021-12-01 00:11:07.141 For the client (IP address: 167.248.133.58, host name: "scanner-09.ch1.censys-scanner.com", port number: 40418), connection "CID-8671" has been created.
2021-12-01 00:11:08.058 Connection "CID-8671" has been terminated.
2021-12-01 00:11:08.058 The connection with the client (IP address 167.248.133.58, Port number 40418) has been disconnected.
2021-12-01 00:11:08.289 On the TCP Listener (Port 5555), a Client (IP address 167.248.133.58, Host name "scanner-09.ch1.censys-scanner.com", Port number 34038) has connected.
2021-12-01 00:11:08.289 For the client (IP address: 167.248.133.58, host name: "scanner-09.ch1.censys-scanner.com", port number: 34038), connection "CID-8672" has been created.
2021-12-01 00:11:08.531 SSL communication for connection "CID-8672" has been started. The encryption algorithm name is "AES128-SHA".
2021-12-01 00:11:10.011 Connection "CID-8672" terminated by the cause "A client which is non-SoftEther VPN software has connected to the port." (code 5).

docker network create --driver bridge leiqin

FROM docker.elastic.co/logstash/logstash:7.13.0
RUN logstash-plugin install --version=3.1.5 logstash-output-mongodb

docker build -f logstash.dockerfile -t dakewe/logstash:1.0 .

version: '3.0'
services:

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.13.0
    container_name: filebeat
    volumes:
      - ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - ./filebeat/data:/usr/share/filebeat/data
      - ./filebeat/logs:/usr/share/filebeat/logs
      - ./filebeat/logfiles:/usr/share/filebeat/logfiles
    environment:
      LS_JAVA_OPTS: "-Xmx1024m -Xms1024m"
    networks:
     - leiqin
     
  logstash:
    image: dakewe/logstash:1.0
    container_name: logstash
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx1024m -Xms1024m"
    networks:
     - leiqin

networks:
  leiqin:
    external: true

bin/logstash-plugin install --version=3.1.5 logstash-output-mongodb

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /usr/share/filebeat/logfiles/*.log
  include_lines: ['A large volume of broadcast packets has been detected']

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  # The Logstash hosts
  hosts: ["logstash:5044"]

input {
	beats {
		port => 5044
	}
}

output {
	stdout { 
		codec => rubydebug 
	}
}

input {
	beats {
		port => 5044
	}
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[HUB \"%{NOTSPACE:hub}\"\] Session \"%{NOTSPACE:session}\": A large volume of broadcast packets has been detected. There are cases where packets are discarded based on the policy. The source MAC address is %{NOTSPACE:mac_address}, the source IP address is %{IP:source_ip}, the destination IP address is %{IP:destination_ip}. The number of broadcast packets is equal to or larger than %{NUMBER:items_per_second} items per 1 second "}
  }

  grok {
    match => { "[log][file][path]" => ".*(\\|\/).*(\\|\/)(?<file_name>.*).*"}
  }

  date {
    match => [ "time","ISO8601"]
    timezone => "Asia/Chongqing"
    target => "created_at"
  }

  mutate{
	  remove_field => ["host"]
	  remove_field => ["agent"]
    remove_field => ["input"]
    remove_field => ["tags"]
    remove_field => ["ecs"]
    remove_field => ["time"]
    remove_field => ["log"]
  }
}

output {
	stdout { 
		codec => rubydebug 
	}

	mongodb {
  	collection => "vpn_log"
  	generateId => "true"
  	database => "log"
 		uri => "mongodb://localhost:27017"
	}
}

// 创建流水号（流水号表自增）

throw Error -> 异常中断 

// 创建订单

// 订单入库
await ctx.model.Delivery.VirtualOrder.create(orders)

// 发送通知  -> 异常出现redis报错
const jobs = await ctx.app.noticeQueue.addBulk(datas)

module.exports = {
  async getSession(
    opt = {
      readPreference: { mode: 'primary' },
    }
  ) {
    const { mongoose } = this.app
    const session = await mongoose.startSession(opt)
    await session.startTransaction({
      readConcern: { level: 'majority' },
      writeConcern: { w: 'majority' },
    })
    return session
  },
}

const { ctx } = this
const session = await this.ctx.getSession()
try {
  // 订单入库
  await ctx.model.Delivery.VirtualOrder.create(orders, { session })
  // 发送通知  -> 异常出现redis报错
  const jobs = await ctx.app.noticeQueue.addBulk(datas)
 } catch (error) {
  console.log('开始回滚')
  await session.abortTransaction()
  session.endSession()
  console.log(error)
}

try {
  // 创建
  // 创建
  // 创建
  // 创建
  // 创建
  // 创建
} catch{
  // 人工删除创建
}

#!/bin/bash
# 生成 keyfile
mkdir ./keyfile
openssl rand -base64 745 > ./keyfile/mongoReplSet-keyfile
chmod 600 ./keyfile/mongoReplSet-keyfile

configsvr01    | {"t":{"$date":"2021-05-29T17:38:02.750+00:00"},"s":"I",  "c":"ACCESS",   "id":20254,   "ctx":"main","msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"error opening file: /data/mongo.key: bad file"}}}

version: '3.1'
services:
  mongo1:
    image: mongo
    hostname: mongo1
    container_name: mongo1
    restart: always
    ports:
      - 27011:27017
    volumes:
      - ./keyfile:/data/keyfile
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: 123123
    command: mongod --auth --keyFile /data/keyfile/mongoReplSet-keyfile --bind_ip_all --replSet rs0

  mongo2:
    image: mongo
    hostname: mongo2
    container_name: mongo2
    restart: always
    ports:
      - 27012:27017
    volumes:
      - ./keyfile:/data/keyfile
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: 123123
    command: mongod --auth --keyFile /data/keyfile/mongoReplSet-keyfile --bind_ip_all --replSet rs0

  mongo3:
    image: mongo
    hostname: mongo3
    container_name: mongo3
    restart: always
    ports:
      - 27013:27017
    volumes:
      - ./keyfile:/data/keyfile
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: 123123
    command: mongod --auth --keyFile /data/keyfile/mongoReplSet-keyfile --bind_ip_all --replSet rs0

docker exec -it <container> mongo
mongo -u root -p 123123
use admin
rs.initiate(
  {
    _id : 'rs0',
    members: [
      { _id : 0, host : "10.8.99.44:27011", priority:3 },
      { _id : 1, host : "10.8.99.44:27012", priority:3 },
      { _id : 2, host : "10.8.99.44:27013", priority:3 }
    ]
  }
)

rs.reconfig(
  {
    _id : 'rs0',
    members: [
      { _id : 0, host : "10.8.99.44:27011", priority:3 },
      { _id : 1, host : "10.8.99.44:27012", priority:1 },
      { _id : 2, host : "10.8.99.44:27013", priority:1 }
    ]
  }
)

rs.config()

rs.reconfig(
  {
    _id : 'rs0',
    members: [
      { _id : 0, host : "192.168.199.98:27011", priority:3 },
      { _id : 1, host : "192.168.199.98:27012", priority:1 },
      { _id : 2, host : "192.168.199.98:27013", priority:1 }
    ]
  },
  {
    "force":true
  }
)

PRIMARY> config=rs.conf()
PRIMARY>config.members[0].priority = 6
PRIMARY> rs.reconfig(config)

查看副本集状态
rs.status()
rs.isMaster()
查看副本集配置
rs.conf()
查看Secondary同步状态
rs.printReplicationInfo()
rs.printSecondaryReplicationInfo()

config.mongoose = {
  url: 'mongodb://10.8.99.44:27011,10.8.99.44:27012,10.8.99.44:27013/log',
  options: {
    auth: {
      user: 'admin',
      password: '123123',
    },
    replicaSet: 'rs0', // 副本集
    readPreference: 'secondaryPreferred', // 读哪个副本集 从哪个副本集读数据
    readConcern: { level: 'majority' }, // 读的策略 解决脏读 决定这个节点上的数据哪些是可读的，类似于关系数据库的隔离级别 读取在大多数节点上提交完成的数据;
    // write concern 当配置majority时，大多数节点为2，Primary感知到两个节点被写入完成时就代表成功写入
    writeConcern: {
      w: 'majority',
      // j，该参数表示是否写操作要进行journal持久化之后才向用户确认；
      // {j: true} 要求primary写操作进行了journal持久化之后才向用户确认；
      // {j: false} 要求写操作已经在journal缓存中即可向用户确认；journal后续会将持久化到磁盘，默认是100ms；
      // j: true,
      j: true,
      wtimeout: 1000,
    },
    keepAlive: true,
    keepAliveInitialDelay: 300000,
    useNewUrlParser: true,
    useFindAndModify: false,
    useCreateIndex: true,
    useUnifiedTopology: true,
    serverSelectionTimeoutMS: 30000,
    socketTimeoutMS: 45000,
  },
}

module.exports = {
  async getSession(
    opt = {
      readPreference: { mode: 'primary' },
    }
  ) {
    const { mongoose } = this.app
    const session = await mongoose.startSession(opt)
    await session.startTransaction({
      readConcern: { level: 'majority' },
      writeConcern: { w: 'majority' },
    })
    return session
  },
}

async transaction() {
    const { ctx } = this
    const session = await this.ctx.getSession()
    const db = this.app.mongoose.connection
    try {
      const res1 = await ctx.model.VpnLog.create(
        [
          {
            mac_address: 'test',
            session: 'test',
            source_ip: 'test',
            destination_ip: 'test',
            hub: 'test',
            items_per_second: 111,
            created_at: new Date(),
          },
        ],
        { session }
      )
      throw new Error('出错')   -> 出错，如其他程序的操作错误
      const res2 = await ctx.model.VpnLog.create(
        [
          {
            mac_address: 'test2',
            session: 'test2',
            source_ip: 'test2',
            destination_ip: 'test2',
            hub: 'test2',
            items_per_second: 222,
            created_at: new Date(),
          },
        ],
        { session }
      )
      await session.commitTransaction()
      session.endSession()
      return {
        res1,
        res2,
      }
    } catch (error) {
      console.log('开始回滚')
      await session.abortTransaction()
      session.endSession()
      console.log(error)
    }
  }

// 开启mongoose调试
this.app.mongoose.set('debug', true)

module.exports = (app) => {
  const mongoose = app.mongoose
  const Schema = mongoose.Schema

  const FileSchema = new Schema(
    {
      fileName: { type: String }, // 文件名
      uploadDate: { type: Date }, // 上传创建日期
      creatorName: { type: String }, // 创建者姓名
    },
    {
      collection: 'sys_files',
    }
  )

  return mongoose.model('File', FileSchema)
}

Mongoose: sys_files.insertOne({ isDel: false, _id: ObjectId("61b5599cba1fcfeeb79c57cd"), fileName: '文件名', uploadDate: new Date("Sat, 11 Dec 2021 16:00:00 GMT"), creatorName: '张三', __v: 0}, { session: null })

const file = await ctx.model.Delivery.File.create({
  fileName: '文件名',
  uploadDate: new Date('2021-12-12'),
  creatorName: '刘九',
})

toJSON: {
  virtuals: true,
},
toObject: {
  virtuals: true,
},

personSchema.virtual('fullName').get(function () {
  return this.name.first + ' ' + this.name.last;
});

var Person = mongoose.model('Person', personSchema);

var p = new Person({
  name: { first: 'junyao', last: 'hong' }
});

p.name.fullName // junyao hong

BookSchema.virtual('author', {
  ref: 'authors',
  localField: '_id',
  foreignField: 'books',
  justOne: true,
})

const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const authorSchema = new Schema({
    "author": String,
    "age": Number,
    "books": [{
        type: Schema.Types.ObjectId,
        ref: 'Book' // 关联的Model
    }]
});
module.exports = mongoose.model("Author", authorSchema, "authors"); // 分别为Model名、Schema、数据库中集合名

let result = await Author.find({
    "author": "Zander"
  }).populate("books");

[
        {
            "books": [
                {
                    "_id": "5dccfcb5a3fab06c89020c8d",
                    "name": "代码的弱点"
                },
                {
                    "_id": "5dccfd30a3fab06c89020caa",
                    "name": "代码与六便士"
                },
                {
                    "_id": "5dccfda6a3fab06c89020cc3",
                    "name": "代码失格"
                }
            ],
            "_id": "5dccfc3aa3fab06c89020c65",
            "author": "Zander",
            "age": 18
        }
    ]

let result = await Author.aggregate([{ // 操作的Model为Author
    $lookup: {
      from: "books", // 数据库中关联的集合名
      localField: "books", // author文档中关联的字段
      foreignField: "_id", // book文档中关联的字段
      as: "bookList" // 返回数据的字段名
    }
  }, {
    $match: { // 筛选条件
      "author": "Zander"
    }
  }]);

[
        {
            "_id": "5dccfc3aa3fab06c89020c65",
            "author": "Zander",
            "age": 18,
            "bookList": [
                {
                    "_id": "5dccfcb5a3fab06c89020c8d",
                    "name": "代码的弱点"
                },
                {
                    "_id": "5dccfd30a3fab06c89020caa",
                    "name": "代码与六便士"
                },
                {
                    "_id": "5dccfda6a3fab06c89020cc3",
                    "name": "代码失格"
                }
            ]
        }
    ]

$lookup: {
  from: "authors",
  localField: "_id",
  foreignField: "books",
  as: "author"
}

$lookup: {
  from: "authors",
  localField: "_id",
  foreignField: "books",
  as: "author"
}

const lookup = {
  $lookup: {
    from: 'sys_invoices',
    localField: 'invoice',
    foreignField: '_id',
    as: 'invoice',
  },
}

const project = {
  $project: {
    invoice: { $arrayElemAt: ['$invoice', 0] },
  },
}

//写入文档
PUT mapping_test/_doc/1
{
  "firstName":"Lee",
  "lastName":"Crazy",
  "loginDate":"2019-10-22T21:08:48"
}
//查看Mapping 文件
GET mapping_test/_mapping
{
  "mapping_test" : {
    "mappings" : {
      "properties" : {
        "firstName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "lastName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "loginDate" : {
          "type" : "date"
        }
      }
    }
  }
}

//dynamic mapping 推断字符的类型
PUT mapping_test/_doc/1
{
  "uid":"123",
  "isVip": false,
  "isAdmin":"true",
  "age": 18,
  "heigh" : 180
}
//返回结果
{
  "mapping_test" : {
    "mappings" : {
      "properties" : {
        "age" : {
          "type" : "long"
        },
        "heigh" : {
          "type" : "long"
        },
        "isAdmin" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "isVip" : {
          "type" : "boolean"
        },
        "uid" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

PUT users
{
  "mappings" : {
    "properties" : {
      "firstName" : {
        "type" : "text"
      },
      "lastName" : {
        "type" : "text"
      },
      "mobile" : {
        "type" : "text",
        "index": false
      }
    }
  }
}

DELETE users
PUT users
{
  "mappings" : {
    "properties" : {
      "firstName" : {
        "type" : "text"
      },
      "lastName" : {
        "type" : "text"
      },
      "mobile" : {
        "type" : "keyword", //这个如果是text 无法设置为空
        "null_value": "NULL"
      }
    }
  }
}
PUT users/_doc/2
{
"firstName":"Li",
"lastName": "Sunke",
"mobile": null
}
GET users/_search?q=mobile:NULL
//搜索结果
"_source" : {
        "firstName" : "Li",
        "lastName" : "Sunke",
        "mobile" : null
      }

DELETE users
PUT users
{
"mappings": {
  "properties": {
    "firstName":{
      "type": "text",
      "copy_to": "fullName"
    },
    "lastName":{
      "type": "text",
      "copy_to": "fullName"
    }
  }
}
}
PUT users/_doc/1
{
  "firstName":"Li",
  "lastName": "Sunke"
}
//没有新建字段
GET users/_doc/1
{
  "_index" : "users",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "firstName" : "Li",
    "lastName" : "Sunke"
  }
}
GET users/_search?q=fullName:(Li sunke)

GET _analyze
{
"analyzer": "stop",
"text": "2 running Quick brown-foxes leap over lazy dogs in the summer evening."
}

GET _analyze
{
  "analyzer": "standard",
  "text" : "Mastering Elasticsearch , elasticsearch in Action"
}
//返回结果
{
  "tokens" : [
    {
      "token" : "mastering",
      "start_offset" : 0,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "elasticsearch",
      "start_offset" : 10,
      "end_offset" : 23,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "elasticsearch",
      "start_offset" : 26,
      "end_offset" : 39,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "in",
      "start_offset" : 40,
      "end_offset" : 42,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "action",
      "start_offset" : 43,
      "end_offset" : 49,
      "type" : "<ALPHANUM>",
      "position" : 4
    }
  ]
}

POST /_analyze
{
  "tokenizer": "standard", 
  "filter": ["lowercase"],
  "text": "Mastering Elasticesearch"
}
//结果返回
{
  "tokens" : [
    {
      "token" : "mastering",
      "start_offset" : 0,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "elasticesearch",
      "start_offset" : 10,
      "end_offset" : 24,
      "type" : "<ALPHANUM>",
      "position" : 1
    }
  ]
}

裙子,裙
西红柿,番茄
china,中国,中华人民共和国
男生,男士,man
女生,女士,women

POST _analyze
{
"tokenizer":"keyword",
"char_filter":["html_strip"],
"text": "<b>hello world</b>"
}
//结果
{
"tokens" : [
  {
    "token" : "hello world",
    "start_offset" : 3,
    "end_offset" : 18,
    "type" : "word",
    "position" : 0
  }
]
}

POST _analyze
{
"tokenizer": "standard",
"char_filter": [
    {
      "type" : "mapping",
      "mappings" : [ "- => _"]
    }
  ],
"text": "123-456, I-test! test-990 650-555-1234"
}
//返回
{
"tokens" : [
  {
    "token" : "123_456",
    "start_offset" : 0,
    "end_offset" : 7,
    "type" : "<NUM>",
    "position" : 0
  },
  {
    "token" : "I_test",
    "start_offset" : 9,
    "end_offset" : 15,
    "type" : "<ALPHANUM>",
    "position" : 1
  },
  {
    "token" : "test_990",
    "start_offset" : 17,
    "end_offset" : 25,
    "type" : "<ALPHANUM>",
    "position" : 2
  },
  {
    "token" : "650_555_1234",
    "start_offset" : 26,
    "end_offset" : 38,
    "type" : "<NUM>",
    "position" : 3
  }
]
}

GET _analyze
{
"tokenizer": "standard",
"char_filter": [
    {
      "type" : "pattern_replace",
      "pattern" : "http://(.*)",
      "replacement" : "$1"
    }
  ],
  "text" : "http://www.elastic.co"
}
//返回
{
"tokens" : [
  {
    "token" : "www.elastic.co",
    "start_offset" : 0,
    "end_offset" : 21,
    "type" : "<ALPHANUM>",
    "position" : 0
  }
]
}

POST _analyze
{
"tokenizer":"path_hierarchy",
"text":"/user/ymruan/a"
}
{
"tokens" : [
  {
    "token" : "/user",
    "start_offset" : 0,
    "end_offset" : 5,
    "type" : "word",
    "position" : 0
  },
  {
    "token" : "/user/ymruan",
    "start_offset" : 0,
    "end_offset" : 12,
    "type" : "word",
    "position" : 0
  },
  {
    "token" : "/user/ymruan/a",
    "start_offset" : 0,
    "end_offset" : 14,
    "type" : "word",
    "position" : 0
  }
]
}

GET _analyze
{
"tokenizer": "whitespace", 
"filter": ["stop","snowball"], //on the a
"text": ["The gilrs in China are playing this game!"]
}
{
"tokens" : [
  {
    "token" : "The", //大写的The 不做过滤
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "word",
    "position" : 0
  },
  {
    "token" : "gilr",
    "start_offset" : 4,
    "end_offset" : 9,
    "type" : "word",
    "position" : 1
  },
  {
    "token" : "China",
    "start_offset" : 13,
    "end_offset" : 18,
    "type" : "word",
    "position" : 3
  },
  {
    "token" : "play",
    "start_offset" : 23,
    "end_offset" : 30,
    "type" : "word",
    "position" : 5
  },
  {
    "token" : "game!",
    "start_offset" : 36,
    "end_offset" : 41,
    "type" : "word",
    "position" : 7
  }
]
}

自定义分析器标准格式是：
PUT /my_index
{
  "settings": {
      "analysis": {
          "char_filter": { ... custom character filters ... },//字符过滤器
          "tokenizer": { ... custom tokenizers ... },//分词器
          "filter": { ... custom token filters ... }, //词单元过滤器
          "analyzer": { ... custom analyzers ... }
      }
  }
}

#定义自己的分词器
PUT my_index
{
"settings": {
  "analysis": {
    "analyzer": {
      "my_custom_analyzer":{
        "type":"custom",
        "char_filter":[
          "emoticons"
        ],
        "tokenizer":"punctuation",
        "filter":[
          "lowercase",
          "english_stop"
        ]
      }
    },
    "tokenizer": {
      "punctuation":{
        "type":"pattern",
        "pattern": "[ .,!?]"
      }
    },
    "char_filter": {
      "emoticons":{
        "type":"mapping",
        "mappings" : [ 
          ":) => happy",
          ":( => sad"
        ]
      }
    },
    "filter": {
      "english_stop":{
        "type":"stop",
        "stopwords":"_english_"
      }
    }
  }
}
}

POST my_index/_analyze
{
  "field": "title",
  "text": "Elizabeth is the English queen",
  "analyzer": "my_custom_analyzer"
}

{
  "tokens" : [
    {
      "token" : "elizabeth",
      "start_offset" : 0,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "is",
      "start_offset" : 10,
      "end_offset" : 12,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "the",
      "start_offset" : 13,
      "end_offset" : 16,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "english",
      "start_offset" : 17,
      "end_offset" : 24,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "british",
      "start_offset" : 17,
      "end_offset" : 24,
      "type" : "SYNONYM",
      "position" : 3
    },
    {
      "token" : "queen",
      "start_offset" : 25,
      "end_offset" : 30,
      "type" : "<ALPHANUM>",
      "position" : 4
    },
    {
      "token" : "monarch",
      "start_offset" : 25,
      "end_offset" : 30,
      "type" : "SYNONYM",
      "position" : 4
    }
  ]
}

PUT my_index/_doc/1
{
  "title": "Elizabeth is the English queen"
}


GET my_index/_search
{
  "query": {
    "match": {
      "title": "monarch"
    }
  }
}

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "my_index2",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.2876821,
        "_source" : {
          "title" : "Elizabeth is the English queen"
        }
      }
    ]
  }
}

PUT my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "word_syn"
          ]
        }
      },
      "filter": {
        "word_syn": {
          "type": "synonym_graph",
          "synonyms_path": "analysis/synonym.txt",
          ## 开启热更新
          "updateable": true  
        },
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "english",
        "search_analyzer": "my_custom_analyzer"
      },
      "author": {
        "type": "keyword"
      }
    }
  }
}

POST my_index/_reload_search_analyzers

PUT test/_doc/1
{
  "msg":"Eating an apple a day keeps doctor away"
}

POST test/_search
{
  "query":{
    "match":{
      "msg":"eat"
    }
  }
}

POST test/_analyze
{
  "field": "msg",
  "text": "Eating an apple a day keeps doctor away"
}

{
  "tokens": [
    {
      "token": "eating",
      "start_offset": 0,
      "end_offset": 6,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "an",
      "start_offset": 7,
      "end_offset": 9,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "apple",
      "start_offset": 10,
      "end_offset": 15,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "a",
      "start_offset": 16,
      "end_offset": 17,
      "type": "<ALPHANUM>",
      "position": 3
    },
    {
      "token": "day",
      "start_offset": 18,
      "end_offset": 21,
      "type": "<ALPHANUM>",
      "position": 4
    },
    {
      "token": "keeps",
      "start_offset": 22,
      "end_offset": 27,
      "type": "<ALPHANUM>",
      "position": 5
    },
    {
      "token": "doctor",
      "start_offset": 28,
      "end_offset": 34,
      "type": "<ALPHANUM>",
      "position": 6
    },
    {
      "token": "away",
      "start_offset": 35,
      "end_offset": 39,
      "type": "<ALPHANUM>",
      "position": 7
    }
  ]
}

PUT test/_doc/_mapping?include_type_name=true
{
  "properties": {
    "msg_english":{
      "type":"text",
      "analyzer": "english"
    }
  }
}

POST test/_search
  {
    "query":{
      "match":{
        "msg":{
          "query": "eating",
          "analyzer": "english"
        }
      }
    }
  }

POST _analyze
{
  "text":"eating",
  "analyzer":"english"
}

{
  "tokens": [
    {
      "token": "eat",
      "start_offset": 0,
      "end_offset": 6,
      "type": "<ALPHANUM>",
      "position": 0
    }
  ]
}

# 增加字段 msg_english，与 msg 做对比
PUT test/_doc/_mapping?include_type_name=true
{
  "properties": {
    "msg_english":{
      "type":"text",
      "analyzer": "english"
    }
  }
}

# 写入相同文档
PUT test/doc/1
{
  "msg":"Eating an apple a day keeps doctor away",
  "msg_english":"Eating an apple a day keeps doctor away"
}

# 搜索 msg_english 字段
POST test/_search
{
  "query": {
    "match": {
      "msg_english": "eat"
    }
  }
}

FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
docker network create flink-network

docker run \
--rm \
--name=jobmanager \
--network flink-network \
--publish 8081:8081 \
--env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:1.13.1-scala_2.11 jobmanager

docker run \
--rm \
--name=taskmanager \
--network flink-network \
--env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:1.13.1-scala_2.11 taskmanager

// 停止集群
bin/stop-cluster.sh
// 启动集群
bin/start-cluster.sh
// 进入flink sql 客户端命令行界面
bin/sql-client.sh embedded
// 查看当前运行的jobs
bin/flink list
// 查看所有的任务，包括失败、成功、取消的
bin/flink list -a
// 取消命令
bin/flink cancel jobID

version: '2.1'
services:
  postgres:
    image: debezium/example-postgres:1.1
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_PASSWORD=1234
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
  mysql:
    image: debezium/example-mysql:1.1
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=123456
      - MYSQL_USER=mysqluser
      - MYSQL_PASSWORD=mysqlpw
  elasticsearch:
    image: elastic/elasticsearch:7.6.0
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    ports:
      - "9200:9200"
      - "9300:9300"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
  kibana:
    image: elastic/kibana:7.6.0
    ports:
      - "5601:5601"
  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:2.12-2.2.1
    ports:
      - "9092:9092"
      - "9094:9094"
    depends_on:
      - zookeeper
    environment:
      - KAFKA_ADVERTISED_LISTENERS=INSIDE://:9094,OUTSIDE://localhost:9092
      - KAFKA_LISTENERS=INSIDE://:9094,OUTSIDE://:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CREATE_TOPICS="user_behavior:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

docker-compose exec mysql mysql -uroot -p123456

-- MySQL
CREATE DATABASE mydb;
USE mydb;
CREATE TABLE products (
  id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  description VARCHAR(512)
);
ALTER TABLE products AUTO_INCREMENT = 101;

INSERT INTO products
VALUES (default,"scooter","Small 2-wheel scooter"),
       (default,"car battery","12V car battery"),
       (default,"12-pack drill bits","12-pack of drill bits with sizes ranging from #40 to #3"),
       (default,"hammer","12oz carpenter's hammer"),
       (default,"hammer","14oz carpenter's hammer"),
       (default,"hammer","16oz carpenter's hammer"),
       (default,"rocks","box of assorted rocks"),
       (default,"jacket","water resistent black wind breaker"),
       (default,"spare tire","24 inch spare tire");

CREATE TABLE orders (
  order_id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
  order_date DATETIME NOT NULL,
  customer_name VARCHAR(255) NOT NULL,
  price DECIMAL(10, 5) NOT NULL,
  product_id INTEGER NOT NULL,
  order_status BOOLEAN NOT NULL -- 是否下单
) AUTO_INCREMENT = 10001;

INSERT INTO orders
VALUES (default, '2020-07-30 10:08:22', 'Jark', 50.50, 102, false),
       (default, '2020-07-30 10:11:09', 'Sally', 15.00, 105, false),
       (default, '2020-07-30 12:00:30', 'Edward', 25.25, 106, false);

docker-compose exec postgres psql -h localhost -U postgres

-- PG
CREATE TABLE shipments (
  shipment_id SERIAL NOT NULL PRIMARY KEY,
  order_id SERIAL NOT NULL,
  origin VARCHAR(255) NOT NULL,
  destination VARCHAR(255) NOT NULL,
  is_arrived BOOLEAN NOT NULL
);
ALTER SEQUENCE public.shipments_shipment_id_seq RESTART WITH 1001;
ALTER TABLE public.shipments REPLICA IDENTITY FULL;

INSERT INTO shipments
VALUES (default,10001,'Beijing','Shanghai',false),
       (default,10002,'Hangzhou','Shanghai',false),
       (default,10003,'Shanghai','Hangzhou',false);