AOP实现系统告警

Posted 2022-09-21 格格巫 MMQ!!

一、业务背景

在开发的过程中会遇到各种各样的开发问题，服务器宕机、网络抖动、代码本身的bug等等。针对代码的bug，我们可以提前预支，通过发送告警信息来警示我们去干预，尽早处理。

二、告警的方式

1、钉钉告警
通过在企业钉钉群，添加群机器人的方式，通过机器人向群内发送报警信息。至于钉钉机器人怎么创建，发送消息的api等等，请参考官方文档

2、企业微信告警
同样的套路，企业微信也是，在企业微信群中，添加群机器人。通过机器人发送告警信息。具体请看官方文档

3、邮件告警
与上述不同的是，邮件是发送给个人的，当然也可以是批量发送，只实现了发送文本格式的方式，至于markdown格式，有待考察。邮件发送相对比较简单，这里就不展开赘述。

三、源码解析

1、Alarm自定义注解
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
public @interface Alarm

/**
 * 报警标题
 *
 * @return String
 */
String title() default "";

/**
 * 发送报警格式：目前支持text，markdown
 * @return
 */
MessageTye messageType() default MessageTye.TEXT;

/**
 * 告警模板id
 * @return
 */
String templateId() default "";

/**
 * 成功是否通知：true-通知，false-不通知
 * @return
 */
boolean successNotice() default false;

1.1、注解使用
@Alarm标记在方法上使用，被标记的方法发生异常，会根据配置，读取配置信息，发送异常堆栈信息。使用方法如下所示：

@Alarm(title = “某某业务告警”, messageType = MessageTye.MARKDOWN, templateId = “errorTemp”)
1.2、注解字段解析
title
告警消息标题：可以定义为业务信息，如导师身份计算

messageType
告警消息展示类型：目前支持text文本类型，markdown类型

templateId
消息模板id：与配置文件中配置的模板id一致

successNotice
正常情况是否也需要发送告警信息，默认值是fasle，表示不需要发送。当然，有些业务场景正常情况也需要发送，比如：支付出单通知等。

2、配置文件分析
2.1、钉钉配置文件
spring:
alarm:
dingtalk:
# 开启钉钉发送告警
enabled: true
# 钉钉群机器人唯一的token
token: xxxxxx
# 安全设置：加签的密钥
secret: xxxxxxx
2.2、企业微信配置文件
spring:
alarm:
wechat:
# 开启企业微信告警
enabled: true
# 企业微信群机器人唯一key
key: xxxxxdsf
# 被@人的手机号
to-user: 1314243
2.3、邮件配置文件
spring:
alarm:
mail:
enabled: true
smtpHost: [e-mail]
smtpPort: 22
to: [e-mail]
from: [e-mail]
username: wsrf
password: xxx
2.4、自定义模板配置
spring:
alarm:
template:
# 开启通过模板配置
enabled: true
# 配置模板来源为文件
source: FILE
# 配置模板数据
templates:
errorTemp:
templateId: errorTemp
templateName: 服务异常模板
templateContent: 这里是配置模板的内容
spring:alarm:template:enabled，Boolean类型，表示开启告警消息使用模板发送。
spring:alarm:template:source，模板来源，枚举类：JDBC（数据库）、FILE（配置文件）、MEMORY（内存），目前只支持FILE，其他两种可自行扩展。
spring:alarm:template:templates，配置模板内容，是一个map，errorTemp是模板id，需要使用哪种模板，就在@Alarm中的templateId设置为对应配置文件中的templateId。
3、核心AOP分析
3.1、原理分析

3.2、自定义切面
@Aspect
@Slf4j
@RequiredArgsConstructor
public class AlarmAspect
private final AlarmTemplateProvider alarmTemplateProvider;

private final static String ERROR_TEMPLATE = "\\n\\n<font color=\\"#F37335\\">异常信息:</font>\\n" +
        "```java\\n" +
        "#[exception]\\n" +
        "```\\n";

private final static String TEXT_ERROR_TEMPLATE = "\\n异常信息:\\n" +
        "#[exception]";

private final static String MARKDOWN_TITLE_TEMPLATE = "# 【#[title]】\\n" +
        "\\n请求状态：<font color=\\"#[stateColor]\\">#[state]</font>\\n\\n";

private final static String TEXT_TITLE_TEMPLATE = "【#[title]】\\n" +
        "请求状态：#[state]\\n";

@Pointcut("@annotation(alarm)")
public void alarmPointcut(Alarm alarm) 



@Around(value = "alarmPointcut(alarm)", argNames = "joinPoint,alarm")
public Object around(ProceedingJoinPoint joinPoint, Alarm alarm) throws Throwable 
    Object result = joinPoint.proceed();
    if (alarm.successNotice()) 
        String templateId = alarm.templateId();
        String fileTemplateContent = "";
        if (Objects.nonNull(alarmTemplateProvider)) 
            AlarmTemplate alarmTemplate = alarmTemplateProvider.loadingAlarmTemplate(templateId);
            fileTemplateContent = alarmTemplate.getTemplateContent();
        
        String templateContent = "";
        MessageTye messageTye = alarm.messageType();
        if (messageTye.equals(MessageTye.TEXT)) 
            templateContent = TEXT_TITLE_TEMPLATE.concat(fileTemplateContent);
         else if (messageTye.equals(MessageTye.MARKDOWN)) 
            templateContent = MARKDOWN_TITLE_TEMPLATE.concat(fileTemplateContent);
        
        Map<String, Object> alarmParamMap = new HashMap<>();
        alarmParamMap.put("title", alarm.title());
        alarmParamMap.put("stateColor", "#45B649");
        alarmParamMap.put("state", "成功");
        sendAlarm(alarm, templateContent, alarmParamMap);
    
    return result;



@AfterThrowing(pointcut = "alarmPointcut(alarm)", argNames = "joinPoint,alarm,e", throwing = "e")
public void doAfterThrow(JoinPoint joinPoint, Alarm alarm, Exception e) 
    log.info("请求接口发生异常 : []", e.getMessage());
    String templateId = alarm.templateId();
    // 加载模板中配置的内容，若有
    String templateContent = "";
    String fileTemplateContent = "";
    if (Objects.nonNull(alarmTemplateProvider)) 
        AlarmTemplate alarmTemplate = alarmTemplateProvider.loadingAlarmTemplate(templateId);
        fileTemplateContent = alarmTemplate.getTemplateContent();
    
    MessageTye messageTye = alarm.messageType();
    if (messageTye.equals(MessageTye.TEXT)) 
        templateContent = TEXT_TITLE_TEMPLATE.concat(fileTemplateContent).concat(TEXT_ERROR_TEMPLATE);
     else if (messageTye.equals(MessageTye.MARKDOWN)) 
        templateContent = MARKDOWN_TITLE_TEMPLATE.concat(fileTemplateContent).concat(ERROR_TEMPLATE);
    
    Map<String, Object> alarmParamMap = new HashMap<>();
    alarmParamMap.put("title", alarm.title());
    alarmParamMap.put("stateColor", "#FF4B2B");
    alarmParamMap.put("state", "失败");
    alarmParamMap.put("exception", ExceptionUtil.stacktraceToString(e));
    sendAlarm(alarm, templateContent, alarmParamMap);


private void sendAlarm(Alarm alarm, String templateContent, Map<String, Object> alarmParamMap) 
    ExpressionParser parser = new SpelExpressionParser();
    TemplateParserContext parserContext = new TemplateParserContext();
    String message = parser.parseExpression(templateContent, parserContext).getValue(alarmParamMap, String.class);
    MessageTye messageTye = alarm.messageType();
    NotifyMessage notifyMessage = new NotifyMessage();
    notifyMessage.setTitle(alarm.title());
    notifyMessage.setMessageTye(messageTye);
    notifyMessage.setMessage(message);
    AlarmFactoryExecute.execute(notifyMessage);

4、模板提供器
4.1、AlarmTemplateProvider
定义一个抽象接口AlarmTemplateProvider，用于被具体的子类实现

public interface AlarmTemplateProvider

/**
 * 加载告警模板
 *
 * @param templateId 模板id
 * @return AlarmTemplate
 */
AlarmTemplate loadingAlarmTemplate(String templateId);

4.2、BaseAlarmTemplateProvider
抽象类BaseAlarmTemplateProvider实现该抽象接口

public abstract class BaseAlarmTemplateProvider implements AlarmTemplateProvider

@Override
public AlarmTemplate loadingAlarmTemplate(String templateId) 
    if (StringUtils.isEmpty(templateId)) 
        throw new AlarmException(400, "告警模板配置id不能为空");
    
    return getAlarmTemplate(templateId);


/**
 * 查询告警模板
 *
 * @param templateId 模板id
 * @return AlarmTemplate
 */
abstract AlarmTemplate getAlarmTemplate(String templateId);

4.3、YamlAlarmTemplateProvider
具体实现类YamlAlarmTemplateProvider，实现从配置文件中读取模板，该类在项目启动时，会被加载进spring的bean容器

@RequiredArgsConstructor
public class YamlAlarmTemplateProvider extends BaseAlarmTemplateProvider

private final TemplateConfig templateConfig;

@Override
AlarmTemplate getAlarmTemplate(String templateId) 
    Map<String, AlarmTemplate> configTemplates = templateConfig.getTemplates();
    AlarmTemplate alarmTemplate = configTemplates.get(templateId);
    if (ObjectUtils.isEmpty(alarmTemplate)) 
        throw new AlarmException(400, "未发现告警配置模板");
    
    return alarmTemplate;

4.4、MemoryAlarmTemplateProvider和JdbcAlarmTemplateProvider
抽象类BaseAlarmTemplateProvider还有其他两个子类，分别是MemoryAlarmTemplateProvider和JdbcAlarmTemplateProvider。但是这两个子类暂时还未实现逻辑，后续可以自行扩展。

@RequiredArgsConstructor
public class MemoryAlarmTemplateProvider extends BaseAlarmTemplateProvider

private final Function<String, AlarmTemplate> function;
@Override
AlarmTemplate getAlarmTemplate(String templateId) 
    AlarmTemplate alarmTemplate = function.apply(templateId);
    if (ObjectUtils.isEmpty(alarmTemplate)) 
        throw new AlarmException(400, "未发现告警配置模板");
    
    return alarmTemplate;

@RequiredArgsConstructor
public class JdbcAlarmTemplateProvider extends BaseAlarmTemplateProvider

private final Function<String, AlarmTemplate> function;

@Override
AlarmTemplate getAlarmTemplate(String templateId) 
    AlarmTemplate alarmTemplate = function.apply(templateId);
    if (ObjectUtils.isEmpty(alarmTemplate)) 
        throw new AlarmException(400, "未发现告警配置模板");
    
    return alarmTemplate;

两个类中都有Function<String, AlarmTemplate>接口，为函数式接口，可以供外部自行去实现逻辑。

5、告警发送
5.1、AlarmFactoryExecute
该类内部保存了一个容器，主要用于缓存真正的发送类

public class AlarmFactoryExecute

private static List<AlarmWarnService> serviceList = new ArrayList<>();

public AlarmFactoryExecute(List<AlarmWarnService> alarmLogWarnServices) 
    serviceList = alarmLogWarnServices;


public static void addAlarmLogWarnService(AlarmWarnService alarmLogWarnService) 
    serviceList.add(alarmLogWarnService);


public static List<AlarmWarnService> getServiceList() 
    return serviceList;


public static void execute(NotifyMessage notifyMessage) 
    for (AlarmWarnService alarmWarnService : getServiceList()) 
        alarmWarnService.send(notifyMessage);
    

5.2、AlarmWarnService
抽象接口，只提供一个发送的方法

public interface AlarmWarnService

/**
 * 发送信息
 *
 * @param notifyMessage message
 */
void send(NotifyMessage notifyMessage);

5.3、BaseWarnService
与抽象的模板提供器AlarmTemplateProvider一样的套路，该接口有一个抽象的实现类BaseWarnService,该类对外暴露send方法，用于发送消息，内部用doSendMarkdown，doSendText方法实现具体的发送逻辑，当然具体发送逻辑还是得由其子类去实现。

@Slf4j
public abstract class BaseWarnService implements AlarmWarnService

@Override
public void send(NotifyMessage notifyMessage) 
    if (notifyMessage.getMessageTye().equals(MessageTye.TEXT)) 
        CompletableFuture.runAsync(() -> 
            try 
                doSendText(notifyMessage.getMessage());
             catch (Exception e) 
                log.error("send text warn message error", e);
            
        );
     else if (notifyMessage.getMessageTye().equals(MessageTye.MARKDOWN)) 
        CompletableFuture.runAsync(() -> 
            try 
                doSendMarkdown(notifyMessage.getTitle(), notifyMessage.getMessage());
             catch (Exception e) 
                log.error("send markdown warn message error", e);
            
        );
    


/**
 * 发送Markdown消息
 *
 * @param title   Markdown标题
 * @param message Markdown消息
 * @throws Exception 异常
 */
protected abstract void doSendMarkdown(String title, String message) throws Exception;

/**
 * 发送文本消息
 *
 * @param message 文本消息
 * @throws Exception 异常
 */
protected abstract void doSendText(String message) throws Exception;

5.4、DingTalkWarnService
主要实现了钉钉发送告警信息的逻辑

@Slf4j
public class DingTalkWarnService extends BaseWarnService

private static final String ROBOT_SEND_URL = "https://oapi.dingtalk.com/robot/send?access_token=";
private final String token;

private final String secret;

public DingTalkWarnService(String token, String secret) 
    this.token = token;
    this.secret = secret;


public void sendRobotMessage(DingTalkSendRequest dingTalkSendRequest) throws Exception 
    String json = JSONUtil.toJsonStr(dingTalkSendRequest);
    String sign = getSign();
    String body = HttpRequest.post(sign).contentType(ContentType.JSON.getValue()).body(json).execute().body();
    log.info("钉钉机器人通知结果：", body);


/**
 * 获取签名
 *
 * @return 返回签名
 */
private String getSign() throws Exception 
    long timestamp = System.currentTimeMillis();
    String stringToSign = timestamp + "\\n" + secret;
    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
    byte[] signData = mac.doFinal(stringToSign.getBytes(StandardCharsets.UTF_8));
    return ROBOT_SEND_URL + token + "&timestamp=" + timestamp + "&sign=" + URLEncoder.encode(new String(Base64.getEncoder().encode(signData)), StandardCharsets.UTF_8.toString());


@Override
protected void doSendText(String message) throws Exception 
    DingTalkSendRequest param = new DingTalkSendRequest();
    param.setMsgtype(DingTalkSendMsgTypeEnum.TEXT.getType());
    param.setText(new DingTalkSendRequest.Text(message));
    sendRobotMessage(param);


@Override
protected void doSendMarkdown(String title, String message) throws Exception 
    DingTalkSendRequest param = new DingTalkSendRequest();
    param.setMsgtype(DingTalkSendMsgTypeEnum.MARKDOWN.getType());
    DingTalkSendRequest.Markdown markdown = new DingTalkSendRequest.Markdown(title, message);
    param.setMarkdown(markdown);
    sendRobotMessage(param);

5.5、WorkWeXinWarnService
主要实现了发送企业微信告警信息的逻辑

@Slf4j
public class WorkWeXinWarnService extends BaseWarnService
private static final String SEND_MESSAGE_URL = “https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=%s”;
private final String key;

private final String toUser;

public WorkWeXinWarnService(String key, String toUser) 
    this.key = key;
    this.toUser = toUser;


private String createPostData(WorkWeXinSendMsgTypeEnum messageTye, String contentValue) 
    WorkWeXinSendRequest wcd = new WorkWeXinSendRequest();
    wcd.setMsgtype(messageTye.getType());
    List<String> toUsers = Arrays.asList("@all");
    if (StringUtils.isNotEmpty(toUser)) 
        String[] split = toUser.split("\\\\|");
        toUsers = Arrays.asList(split);
    
    if (messageTye.equals(WorkWeXinSendMsgTypeEnum.TEXT)) 
        WorkWeXinSendRequest.Text text = new WorkWeXinSendRequest.Text(contentValue, toUsers);
        wcd.setText(text);
     else if (messageTye.equals(WorkWeXinSendMsgTypeEnum.MARKDOWN)) 
        WorkWeXinSendRequest.Markdown markdown = new WorkWeXinSendRequest.Markdown(contentValue, toUsers);
        wcd.setMarkdown(markdown);
    
    return JSONUtil.toJsonStr(wcd);


@Override
protected void doSendText(String message) 
    String data = createPostData(WorkWeXinSendMsgTypeEnum.TEXT, message);
    String url = String.format(SEND_MESSAGE_URL, key);
    String resp = HttpRequest.post(url).body(data).execute().body();
    log.info("send work weixin message call [], param:, resp:", url, data, resp);


@Override
protected void doSendMarkdown(String title, String message) 
    String data = createPostData(WorkWeXinSendMsgTypeEnum.MARKDOWN, message);
    String url = String.format(SEND_MESSAGE_URL, key);
    String resp = HttpRequest.post(url).body(data).execute().body();
    log.info("send work weixin message call [], param:, resp:", url, data, resp);

5.6、MailWarnService
主要实现邮件告警逻辑

@Slf4j
public class MailWarnService extends BaseWarnService

private final String smtpHost;

private final String smtpPort;

private final String to;

private final String from;

private final String username;

private final String password;

private Boolean ssl = true;

private Boolean debug = false;

public MailWarnService(String smtpHost, String smtpPort, String to, String from, String username, String password) 
    this.smtpHost = smtpHost;
    this.smtpPort = smtpPort;
    this.to = to;
    this.from = from;
    this.username = username;
    this.password = password;


public void setSsl(Boolean ssl) 
    this.ssl = ssl;


public void setDebug(Boolean debug) 
    this.debug = debug;


@Override
protected void doSendText(String message) throws Exception 
    Properties props = new Properties();
    props.setProperty("mail.smtp.auth", "true");
    props.setProperty("mail.transport.protocol", "smtp");
    props.setProperty("mail.smtp.host", smtpHost);
    props.setProperty("mail.smtp.port", smtpPort);
    props.put("mail.smtp.ssl.enable", true);
    Session session = Session.getInstance(props);
    session.setDebug(false);
    MimeMessage msg = new MimeMessage(session);
    msg.setFrom(new InternetAddress(from));
    for (String toUser : to.split(",")) 
        msg.setRecipient(MimeMessage.RecipientType.TO, new InternetAddress(toUser));
    
    Map<String, String> map = JSONUtil.toBean(message, Map.class);
    msg.setSubject(map.get("subject"), "UTF-8");
    msg.setContent(map.get("content"), "text/html;charset=UTF-8");
    msg.setSentDate(new Date());
    Transport transport = session.getTransport();
    transport.connect(username, password);
    transport.sendMessage(msg, msg.getAllRecipients());
    transport.close();


@Override
protected void doSendMarkdown(String title, String message) throws Exception 
    log.warn("暂不支持发送Markdown邮件");

6、AlarmAutoConfiguration自动装配类
运用了springboot自定义的starter，再META-INF包下的配置文件spring.factories下，配置上该类

org.springframework.boot.autoconfigure.EnableAutoConfiguration=
com.seven.buttemsg.autoconfigure.AlarmAutoConfiguration
自动装配类，用于装载自定义的bean

@Slf4j
@Configuration
public class AlarmAutoConfiguration

// 邮件相关配置装载
@Configuration
@ConditionalOnProperty(prefix = MailConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(MailConfig.class)
static class MailWarnServiceMethod 

    @Bean
    @ConditionalOnMissingBean(MailWarnService.class)
    public MailWarnService mailWarnService(final MailConfig mailConfig) 
        MailWarnService mailWarnService = new MailWarnService(mailConfig.getSmtpHost(), mailConfig.getSmtpPort(), mailConfig.getTo(), mailConfig.getFrom(), mailConfig.getUsername(), mailConfig.getPassword());
        mailWarnService.setSsl(mailConfig.getSsl());
        mailWarnService.setDebug(mailConfig.getDebug());
        AlarmFactoryExecute.addAlarmLogWarnService(mailWarnService);
        return mailWarnService;
    


// 企业微信相关配置装载
@Configuration
@ConditionalOnProperty(prefix = WorkWeXinConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(WorkWeXinConfig.class)
static class WorkWechatWarnServiceMethod 

    @Bean
    @ConditionalOnMissingBean(MailWarnService.class)
    public WorkWeXinWarnService workWechatWarnService(final WorkWeXinConfig workWeXinConfig) 
        return new WorkWeXinWarnService(workWeXinConfig.getKey(), workWeXinConfig.getToUser());
    

    @Autowired
    void setDataChangedListener(WorkWeXinWarnService workWeXinWarnService) 
        AlarmFactoryExecute.addAlarmLogWarnService(workWeXinWarnService);
    


// 钉钉相关配置装载
@Configuration
@ConditionalOnProperty(prefix = DingTalkConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(DingTalkConfig.class)
static class DingTalkWarnServiceMethod 

    @Bean
    @ConditionalOnMissingBean(DingTalkWarnService.class)
    public DingTalkWarnService dingTalkWarnService(final DingTalkConfig dingtalkConfig) 
        DingTalkWarnService dingTalkWarnService = new DingTalkWarnService(dingtalkConfig.getToken(), dingtalkConfig.getSecret());
        AlarmFactoryExecute.addAlarmLogWarnService(dingTalkWarnService);
        return dingTalkWarnService;
    


// 消息模板配置装载
@Configuration
@ConditionalOnProperty(prefix = TemplateConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(TemplateConfig.class)
static class TemplateConfigServiceMethod 

    @Bean
    @ConditionalOnMissingBean
    public AlarmTemplateProvider alarmTemplateProvider(TemplateConfig templateConfig) 
        if (TemplateSource.FILE == templateConfig.getSource()) 
            return new YamlAlarmTemplateProvider(templateConfig);
         else if (TemplateSource.JDBC == templateConfig.getSource()) 
            // 数据库（如mysql）读取文件，未实现，可自行扩展
            return new JdbcAlarmTemplateProvider(templateId -> null);
         else if (TemplateSource.MEMORY == templateConfig.getSource()) 
            // 内存（如redis，本地内存）读取文件，未实现，可自行扩展
            return new MemoryAlarmTemplateProvider(templateId -> null);
        
        return new YamlAlarmTemplateProvider(templateConfig);
    



@Bean
public AlarmAspect alarmAspect(@Autowired(required = false) AlarmTemplateProvider alarmTemplateProvider) 
    return new AlarmAspect(alarmTemplateProvider);

四、总结

主要借助spring的切面技术，以及springboot的自动装配原理，实现了发送告警逻辑。对业务代码无侵入，只需要在业务代码上标记注解，就可实现可插拔的功能，比较轻量。