机一次cpu100%的线上故障

Posted 2020-10-07 zjhgx

某天发现线上crm机器cpu100%了，估计是哪里写了个死循环，用jstack看了下

at java.util.HashMap.hash(HashMap.java:351)
    at java.util.HashMap.getEntry(HashMap.java:443)
    at java.util.HashMap.containsKey(HashMap.java:434)
    at ognl.OgnlContext.get(OgnlContext.java:491)
    at com.opensymphony.xwork2.ActionContext.get(ActionContext.java:341)
    at org.apache.struts2.ServletActionContext.getRequest(ServletActionContext.java:112)
    at com.upg.ucars.framework.base.SessionTool.getAttribute(SessionTool.java:81)
    at com.upg.ucars.framework.interceptor.AsyncInterceptor.actionIsLocked(AsyncInterceptor.java:71)
    at com.upg.ucars.framework.interceptor.AsyncInterceptor.intercept(AsyncInterceptor.java:38)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.upg.ucars.framework.interceptor.AuthorizationInterceptor.intercept(AuthorizationInterceptor.java:83)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:249)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:249)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:252)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:139)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

定位到 at com.upg.ucars.framework.interceptor.AsyncInterceptor.actionIsLocked(AsyncInterceptor.java:71)这一句

相关的代码如下：

    public String intercept(ActionInvocation arg0) throws Exception {
        try{
            if(SessionTool.getLocale()==null){
                SessionTool.setDefaultLocale();
            }
            String actionName=arg0.getProxy().getActionName().toUpperCase();
            if("LOGIN".equals(actionName)){
                unlockAction();
            }
            boolean lock=true;
            if(isUncontrolled(arg0))
                return arg0.invoke();
            else{
                while(lock){
                    lock=actionIsLocked();
                    if(!lock){
                        lockAction();
                        String target=arg0.invoke();
                        unlockAction();
                        return target;
                    }
                }
            }
        }catch(Exception e){
            unlockAction();
            throw e;
        }
        return "";
    }

 

因为这里有个while循环，估计就是在这里死循环了

    /**
     * 判断action是否已加锁
     * @return
     */
    private boolean actionIsLocked(){
        String flag=(String)SessionTool.getAttribute(SessionKeyConst.ASYNC_ACTION_LOCK);
        if(flag==null)
            return false;
        return flag.equals("true");
    }

/**
* 为action加锁
*/
private void lockAction(){
SessionTool.setAttribute(SessionKeyConst.ASYNC_ACTION_LOCK, "true");
}
/**
* 为action解锁
*/
private void unlockAction(){
SessionTool.setAttribute(SessionKeyConst.ASYNC_ACTION_LOCK, "false");
}
}

    /**
     * (获取session中存储的数据对象)
     * @param key 数据对象在session中的唯一标识
     * @return  存储在session中的数据对象
     * @date 
     */
    public static Object getAttribute(String key){
        Object ret=null;
        try{
            HttpServletRequest request=ServletActionContext.getRequest();
            if(request!=null){
                ret=request.getSession().getAttribute(key);
            }
        }catch(NullPointerException e){
            return ret;
        }
        
        return ret;
    }

 

 这里的加锁和解锁就是在session里把SessionKeyConst.ASYNC_ACTION_LOCK这个属性设成true和false，但HttpSession这个类对于同一个用户的并发操作是非线程安全的，有可能一个在解锁同时另一个在加锁，一个线程在加锁后变成他true，此时另一个线程刚好调用actionIsLocked返回true则就进入死循环了。去线上一看，果然有同一个用户