解决tomcat 执行shutdown.sh 未能正常停止服务,释放资源 出现如 * create a memory leak

Posted 2020-10-10

Made with Remarkable!

解决tomcat 执行shutdown.sh 未能正常停止服务,释放资源 出现如 * create a memory leak

近日,同事反馈说在搭建jenkins 部署,在打包完成,执行自动部署时,执行server tomcat 的shutdown.sh 后,tomcat 进程未关闭,资源未得到释放,日志输出如下:

警告: The web application [XXXX] appears to have started a thread named [commons-pool-EvictionTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
十月 13, 2017 10:10:19 上午 org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
警告: The web application [XXXX appears to have started a thread named [Thread-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492) java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680) com.erp.cloudfi.report.util.LongTimeWorker.startWork(LongTimeWorker.java:38) com.erp.cloudfi.report.util.LongTimeWorker$ 1.run(LongTimeWorker.java:27)
java.lang.Thread.run(Thread.java:748)

经过日志反馈,查询到 有两种情况会造成线程存留.我们需要在spring 应用 shutdown时,destroy 相关资源并退出相关线程

1.LongTimeWorker(业务线程)引起的

代码如下:

public class LongTimeWorker {
    /**
     * 任务队列
     */
    BlockingQueue<Worker> works;
    public LongTimeWorker() {
        works = new LinkedBlockingDeque<Worker>();
        Thread thread = new Thread(new Runnable() {
            @Override
            public void run() {
                startWork();
            }
        });
        thread.setDaemon(true);
        thread.start();
    }
    public void startWork() {
        try {
            while(true) {
                Worker work = works.take();
                Worksheet sheet = work.getWorksheet();              
                work.run();
                if(sheet == null){
                    Thread.sleep(100);
                }else{
                    String id = sheet.getId();
                    existsKey.remove(id);
                }
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    /**
     * 
     * @param work
     */
    public void setWork(Worker work) {
        try {
            works.put(work);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

可以看到,在创建对象的时候,以demon 方式启动了一个 死循环的 程序,不包含退出条件.

修改方式如下,添加 退出标志,放弃创建对象的时候启动线程,增加启动线程方法,与退出线程方法.
修改后代码如下:

public class LongTimeWorker {
    /**
     * 任务队列
     */
    BlockingQueue<Worker> works;
    private boolean workstatus=false;
    private Thread workthread = null;
    public LongTimeWorker() {
        works = new LinkedBlockingDeque<Worker>();
        workthread = new Thread(new Runnable() {
            @Override
            public void run() {
                startWork();
            }
        });
        workstatus=false;
        workthread.setDaemon(true);
    }
    public void startWork() {
        workstatus = true;
        workthread.start();
    }
    public void stopWork() {
        workstatus=false;
    }
    public void work() {
        try {
            while(workstatus) {
                Worker work = works.take();
                Worksheet sheet = work.getWorksheet();              
                work.run();
                if(sheet == null){
                    Thread.sleep(100);
                }
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    /**
     * @param work
     */
    public void setWork(Worker work) {
        try {
            works.put(work);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

2.commons-pool-EvictionTimer 未正常释放资源引起的

实现类为 org.apache.commons.pool.impl.EvictionTimer 所属于 commons-pool 包下,class 描述如下

Provides a shared idle object eviction timer for all pools. This class wraps the standard java.util.Timer and keeps track of how many pools are using it. If no pools are using the timer, it is canceled. This prevents a thread being left running which, in application server environments, can lead to memory leads and/or prevent applications from shutting down or reloading cleanly.

This class has package scope to prevent its inclusion in the pool public API. The class declaration below should *not* be changed to public.

提供了一个基于timer 机制的共享空闲 对象的通用池.

项目中涉及到 对象共享池的相关有 数据库连接池,JedisPool,因为 未使用spring 提供的 redis 模块 ,而是基于jedis 进行的封装.所以针对JedisPool 连接池 .(数据库连接池比较常用,以前未暴露此问题).经过检查工程代码,发现应用中 在应用启动时,声明了一个JedisPool,来进行业务缓存处理,但在应用退出时并未对jedispool 有任何处理.
参考 https://github.com/xetorthio/jedis/issues/936

Pool needs to be closed when it is no longer used - Failed to stop thread named [commons-pool-EvictionTimer]
pool在不被使用的时候需要释放

guys from commons-pool you need to create a ServletContextListener and implement the contextDestroyed method. In thaUImethod you should get the reference to your JedisPool and call the close method.
需要在ServletContextListener 实现 contextDestroyed 方法 , 执行JedisPool 对象的destroy 方法,对资源进行释放.

---

在应用启动与关闭时,初始化与销毁相关资源,步骤如下:

a.实现ServletContextListener 接口,分别在contextInitialized 与 contextDestroyed 中实现自己的资源初始化与销毁逻辑(在 destroy 方法中 处理掉问题中未关闭的线程,与未关闭的redis pool )

    public class CloudfiApplicationContextListener implements ServletContextListener {
        private ServletContext servletContext;
        private LongTimeWorker longTimeWorker;
        private MybatisRedisCache redis;

        public void setRedis(MybatisRedisCache redis) {
            this.redis = redis;
        }

        public void setLongTimeWorker(LongTimeWorker longTimeWorker) {
            this.longTimeWorker = longTimeWorker;
        }

        @Override
        public void contextInitialized(ServletContextEvent sce) {
            servletContext = sce.getServletContext();
            SpringContextUtil.setApplicationContext(WebApplicationContextUtils.getWebApplicationContext(servletContext));

            longTimeWorker = SpringContextUtil.getBean( LongTimeWorker.class);
            redis = SpringContextUtil.getBean(MybatisRedisCache.class);
            longTimeWorker.startWork();
        }

        @Override
        public void contextDestroyed(ServletContextEvent sce) {
            longTimeWorker.stopWork();
            redis.getJedisPool().close();
        }
    }

b.在web.xml 中配置 a 中定义的listener

<servlet>
    <servlet-name>springmvc</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
    <init-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:springmvc.xml</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
</servlet>
<listener>  
    <listener-class>org.springframework.web.context.ContextLoaderListener  
    </listener-class>  
</listener>
<listener>
    <listener-class>
        javacommon.init.CloudfiApplicationContextListener
    </listener-class>
</listener>

注意,如果需要在实现的listener 中,使用spring 中的bean ,由于listener 与 servlet 加载顺序的限制,此处只能读取 ContextLoaderListener
扫描到spring 的bean,无法获取到 DispatcherServlet 扫描到spring 的bean (若需要获取 ContextLoaderListener 的bean ,注意自定义listener 的配置需要在 ContextLoaderListener 配置之后)

至此,引起tomcat 执行 shutdown.sh 无法正常关闭的问题解决.

如果上述中出现错误,欢迎指正,若有问题请联系[email protected]
crazy~smn 星期五, 13. 十月 2017 02:55下午