JVM问题排查

Posted 2021-02-05 wangzhongqiu

jetty的调用场景是：为了支持Servlet规范中的注解方式（使得不再需要在web.xml文件中进行Servlet的部署描述，简化开发流程），jetty在启动时会扫描class、lib包，将使用注解方式声明的Servlet、Listener注册到jetty容器，在扫描jar包的时候调用了inflate，分配了大量的内存，此时通过关键词搜索到Memory leak while scanning annotations，这篇文章给出了两种解决方法，一种是评论处所说，禁用缓存（说是jdk1.8的bug）：

Here’s a link to the java bugs database issue: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8156014

I’d like to be able to comment on it, but I can’t seem to find a link to allow me to do that.

The comments I’d like to add are:

the problem is still reproduceable as of jdk8u112

the problem seems to be fixed in jdk9: I tested jdk9ea+149 and couldn’t reproduce

I’ve tried some workarounds for jdk8: it seems the ServiceLoader impl uses the jarurlconnection caching, so it may be of some benefit to try to disable url caching (although not sure of the effects on performance).

Try creating an xml file (eg called “caches.xml”) with the following contents:

<Configure id="Server" class="org.eclipse.jetty.server.Server">

 <Set class="org.eclipse.jetty.util.resource.Resource" name="defaultUseCaches">false</Set> 
 <New class="java.io.File">
   <Arg>
    <SystemProperty name="java.io.tmpdir"/>
   </Arg>
   <Call name="toURI">
     <Call id="url" name="toURL">
       <Call name="openConnection">
         <Set name="defaultUseCaches">false</Set>
       </Call>
     </Call>
   </Call>
 </New>
</Configure>

尝试之后，java进程占用的物理内存由6.5G降到了5.9G，有一定的效果，但是我们估算的java进程应该占4.5G左右，还有1.4G内存被谁占用了呢？

再尝试另外一种方法：

We were investigating a Jetty application that used much more memory than it was supposed to do (near a Gb more than the max heap size plus the max metaspace size). We found out that the extra memory was Native memory, being allocated using malloc via some library (we don’t have native code in our app). Then we used jemalloc ( http://www.canonware.com/jemalloc/) to find out which class was doing this allocation and we found out that it was java.util.zip.Inflater. Via jstack we were able to find calls to the library and the main caller was Jetty, while in the process of looking for annotations. We disable annotations for the application and the memory usage went back to normal.

既然jetty调用是因为扫描jar包中的注解，我们应用中没有通过注解声明的Servlet、Listener，是不是可以不需要扫描这一步，故而在jetty.xml中将注解扫描的方式去掉：

<!-- <Item>org.eclipse.jetty.annotations.AnnotationConfiguration</Item> -->
<!-- Item>com.xxx.boot.RJRAnnotationConfiguration</Item> -->

再重启应用，发现内存使用由6.5G降到了3.9G，并且应用运行稳定后不再占用swap，

出处：

http://www.longtask.net/2018/11/15/swap-used-full/#top