Java GenericObjectPool 对象池化技术--SpringBoot sftp 连接池工具类

Posted VipSoft

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Java GenericObjectPool 对象池化技术--SpringBoot sftp 连接池工具类相关的知识,希望对你有一定的参考价值。

SpringBoot sftp 连接池工具类 使用`GenericObjectPool`只需要创建一个对象工厂类,继承`BasePooledObjectFactory`并重写它的`create()`和`destroyObject()`。Apache Commons Pool是一个对象池的框架,他提供了一整套用于实现对象池化的API。它提供了三种对象池:GenericKeyedObjectPool,SoftReferenceObjectPool和GenericObjectPool,其中GenericObjectPool是我们最常用的对象池,内部实现也最复杂。

Java BasePooledObjectFactory 对象池化技术

通常一个对象创建、销毁非常耗时的时候,我们不会频繁的创建和销毁它,而是考虑复用。复用对象的一种做法就是对象池,将创建好的对象放入池中维护起来,下次再用的时候直接拿池中已经创建好的对象继续用,这就是池化的思想。

Apache Commons Pool是一个对象池的框架,他提供了一整套用于实现对象池化的API。它提供了三种对象池:GenericKeyedObjectPool,SoftReferenceObjectPool和GenericObjectPool,其中GenericObjectPool是我们最常用的对象池,内部实现也最复杂。

GenericObjectPool

GenericObjectPool 是一个通用对象池框架,我们可以借助它实现一个健壮的对象池,UML图如下所示:

GenericObjectPool 实现了ObjectPool接口,而ObjectPool就是对象池的核心接口,它定义了一个对象池应该实现的行为。

public interface ObjectPool<T> extends Closeable 
    /**
     * 从池中借走到一个对象
     */
    T borrowObject() throws Exception, NoSuchElementException, IllegalStateException;
    /**
     * 把对象归还给对象池
     */
    void returnObject(T var1) throws Exception;
    /**
     * 验证对象的有效性
     */
    void invalidateObject(T var1) throws Exception;

    /**
     * 往池中添加一个对象
     */
    void addObject() throws Exception, IllegalStateException, UnsupportedOperationException;
    /**
     * 返回对象池中有多少对象是空闲的,也就是能够被借走的对象的数量。
     */
    int getNumIdle();
    /**
     * 返回对象池中有对象对象是活跃的,也就是已经被借走的,在使用中的对象的数量。
     */
    int getNumActive();
    /**
     * 清理对象池。注意是清理不是清空,该方法要求的是,清理所有空闲对象,释放相关资源。
     */
    void clear() throws Exception, UnsupportedOperationException;
    /**
     * 关闭对象池。这个方法可以达到清空的效果,清理所有对象以及相关资源。
     */
    void close();

BasePooledObjectFactory

Java BasePooledObjectFactory 对象池化技术

使用GenericObjectPool只需要创建一个对象工厂类,继承BasePooledObjectFactory并重写它的create()destroyObject()
如下文中的:SftpPool.java

public interface PooledObjectFactory<T> 
    /**
     * 创建一个可由池提供服务的实例,并将其封装在由池管理的PooledObject中。
     */
    PooledObject<T> makeObject() throws Exception;

    /**
     *  销毁池不再需要的实例
     */
    void destroyObject(PooledObject<T> var1) throws Exception;

    /**
     * 确保实例可以安全地由池返回
     */
    boolean validateObject(PooledObject<T> var1);

    /**
     * 重新初始化池返回的实例
     */
    void activateObject(PooledObject<T> var1) throws Exception;

    /**
     * 取消初始化要返回到空闲对象池的实例
     */
    void passivateObject(PooledObject<T> var1) throws Exception;

配置类GenericObjectPoolConfig

GenericObjectPoolConfig是封装GenericObject池配置的简单“结构”,此类不是线程安全的;它仅用于提供创建池时使用的属性。大多数情况,可以使用GenericObjectPoolConfig提供的默认参数就可以满足日常的需求。

工作原理流程

  1. 构造方法
    当我们执行构造方法时,主要工作就是创建了一个存储对象的LinkedList类型容器,也就是概念意义上的“池”
  2. 从对象池中获取对象
    获取池中的对象是通过borrowObject()命令,源码比较复杂,简单而言就是去LinkedList中获取一个对象,如果不存在的话,要调用构造方法中第一个参数Factory工厂类的makeObject()方法去创建一个对象再获取,获取到对象后要调用validateObject方法判断该对象是否是可用的,如果是可用的才拿去使用。LinkedList容器减一
  3. 归还对象到线程池
    简单而言就是先调用validateObject方法判断该对象是否是可用的,如果可用则归还到池中,LinkedList容器加一,如果是不可以的则调用destroyObject方法进行销毁

上面三步就是最简单的流程,由于取和还的流程步骤都在borrowObject和returnObject方法中固定的,所以我们只要重写Factory工厂类的makeObject()和validateObject以及destroyObject方法即可实现最简单的池的管理控制,通过构造方法传入该Factory工厂类对象则可以创建最简单的对象池管理类。这算是比较好的解耦设计模式,借和还的流程如下图所示:

使用Demo

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-pool2</artifactId>
    <version>2.7.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/com.jcraft/jsch -->
<dependency>
    <groupId>com.jcraft</groupId>
    <artifactId>jsch</artifactId>
    <version>0.1.55</version>
</dependency>
点击查看代码
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>vipsoft-parent</artifactId>
        <groupId>com.vipsoft.boot</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>vipsoft-sftp</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-pool2</artifactId>
            <version>2.7.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.jcraft/jsch -->
        <dependency>
            <groupId>com.jcraft</groupId>
            <artifactId>jsch</artifactId>
            <version>0.1.55</version>
        </dependency>

        <dependency>
            <groupId>org.eclipse.paho</groupId>
            <artifactId>org.eclipse.paho.client.mqttv3</artifactId>
            <version>1.2.5</version>
        </dependency>


        <dependency>
            <groupId>cn.hutool</groupId>
            <artifactId>hutool-all</artifactId>
            <version>5.3.6</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

application.yaml

server:
  port: 8088
  application:
    name: sftp Demo


sftp:
  host: 172.16.3.88 # 服务器ip
  port: 22 # ssh端口
  username: root # 用户名
  password: root # 密码
  # 连接池参数
  pool:
    max-total: 10
    max-idle: 10
    min-idle: 5

SftpPoolException.java

package com.vipsoft.sftp.exception;


/**
 * sftp连接池异常
 */
public class SftpPoolException extends RuntimeException 

    private static final long serialVersionUID = 1L;

    /**
     * Constructs a new runtime exception with @code null as its
     * detail message.  The cause is not initialized, and may subsequently be
     * initialized by a call to @link #initCause.
     */
    public SftpPoolException() 
    

    /**
     * Constructs a new runtime exception with the specified detail message.
     * The cause is not initialized, and may subsequently be initialized by a
     * call to @link #initCause.
     *
     * @param message the detail message. The detail message is saved for
     *                later retrieval by the @link #getMessage() method.
     */
    public SftpPoolException(String message) 
        super(message);
    

    /**
     * Constructs a new runtime exception with the specified detail message and
     * cause.  <p>Note that the detail message associated with
     * @code cause is <i>not</i> automatically incorporated in
     * this runtime exception\'s detail message.
     *
     * @param message the detail message (which is saved for later retrieval
     *                by the @link #getMessage() method).
     * @param cause   the cause (which is saved for later retrieval by the
     *                @link #getCause() method).  (A <tt>null</tt> value is
     *                permitted, and indicates that the cause is nonexistent or
     *                unknown.)
     * @since 1.4
     */
    public SftpPoolException(String message, Throwable cause) 
        super(message, cause);
    

    /**
     * Constructs a new runtime exception with the specified cause and a
     * detail message of <tt>(cause==null ? null : cause.toString())</tt>
     * (which typically contains the class and detail message of
     * <tt>cause</tt>).  This constructor is useful for runtime exceptions
     * that are little more than wrappers for other throwables.
     *
     * @param cause the cause (which is saved for later retrieval by the
     *              @link #getCause() method).  (A <tt>null</tt> value is
     *              permitted, and indicates that the cause is nonexistent or
     *              unknown.)
     * @since 1.4
     */
    public SftpPoolException(Throwable cause) 
        super(cause);
    

    /**
     * Constructs a new runtime exception with the specified detail
     * message, cause, suppression enabled or disabled, and writable
     * stack trace enabled or disabled.
     *
     * @param message            the detail message.
     * @param cause              the cause.  (A @code null value is permitted,
     *                           and indicates that the cause is nonexistent or unknown.)
     * @param enableSuppression  whether or not suppression is enabled
     *                           or disabled
     * @param writableStackTrace whether or not the stack trace should
     *                           be writable
     * @since 1.7
     */
    public SftpPoolException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) 
        super(message, cause, enableSuppression, writableStackTrace);
    



config

SftpConfig.java

package com.vipsoft.sftp.config;

import com.vipsoft.sftp.pool.SftpFactory;
import com.vipsoft.sftp.pool.SftpPool;
import com.vipsoft.sftp.utils.SftpUtil;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableConfigurationProperties(SftpProperties.class)
public class SftpConfig 
    // 工厂
    @Bean
    public SftpFactory sftpFactory(SftpProperties properties) 
        return new SftpFactory(properties);
    

    // 连接池
    @Bean
    public SftpPool sftpPool(SftpFactory sftpFactory) 
        return new SftpPool(sftpFactory);
    

    // 辅助类
    @Bean
    public SftpUtil sftpUtil(SftpPool sftpPool) 
        return new SftpUtil(sftpPool);
    


SftpProperties.java

package com.vipsoft.sftp.config;

import com.jcraft.jsch.ChannelSftp;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "sftp")
public class SftpProperties 

    private String host;
    private int port = 22;
    private String username = "root";
    private String password = "root";
    private Pool pool = new Pool();

    public String getHost() 
        return host;
    

    public void setHost(String host) 
        this.host = host;
    

    public int getPort() 
        return port;
    

    public void setPort(int port) 
        this.port = port;
    

    public String getUsername() 
        return username;
    

    public void setUsername(String username) 
        this.username = username;
    

    public String getPassword() 
        return password;
    

    public void setPassword(String password) 
        this.password = password;
    

    public Pool getPool() 
        return pool;
    

    public void setPool(Pool pool) 
        this.pool = pool;
    

    public static class Pool extends GenericObjectPoolConfig<ChannelSftp> 

        private int maxTotal = DEFAULT_MAX_TOTAL;
        private int maxIdle = DEFAULT_MAX_IDLE;
        private int minIdle = DEFAULT_MIN_IDLE;

        public Pool() 
            super();
        
        @Override
        public int getMaxTotal() 
            return maxTotal;
        
        @Override
        public void setMaxTotal(int maxTotal) 
            this.maxTotal = maxTotal;
        
        @Override
        public int getMaxIdle() 
            return maxIdle;
        
        @Override
        public void setMaxIdle(int maxIdle) 
            this.maxIdle = maxIdle;
        
        @Override
        public int getMinIdle() 
            return minIdle;
        
        @Override
        public void setMinIdle(int minIdle) 
            this.minIdle = minIdle;
        
    

Pool

SftpFactory.java

package com.vipsoft.sftp.pool;

import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
import com.vipsoft.sftp.config.SftpProperties;
import com.vipsoft.sftp.exception.SftpPoolException;
import org.apache.commons.pool2.BasePooledObjectFactory;
import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.impl.DefaultPooledObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Properties;

public class SftpFactory extends BasePooledObjectFactory<ChannelSftp> 

    private  final Logger logger = LoggerFactory.getLogger(this.getClass());

    private SftpProperties properties;

    public SftpProperties getProperties() 
        return properties;
    

    public void setProperties(SftpProperties properties) 
        this.properties = properties;
    

    public SftpFactory(SftpProperties properties) 
        this.properties = properties;
    

    @Override
    public ChannelSftp create() 
        try 
            JSch jsch = new JSch();
            Session sshSession = jsch.getSession(properties.getUsername(), properties.getHost(), properties.getPort());
            sshSession.setPassword(properties.getPassword());
            Properties sshConfig = new Properties();
            sshConfig.put("StrictHostKeyChecking", "no");
            sshSession.setConfig(sshConfig);
            sshSession.connect();
            ChannelSftp channel = (ChannelSftp) sshSession.openChannel("sftp");
            channel.connect();
            return channel;
         catch (JSchException e) 
            throw new SftpPoolException("连接sfpt失败", e);
        
    

    @Override
    public PooledObject<ChannelSftp> wrap(ChannelSftp channelSftp) 
        return new DefaultPooledObject<>(channelSftp);
    

    // 销毁对象
    @Override
    public void destroyObject(PooledObject<ChannelSftp> p) 
        ChannelSftp channelSftp = p.getObject();
        channelSftp.disconnect();
    


SftpPool.java

package com.vipsoft.sftp.pool;

import com.jcraft.jsch.ChannelSftp;
import org.apache.commons.pool2.impl.GenericObjectPool;

public class SftpPool<T> extends GenericObjectPool<ChannelSftp> 

    public SftpPool(SftpFactory factory) 
        super(factory,factory.getProperties().getPool());
    

    /**
     * 获取一个sftp连接对象
     * @return sftp连接对象
     */
    @Override
    public ChannelSftp borrowObject() throws Exception 
        return super.borrowObject();
    

    /**
     * 归还一个sftp连接对象
     * @param channelSftp sftp连接对象
     */
    @Override
    public void returnObject(ChannelSftp channelSftp) 
        if (channelSftp!=null) 
            super.returnObject(channelSftp);
        
    


Utils

ByteUtil.java

package com.vipsoft.sftp.utils;

import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.SftpException;
import com.vipsoft.sftp.exception.SftpPoolException;
import com.vipsoft.sftp.pool.SftpPool;

import java.io.InputStream;

public class SftpUtil 

    private SftpPool pool;

    public SftpUtil(SftpPool pool) 
        this.pool = pool;
    

    /**
     * 下载文件
     *
     * @param dir  远程目录
     * @param name 远程文件名
     * @return 文件字节数组
     */
    public byte[] download(String dir, String name) 
        ChannelSftp sftp = null;
        try 
            sftp = pool.borrowObject();
            sftp.cd(dir);
            InputStream in = sftp.get(name);
            return ByteUtil.inputStreamToByteArray(in);
         catch (Exception e) 
            throw new SftpPoolException("sftp下载文件出错", e);
         finally 
            pool.returnObject(sftp);
        
    

    /**
     * 上传文件
     *
     * @param dir  远程目录
     * @param name 远程文件名
     * @param in   输入流
     */
    public void upload(String dir, String name, InputStream in) 
        ChannelSftp sftp = null;
        try 
            sftp = pool.borrowObject();
            mkdirs(sftp, dir);
            sftp.cd(dir);
            sftp.put(in, name);
         catch (Exception e) 
            throw new SftpPoolException("sftp上传文件出错", e);
         finally 
            pool.returnObject(sftp);
        
    

    /**
     * 删除文件
     *
     * @param dir  远程目录
     * @param name 远程文件名
     */
    public void delete(String dir, String name) 
        ChannelSftp sftp = null;
        try 
            sftp = pool.borrowObject();
            sftp.cd(dir);
            sftp.rm(name);
         catch (Exception e) 
            throw new SftpPoolException("sftp删除文件出错", e);
         finally 
            pool.returnObject(sftp);
        
    

    /**
     * 递归创建多级目录
     *
     * @param dir 多级目录
     */
    private void mkdirs(ChannelSftp sftp, String dir) 
        String[] folders = dir.split("/");
        try 
            sftp.cd("/");
            for (String folder : folders) 
                if (folder.length() > 0) 
                    try 
                        sftp.cd(folder);
                     catch (Exception e) 
                        sftp.mkdir(folder);
                        sftp.cd(folder);
                    
                
            
         catch (SftpException e) 
            throw new SftpPoolException("sftp创建目录出错", e);
        
    


Test

SftpTest.java

package com.vipsoft.sftp;

import com.vipsoft.sftp.utils.SftpUtil;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@SpringBootTest
public class SftpTest 

    @Autowired
    private SftpUtil sftpUtil;

    @Test
    void downloadTest() 
        byte[] dockerfiles = sftpUtil.download("/opt/demo/", "Dockerfile");
        System.out.println("FileSize =>" + dockerfiles.length);
    



GenericObjectPool

我想我在这里找到了一个很好的答案:log4j2-jdbc-manager-cannot-connect-to-the-database

我试图在我的项目中实现代码,除了这个小错误之外,一切正常:

GenericObjectPool<PoolableConnection>

类型GenericObjectPool不接受参数。

我不知道如何做一个解决方法,或者改变什么。

谢谢你的帮助!

答案

试试这个

https://git-wip-us.apache.org/repos/asf?p=commons-dbcp.git;a=blob_plain;f=doc/PoolingDataSourceExample.java;hb=HEAD

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-dbcp2</artifactId>
    <version>2.1</version>
</dependency>
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-pool2</artifactId>
    <version>2.3</version>
</dependency>

然后在你的代码中添加:

import org.apache.commons.pool2.impl.GenericObjectPool;

以上是关于Java GenericObjectPool 对象池化技术--SpringBoot sftp 连接池工具类的主要内容,如果未能解决你的问题,请参考以下文章

apache的GenericObjectPool对象池使用经历!

GenericObjectPool

TodoApache-Commons-Pool及对象池学习

commons-pool2中GenericObjectPoolConfig的maxTotalmaxIdleminIdle属性理解

commons-pool2中GenericObjectPoolConfig的maxTotalmaxIdleminIdle属性理解

java 项目开发,出现如下错误dependency. java.lang.NoSuchMethodError:,请大神帮帮忙