连接数据库 in clojure ---korma

Posted willwillie

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了连接数据库 in clojure ---korma相关的知识,希望对你有一定的参考价值。

前言:目前本文写的一塌糊涂,还在更新中。。。
先扔上两个链接:http://sqlkorma.com/https://github.com/korma/Korma/blob/master/src/korma
本文会先从数据库连接池和korma/db的角度入手讲起,最后才讲具体的用法。

首先什么是连接池?

In software engineering, a connection pool is a cache of database connections maintained so that the connections can be reused when future requests to the database are required.[citation needed] Connection pools are used to enhance the performance of executing commands on a database. Opening and maintaining a database connection for each user, especially requests made to a dynamic database-driven website application, is costly and wastes resources. In connection pooling, after a connection is created, it is placed in the pool and it is used again so that a new connection does not have to be established. If all the connections are being used, a new connection is made and is added to the pool. Connection pooling also cuts down on the amount of time a user must wait to establish a connection to the database.

在软件工程中(编程中使用到的技巧),连接池就是数据库连接(s)的cache.

于是这里有一个问题,连接数据库需要有什么消耗呢?
1.创建一个连接是不是需要花时间呢?–》(时间虽然不多,但是有的情况下还是消耗挺大的)
2.关闭一个连接也很花时间?–》(对程序的开销还是有的)
3.过多的连接太消耗内存了,所以只使用最大一定限度的连接数目,过多的用户需要等待?–》可以创建新的连接

那么连接怎么创建?
数据库连接使用TCP/http协议?

mysql短连接每次请求操作数据库都需要建立与MySQL服务器建立TCP连接,这是需要时间开销的。 TCP/IP套接字连接方式是MySQL在任何平台都提供的一种连接方式,也是网络中使用最多的一种方式。这种方式在TCP/IP连接上建立一个基于网络的连接请求,一般情况下客户端在一台服务器上,而MySQL实例在另外一台服务器上,这两台机器通过TCP/IP网络连接。

在HTTP/1.0中,默认使用的是短连接。也就是说,浏览器和服务器每进行一次HTTP操作,就建立一次连接,但任务结束就中断连接。如果客户端浏览器访问的某个html或其他类型 Web页中包含有其他的Web资源,如javascript文件、图像文件、CSS文件等;当浏览器每遇到这样一个Web资源,就会建立一个HTTP会话。
但从HTTP/1.1起,默认使用长连接,用以保持连接特性。使用长连接的HTTP协议,会在响应头有加入这行代码:Connection:keep-alive
在使用长连接的情况下,当一个网页打开完成后,客户端和服务器之间用于传输HTTP数据的TCP连接不会关闭,如果客户端再次访问这个服务器上的网页,会继续使用这一条已经建立的连接。Keep-Alive不会永久保持连接,它有一个保持时间,可以在不同的服务器软件(如Apache)中设定这个时间。实现长连接要客户端和服务端都支持长连接。
HTTP协议的长连接和短连接,实质上是TCP协议的长连接和短连接。

现在继续回到连接池的话题上面来,
korma中使用的数据库连接池是c3p0,C3P0是一个开源的JDBC连接池,它实现了数据源和JNDI绑定,支持JDBC3规范和JDBC2的标准扩展。目前使用它的开源项目有Hibernate,Spring等

Java 数据库连接,(Java Database
Connectivity,简称JDBC)是Java语言中用来规范客户端程序如何来访问数据库的应用程序接口,提供了诸如查询和更新数据库中数据的方法.

The Java Naming and Directory Interface (JNDI) is a Java API for
a directory service that allows Java software clients to discover and
look up data and objects via a name

korma还必须要引入com.mchange.v2.c3p0.ComboPooledDataSource才创建连接池。

(defn connection-pool
  "Create a connection pool for the given database spec."
  [:keys [connection-uri subprotocol subname classname
           excess-timeout idle-timeout
           initial-pool-size minimum-pool-size maximum-pool-size
           test-connection-query
           idle-connection-test-period
           test-connection-on-checkin
           test-connection-on-checkout]
    :or excess-timeout (* 30 60)
         idle-timeout (* 3 60 60)
         initial-pool-size 3
         minimum-pool-size 3
         maximum-pool-size 15
         test-connection-query nil
         idle-connection-test-period 0
         test-connection-on-checkin false
         test-connection-on-checkout false
    :as spec]
  (when-not c3p0-enabled?
    (throw (Exception. "com.mchange.v2.c3p0.ComboPooledDataSource not found in class path.")))
  :datasource (doto (resolve-new ComboPooledDataSource)
                 (.setDriverClass classname)
                 (.setJdbcUrl (or connection-uri (str "jdbc:" subprotocol ":" subname)))
                 ...

这就是c3p0的ComboPooledDataSource类创建连接池的方法,在java中也是这样实现的,比如:

   ComboPooledDataSource ds = new ComboPooledDataSource();
        String key = "jdbc." + dbname;
        ds.setDriverClass(main.config(key + ".driver"));
        ds.setJdbcUrl(main.config(key + ".url"));
        ds.setUser(main.config(key + ".username"));
        ds.setPassword(main.config(key + ".password"));
        ds.setIdleConnectionTestPeriod(Integer.parseInt(main.config(key + ".idleConnectionTestPeriod")));
        ds.setPreferredTestQuery("SELECT 1");

连接池就是这么创建的,现在回到korma db的实现上来,解决三个问题:怎么定义db?怎样创建连接?怎样获取连接?这三个问题解决后,korma db方面的问题也就都不是问题了。

怎么定义db?

这个问题主要涉及几个方面的问题,是怎样的db?mysql,pgsql,还是别的?然后host和端口号,数据库名,之后还有用户名密码,以及一些其他的配置参数。

(defn create-db
  "Create a db connection object manually instead of using defdb. This is often
   useful for creating connections dynamically, and probably should be followed
   up with:
   (default-connection my-new-conn)
   If the spec includes `:make-pool? true` makes a connection pool from the spec."
  [spec]
  (let [spec (complete-spec spec)]
    :pool    (if (:make-pool? spec)
                (delay-pool spec)
                spec)
     :options (extract-options spec)))

那么,完整的解析spec数据库文件的过程是怎样的?


(defn- complete-spec [:keys [connection-uri subprotocol] :as spec]
  (if-let [uri-or-subprotocol (or connection-uri subprotocol)]
    (let [lookup-key (first (drop-while #"jdbc" (clojure.string/split uri-or-subprotocol #":")))]
      (merge
        :classname (subprotocol->classname lookup-key) :make-pool? true
        (subprotocol->options lookup-key)
        spec))
    spec))

子协议

(defn extract-options [:keys [naming
                               delimiters
                               alias-delimiter]]
  :naming (->naming naming)
   :delimiters (->delimiters delimiters)
   :alias-delimiter (->alias-delimiter alias-delimiter))

所以这个过程其实是会创建连接池的。

怎么创建连接?

使用defdb会在create-db后调用default-connection自动创建连接:

(defonce _default (atom nil))

(defn default-connection
  "Set the database connection that Korma should use by default when no
  alternative is specified."
  [conn]
  (reset! _default conn))
(defmacro defdb
  "Define a database specification. The last evaluated defdb will be used by
  default for all queries where no database is specified by the entity."
  [db-name spec]
  `(let [spec# ~spec]
     (defonce ~db-name (create-db spec#))
     (default-connection ~db-name)))

这个连接其实就是db-name。
另外,顺便在这里,理解一下事务的隔离级别,jdbc创建连接的时候会初始化设置。

事务

我们都知道事务的几种性质,数据库为了维护这些性质,尤其是一致性和隔离性,一般使用加锁这种方式。同时数据库又是个高并发的应用,同一时间会有大量的并发访问,如果加锁过度,会极大的降低并发处理能力。所以对于加锁的处理,可以说就是数据库对于事务处理的精髓所在
那么什么是事务的加锁和解锁?
据库遵循的是两段锁协议,将事务分成两个阶段,加锁阶段和解锁阶段(所以叫两段锁)加锁阶段:在该阶段可以进行加锁操作。在对任何数据进行读操作之前要申请并获得S锁(共享锁,其它事务可以继续加共享锁,但不能加排它锁),在进行写操作之前要申请并获得X锁(排它锁,其它事务不能再获得任何锁)。加锁不成功,则事务进入等待状态,直到加锁成功才继续执行。解锁阶段:当事务释放了一个封锁以后,事务进入解锁阶段,在该阶段只能进行解锁操作不能再进行加锁操作

要有事务的隔离级别?

这是为了保证有效的读取数据,隔离级别只是为了保证读取数据而已。。。写的话不存在这方面的顾虑?。(in update—-或者这部分太长,会抽离)

现在,进入第三个问题,

怎样获取连接?

数据库在执行各种操作的时候都是要获取连接的。

(defn get-connection
  "Get a connection from the potentially delayed connection object."
  [db]
  (let [db (or (:pool db) db)]
    (if-not db
      (throw (Exception. "No valid DB connection selected."))
      (if (delay? db)
        @db
        db))))

如果有连接池就从连接池中获取一个连接,如果没有连接则爆出异常。
比如?

(defmacro with
  "在指定db的环境下执行操作. 当前连接从db的连接池中获取.

  (with erpdb ...)"
  [db & body]
  `(if (and (= kd/*current-db* ~db)
            (some? kd/*current-conn*))
     ~@body
     (kd/with-db ~db ~@body)))
do-query 
(defn do-query [:keys [db] :as query]
  (if *current-conn*
    (exec-sql query)
    (with-db (or db @_default)
      (exec-sql query))))

korma.core

另外,增删改查过程中的一些api

(defn select*
  "Create a select query with fields provided in Ent.  If fields are not provided,
  create an empty select query. Ent can either be an entity defined by defentity,
  or a string of the table name"
  [ent]
  (let [default-fields (not-empty (:fields ent))]
    (make-query ent :type :select
                     :fields (or default-fields [::*])
                     :from [(:ent this-query)]
                     :modifiers []
                     :joins []
                     :where []
                     :order []
                     :aliases #
                     :group []
                     :results :results)))

调用了make-query函数,这说明make-query函数在这里非常的通用?
make-query是一个宏

(defmacro ^:private true make-query [ent m]
  `(let [ent# ~ent]
     (if (:type ent#)
       ent#
       (let [~'this-query (empty-query ent#)]
         (merge ~'this-query ~m)))))

接受两个参数ent 和m。如果ent中存在:type键值对,则返回ent,否则将ent和m进行某些方式的合并。

那么什么是ent呢?

Ent can either be an entity defined by defentity,
or a string of the table name。

select*在这里只是返回了一个map,没有做任何事情,需要调用select* 或者exec才可以完成实际的查询(do something really later,带* 的都是想达到这样的效果)。

(defmacro select
  "Creates a select query, applies any modifying functions in the body and then
  executes it. `ent` is either a string or an entity created by defentity.
  ex: (select user
        (fields :name :email)
        (where :id 2))"
  [ent & body]
  (make-query-then-exec #'select* body ent))

select使用到了make-query-then-exec这个宏,以及#’select*,#’的解释我在《clojure知识点梳理》中有提到,

get Var object instead of the value of a symbol (var-quote)

(defn- make-query-then-exec [query-fn-var body & args]
  `(let [query# (-> (~query-fn-var ~@args)
                    ~@body)]
     (exec query#)))

这里#’select*也就是宏里面所谓的query-fn-var。
然后这里最重要的其实就是这个exec啦,其实我一般也是使用select*+exec的语法。
先看看exec的具体实现:

(defn exec
  "Execute a query map and return the results."
  [query]
  (let [query (apply-prepares query)
        query (bind-query query (eng/->sql query))
        sql (:sql-str query)
        params (:params query)]
    (cond
      (:sql query) sql
      (= *exec-mode* :sql) sql
      (= *exec-mode* :query) query
      (= *exec-mode* :dry-run) (do
                                 (println "dry run ::" sql "::" (vec params))
                                 (let [fk-key (->> query :ent :rel vals (map deref) (filter (comp #:belongs-to :rel-type)) (map :fk-key)) pk-key (-> query :ent :pk) result-keys (concat pk-key (apply concat fk-key)) results (apply-posts query [(zipmap result-keys (repeat 1))])] (first results) results))
      :else (let [results (db/do-query query)]
              (apply-transforms query (apply-posts query results))))))

那么db/do-query的实现是怎样的?

(defn- exec-sql [:keys [results sql-str params]]
  (let [keys (get-in *current-db* [:options :naming :keys])]
    (case results
      :results (jdbc/query *current-conn*
                           (apply vector sql-str params)
                           :identifiers keys)
      :keys (jdbc/db-do-prepared-return-keys *current-conn* sql-str params)
      (jdbc/db-do-prepared *current-conn* sql-str params))))

(defmacro with-db
  "Execute all queries within the body using the given db spec"
  [db & body]
  `(jdbc/with-db-connection [conn# (korma.db/get-connection ~db)]
     (binding [*current-db* ~db
               *current-conn* conn#]
       ~@body)))

(defn do-query [:keys [db] :as query]
  (if *current-conn*
    (exec-sql query)
    (with-db (or db @_default)
      (exec-sql query))))

其实调用的都是jdbc的查询函数。

这里对结果稍微做了一点处理:

:else (let [results (db/do-query query)]
              (apply-transforms query (apply-posts query results)))

apply-transforms对返回的结果做了一丢丢变换,如果是update和delete则不变,如果是查询和新增,会对结果对一些处理:

(defn- apply-transforms
  [query results]
  (if (#:delete :update (:type query))
    results
    (if-let [trans (-> query :ent :transforms seq)]
      (let [trans-fn (apply comp trans)]
        (if (sequential? results) (map trans-fn results) (trans-fn results)))
      results)))

虽然不知道:transforms在这里的作用,有什么别有用心?

下面再讲讲with,这个对于关系数据库来说还是很重要的,这个会和defentity相关。

(defn create-entity
  "Create an entity representing a table in a database."
  [table]
  ^:type ::Entity
  :table table
   :name table
   :pk '(:id)
   :db nil
   :transforms '()
   :prepares '()
   :fields []
   :rel )
(defmacro defentity
  "Define an entity representing a table in the database, applying any modifications in
  the body."
  [ent & body]
  `(let [e# (-> (create-entity ~(name ent))
                ~@body)]
     (def ~ent e#)))

比如我这么定义一个实体:

(defentity Busmodel
           (table :bas_cmdb_busmodel)
           (pk :Id)
           (has-many ResourcePlan :fk :Bussiness3Id)
           (has-many ResPlanApproveFlows :fk :Bussiness3Id)
           (has-one ViewResPlanApproveFlows :fk :PlanProductId)
           (has-many ViewResourcePlan :fk :Bussiness3Id))

这里比如table和pk函数的实现如下:

(defn table
  "Set the name of the table and an optional alias to be used for the entity.
  By default the table is the name of entity's symbol."
  [ent t & [alias]]
  (let [tname (if (or (keyword? t) (string? t))
                (name t)
                (if alias t (throw (Exception. "Generated tables must have aliases."))))
        ent (assoc ent :table tname)]
    (if alias
      (assoc ent :alias (name alias))
      ent)))

(defn pk
  "Set the primary key used for an entity. :id by default."
  ([ent & pk]
   (assoc ent :pk (map keyword pk))))

基本上都是使用assoc函数来实现的。那么has-many呢?

(defmacro has-many
  "Add a has-many relation for the given entity. It is assumed that the foreign key
   is on the sub-entity with the format table_id: user.id = email.user_id
   Can optionally pass a map with a :fk key collection or use the helper `fk` function
   to explicitly set the foreign key.
   (has-many users email)
   (has-many users email (fk :emailID))"
  ([ent sub-ent]
   `(has-many ~ent ~sub-ent nil))
  ([ent sub-ent opts]
   `(rel ~ent (var ~sub-ent) :has-many ~opts)))

这个macro给一个指定的实体增加一个一对多的关系,

看看应该怎么定义两个实体间的关系吧:

(defn rel [ent sub-ent type opts]
  (let [var-name (-> sub-ent meta :name)
        var-ns (-> sub-ent meta :ns)]
    (assoc-in ent [:rel (name var-name)]
              (delay
               (let [resolved (ns-resolve var-ns var-name) sub-ent (when resolved (deref sub-ent))] (when-not (map? sub-ent) (throw (Exception. (format "Entity used in relationship does not exist: %s" (name var-name))))) (create-relation ent sub-ent type opts))))))

至于这关系怎么创建,还是得慢慢道来~还真有点点复杂。

(defn- db-keys-and-foreign-ent [type ent sub-ent opts]
  (case type
    :many-to-many        [(many-to-many-keys ent sub-ent opts) sub-ent]
    (:has-one :has-many) [(get-db-keys ent sub-ent opts) sub-ent]
    :belongs-to          [(get-db-keys sub-ent ent opts) ent]))

(defn create-relation [ent sub-ent type opts]
  (let [[db-keys foreign-ent] (db-keys-and-foreign-ent type ent sub-ent opts)
        fk-override (when-let [fk-key (:fk opts)]
                      :fk (to-raw-key foreign-ent fk-key) :fk-key fk-key)]
    (merge :table (:table sub-ent)
            :alias (:alias sub-ent)
            :rel-type type
           db-keys
           fk-override)))

create-relation调用 (db-keys-and-foreign-ent type ent sub-ent opts)来得到一个数组[db-keys foreign-ent],为什么要得到这样一个数组呢?db-keys有什么用?foreign-ent有什么用?
db-keys-and-foreign-ent调用下面这些函数都可以得到db-keys:

(defn- to-raw-key [ent k]
  (map #(raw (eng/prefix ent %)) k))

(defn- many-to-many-keys [parent child :keys [join-table lfk rfk]]
  :lpk (to-raw-key parent (:pk parent))
   :lfk (delay (to-raw-key :table (name join-table) @lfk))
   :rfk (delay (to-raw-key :table (name join-table) @rfk))
   :rpk (to-raw-key child (:pk child))
   :join-table join-table)

(defn- get-db-keys [parent child :keys [pk fk]]
  (let [pk-key (or pk (:pk parent))
        fk-key (or fk (default-fk-name parent))]
    :pk (to-raw-key parent pk-key)
     :fk (to-raw-key child fk-key)
     :fk-key fk-key))

从这两个函数可以看出:db-keys就是一个有关主键、外键的map;回到create-relation,这也是返回一个有关各种关系的map的函数,再回到rel函数,其实也是返回ent的一些环境变量的值assoc-in ent [:rel (name var-name)]... 这样就定义好了两个ent之间的关系。
现在话题回到with(实际用的还比较多):

(defn with* [query sub-ent body-fn]
  (let [:keys [rel-type] :as rel (get-rel (:ent query) sub-ent)
        transforms (seq (:transforms sub-ent))]
    (cond
      (and (#:belongs-to :has-one rel-type)
           (not transforms))     (with-one-to-one-now rel query sub-ent body-fn)
           (#:belongs-to :has-one rel-type) (with-one-to-one-later rel query sub-ent body-fn)
           (= :has-many rel-type)     (with-one-to-many rel query sub-ent body-fn)
           (= :many-to-many rel-type) (with-many-to-many rel query sub-ent body-fn)
           :else (throw (Exception. (str "No relationship defined for table: " (:table sub-ent)))))))

(defmacro with
  "Add a related entity to the given select query. If the entity has a relationship
  type of :belongs-to or :has-one, the requested fields will be returned directly in
  the result map. If the entity is a :has-many, a second query will be executed lazily
  and a key of the entity name will be assoc'd with a vector of the results.
  (defentity email (entity-fields :email))
  (defentity user (has-many email))
  (select user
    (with email) => [:name \\"chris\\" :email [email: \\"c@c.com\\"] ...
  With can also take a body that will further refine the relation:
  (select user
     (with address
        (with state)
        (fields :address.city :state.state)
        (where :address.zip x)))"
  [query ent & body]
  `(with* ~query ~ent (fn [q#]
                        (-> q#
                            ~@body))))

with会判断各种关系,然后进行相应的处理。(这部分还在更新中,比如with是怎么处理各种关系的,后续再分析。)

以上是关于连接数据库 in clojure ---korma的主要内容,如果未能解决你的问题,请参考以下文章

使用 clojure 的 korma sqlite3 助手时,sqlite3 数据库的默认路径是啥?

Clojure线程优先具有过滤功能

Clojure rises in the workplace

如何在clojure中读取stdin(* in *)中的行

如何使用 korma 在 postgresql 上插入 clj-time 日期时间对象

连接数据库结果的下划线Clojure映射键