关于数据执行效率

Posted 2020-09-02 临冰听雪丶

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了关于数据执行效率相关的知识，希望对你有一定的参考价值。

今天开发碰到了一个问题，从数据库取出一个表中的数据，大约500条左右。然后通过map转换为另外一个Model，总共用时1分钟，有时候会更多。所以，我开始怀疑automapper的执行效率。

于是，我决定写个demo看看

首先我想要先创建一个entity，和一个model

我的entity是这样的

public class TestTable
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public DateTime brithday { get; set; }
        public long Age { get; set; }
        public bool Gender { get; set; }
        public short IDTypeCode { get; set; }

        public int Id2 { get; set; }
        public string Name2 { get; set; }
        public DateTime brithday2 { get; set; }
        public long Age2 { get; set; }
        public bool Gender2 { get; set; }
        public short IDTypeCode2 { get; set; }

        public int Id3 { get; set; }
        public string Name3 { get; set; }
        public DateTime brithday3 { get; set; }
        public long Age3 { get; set; }
        public bool Gender3 { get; set; }
        public short IDTypeCode3 { get; set; }

        public int Id4 { get; set; }
        public string Name4 { get; set; }
        public DateTime brithday4 { get; set; }
        public long Age4 { get; set; }
        public bool Gender4 { get; set; }
        public short IDTypeCode4 { get; set; }

        public int Id5 { get; set; }
        public string Name5 { get; set; }
        public DateTime brithday5 { get; set; }
        public long Age5 { get; set; }
        public bool Gender5 { get; set; }
        public short IDTypeCode5 { get; set; }
}

然后我的Model和entity一样

public class TestModel2
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public DateTime brithday { get; set; }
        public long Age { get; set; }
        public bool Gender { get; set; }
        public short IDTypeCode { get; set; }

        public int Id2 { get; set; }
        public string Name2 { get; set; }
        public DateTime brithday2 { get; set; }
        public long Age2 { get; set; }
        public bool Gender2 { get; set; }
        public short IDTypeCode2 { get; set; }

        public int Id3 { get; set; }
        public string Name3 { get; set; }
        public DateTime brithday3 { get; set; }
        public long Age3 { get; set; }
        public bool Gender3 { get; set; }
        public short IDTypeCode3 { get; set; }

        public int Id4 { get; set; }
        public string Name4 { get; set; }
        public DateTime brithday4 { get; set; }
        public long Age4 { get; set; }
        public bool Gender4 { get; set; }
        public short IDTypeCode4 { get; set; }

        public int Id5 { get; set; }
        public string Name5 { get; set; }
        public DateTime brithday5 { get; set; }
        public long Age5 { get; set; }
        public bool Gender5 { get; set; }
        public short IDTypeCode5 { get; set; }
}

接着我需要一个可以给我生成数据的class，我写了它

public static class BuildTestData
    {
        private static T Data<T>(int? i = null)
        {
            T data = (T)Activator.CreateInstance(typeof(T));
            if (i == null)
            {
                return data;
            }
            var propertys = data.GetType().GetProperties();
            foreach (var type in propertys)
            {
                if (!type.CanWrite)
                {
                    continue;
                }
                if (type.PropertyType == typeof(int) || type.PropertyType == typeof(int?))
                {
                    type.SetValue(data, i);
                }
                else if (type.PropertyType == typeof(long) || type.PropertyType == typeof(long?))
                {
                    type.SetValue(data, Convert.ToInt64(i));
                }
                else if (type.PropertyType == typeof(bool) || type.PropertyType == typeof(bool?))
                {
                    type.SetValue(data, true);
                }
                else if (type.PropertyType == typeof(DateTime) || type.PropertyType == typeof(DateTime?))
                {
                    type.SetValue(data, DateTime.Now);
                }
                else if (type.PropertyType == typeof(string))
                {
                    string value = "test" + type.Name + i;
                    type.SetValue(data, value);
                }
            }
            return data;
        }

        public static T CreateData<T>(this T obj)
        {
            return Data<T>(1);
        }


        public static List<T> CreateData<T>(this List<T> obj, int size, bool isAddNull = false)
        {
            obj = new List<T>();
            for (int i = 1; i <= size; i++)
            {
                var data = Data<T>(i);
                obj.Add(data);
            }
            if (isAddNull)
            {
                obj.Add(Data<T>());
            }
            return obj;
        }

}

然后分别使用for、foreach和automapper转换为TestModel2

for：

public List<TestModel2> ValidationFor(List<TestTable> list)
        {
            List<TestModel2> list2 = new List<TestModel2>();
            for (int i = 0; i < list.Count; i++)
            {
                list2.Add(new TestModel2()
                {
                    Id = list[i].Id,
                    Name = list[i].Name,
                    brithday = list[i].brithday,
                    Age = list[i].Age,
                    Gender = list[i].Gender,
                    IDTypeCode = list[i].IDTypeCode,
                    Id2 = list[i].Id,
                    Name2 = list[i].Name,
                    brithday2 = list[i].brithday,
                    Age2 = list[i].Age,
                    Gender2 = list[i].Gender,
                    IDTypeCode2 = list[i].IDTypeCode,
                    Id3 = list[i].Id,
                    Name3 = list[i].Name,
                    brithday3 = list[i].brithday,
                    Age3 = list[i].Age,
                    Gender3 = list[i].Gender,
                    IDTypeCode3 = list[i].IDTypeCode,
                    Id4 = list[i].Id,
                    Name4 = list[i].Name,
                    brithday4 = list[i].brithday,
                    Age4 = list[i].Age,
                    Gender4 = list[i].Gender,
                    IDTypeCode4 = list[i].IDTypeCode,
                    Id5 = list[i].Id,
                    Name5 = list[i].Name,
                    brithday5 = list[i].brithday,
                    Age5 = list[i].Age,
                    Gender5 = list[i].Gender,
                    IDTypeCode5 = list[i].IDTypeCode,
                });
            }
            return list2;
        }

Foreach:

public List<TestModel2> ValidationForeach(List<TestTable> list)
        {
            List<TestModel2> list2 = new List<TestModel2>();
            foreach (var item in list)
            {
                list2.Add(new TestModel2()
                {
                    Id = item.Id,
                    Name = item.Name,
                    brithday = item.brithday,
                    Age = item.Age,
                    Gender = item.Gender,
                    IDTypeCode = item.IDTypeCode,
                    Id2 = item.Id,
                    Name2 = item.Name,
                    brithday2 = item.brithday,
                    Age2 = item.Age,
                    Gender2 = item.Gender,
                    IDTypeCode2 = item.IDTypeCode,
                    Id3 = item.Id,
                    Name3 = item.Name,
                    brithday3 = item.brithday,
                    Age3 = item.Age,
                    Gender3 = item.Gender,
                    IDTypeCode3 = item.IDTypeCode,
                    Id4 = item.Id,
                    Name4 = item.Name,
                    brithday4 = item.brithday,
                    Age4 = item.Age,
                    Gender4 = item.Gender,
                    IDTypeCode4 = item.IDTypeCode,
                    Id5 = item.Id,
                    Name5 = item.Name,
                    brithday5 = item.brithday,
                    Age5 = item.Age,
                    Gender5 = item.Gender,
                    IDTypeCode5 = item.IDTypeCode,
                });
            }
            return list2;
        }

Automapper(不得不说map可以省很多事)

public List<TestModel2> ValidationAutoMapper(List<TestTable> list)
        {
            //this.CreateMap<List<TestModel>, List<TestModel2>>();

            Mapper.Initialize(cfg => cfg.CreateMap<List<TestTable>, List<TestModel2>>());
            //or
            var config = new MapperConfiguration(cfg => cfg.CreateMap<List<TestTable>, List<TestModel2>>());
            var result = Mapper.Map<List<TestTable>, List<TestModel2>>(list);
            return result;
        }

然后我用我生成数据的类，生成了十万条数据

public static List<TestTable> list = new List<TestTable>();
list = list.CreateData(100000);

然后使用Stopwatch 测试执行时间

Program pa = new Program();
            Stopwatch stopTime = new Stopwatch();
            //pa.CreateEfData();
            Log log = new Log();
            list = list.CreateData(100000);
            //list2 = list2.CreateData(100000);
            stopTime.Start();
            pa.ValidationFor(list);
            stopTime.Stop();
            TimeSpan ts = TimeSpan.FromMilliseconds(stopTime.ElapsedMilliseconds);
            string str = string.Format("{0:D2}:{1:D2}:{2:D2}.{3:D3}", ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds);
            log.log("for：" + str);
            Console.WriteLine("for：" + str);
            stopTime.Reset();

            stopTime.Start();
            pa.ValidationForeach(list);
            stopTime.Stop();
            TimeSpan ts2 = TimeSpan.FromMilliseconds(stopTime.ElapsedMilliseconds);
            string str2 = string.Format("{0:D2}:{1:D2}:{2:D2}.{3:D3}", ts2.Hours, ts2.Minutes, ts2.Seconds, ts2.Milliseconds);
            log.log("foreach：" + str2);
            Console.WriteLine("foreach：" + str2);
            stopTime.Reset();

            stopTime.Start();
            pa.ValidationAutoMapper(list);
            stopTime.Stop();
            TimeSpan ts3 = TimeSpan.FromMilliseconds(stopTime.ElapsedMilliseconds);
            string str3 = string.Format("{0:D2}:{1:D2}:{2:D2}.{3:D3}", ts3.Hours, ts3.Minutes, ts3.Seconds, ts3.Milliseconds);
            log.log("automapper：" + str3);
            Console.WriteLine("automapper：" + str3);
            stopTime.Reset();

            Console.ReadLine();

运行结果：

可以得出这样一个结论

Foreach是最快的一个，其次是for，再接着就是automapper，automapper是for和foreach的整整两倍。

可是，就算是这样，我500条数据map为什么会用了1分钟呢？难道是因为数据库？

我在数据库中又接着创建了这样一张表

--create table TestTable
--(
--    Id int identity(1,1) primary key,
--    Name nvarchar(100),
--    brithday datetime,
--    Age bigint,
--    Gender bit,
--    IDTypeCode smallint,

--    Id2 int ,
--    Name2 nvarchar(100),
--    brithday2 datetime,
--    Age2 bigint,
--    Gender2 bit,
--    IDTypeCode2 smallint,

--    Id3 int  ,
--    Name3 nvarchar(100),
--    brithday3 datetime,
--    Age3 bigint,
--    Gender3 bit,
--    IDTypeCode3 smallint,

--    Id4 int,
--    Name4 nvarchar(100),
--    brithday4 datetime,
--    Age4 bigint,
--    Gender4 bit,
--    IDTypeCode4 smallint,

--    Id5 int,
--    Name5 nvarchar(100),
--    brithday5 datetime,
--    Age5 bigint,
--    Gender5 bit,
--    IDTypeCode5 smallint,
--)

添加了6万条数据

通过sql取得数据

public void CreateSqlData()
        {
            using (SqlConnection con = new SqlConnection("*****"))
            {
                con.Open();
                SqlCommand com = new SqlCommand();
                com.Connection = con;
                com.CommandText = "select Id, Name, brithday, Age, Gender, IDTypeCode, Id2, Name2, brithday2, Age2, Gender2, IDTypeCode2, Id3, Name3, brithday3, Age3, Gender3, IDTypeCode3, Id4, Name4, brithday4, Age4, Gender4, IDTypeCode4, Id5, Name5, brithday5, Age5, Gender5, IDTypeCode5 from TestTable";
                com.CommandType = System.Data.CommandType.Text;
                using (SqlDataReader read = com.ExecuteReader())
                {
                    while (read.Read())
                    {
                        list.Add(new TestTable()
                        {
                            Id = Convert.ToInt32(read["Id"]),
                            Name = read["Name"].ToString(),
                            brithday = Convert.ToDateTime(read["brithday"]),
                            Age = Convert.ToInt64(read["Age"]),
                            Gender = Convert.ToBoolean(read["Gender"]),
                            IDTypeCode = Convert.ToInt16(read["IDTypeCode"]),

                            Id2 = Convert.ToInt32(read["Id2"]),
                            Name2 = read["Name2"].ToString(),
                            brithday2 = Convert.ToDateTime(read["brithday2"]),
                            Age2 = Convert.ToInt64(read["Age2"]),
                            Gender2 = Convert.ToBoolean(read["Gender2"]),
                            IDTypeCode2 = Convert.ToInt16(read["IDTypeCode2"]),

                            Id3 = Convert.ToInt32(read["Id3"]),
                            Name3 = read["Name3"].ToString(),
                            brithday3 = Convert.ToDateTime(read["brithday3"]),
                            Age3 = Convert.ToInt64(read["Age3"]),
                            Gender3 = Convert.ToBoolean(read["Gender3"]),
                            IDTypeCode3 = Convert.ToInt16(read["IDTypeCode3"]),

                            Id4 = Convert.ToInt32(read["Id4"]),
                            Name4 = read["Name4"].ToString(),
                            brithday4 = Convert.ToDateTime(read["brithday4"]),
                            Age4 = Convert.ToInt64(read["Age4"]),
                            Gender4 = Convert.ToBoolean(read["Gender4"]),
                            IDTypeCode4 = Convert.ToInt16(read["IDTypeCode4"]),

                            Id5 = Convert.ToInt32(read["Id5"]),
                            Name5 = read["Name5"].ToString(),
                            brithday5 = Convert.ToDateTime(read["brithday5"]),
                            Age5 = Convert.ToInt64(read["Age5"]),
                            Gender5 = Convert.ToBoolean(read["Gender5"]),
                            IDTypeCode5 = Convert.ToInt16(read["IDTypeCode5"]),
                        });
                    }
                }
            }
        }

结果

这下最快的变成了for，其次是foreach、最后automapper，可以发现automapper已经慢的b不止一个层次的问题了。

那么，如果我把取数据的地方换成ef呢？

首先是配置

public class MyEntities : IdentityDbContext<IdentityUser>
    {
        public MyEntities() : base("TestDb")
        {
            Database.SetInitializer<MyEntities>(null);
        }
        public DbSet<TestTable> TestTable { get; set; }
        protected override void OnModelCreating(DbModelBuilder modelBuilder)
        {
            modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
            base.OnModelCreating(modelBuilder);
        }
}

使用ef取得数据

public void CreateEfData()
        {
            MyEntities myTest = new MyEntities();
            var demo = from c in myTest.TestTable
                       select c;
            list = demo.ToList();
        }

结果

神奇的事情发生了有没有，automapper居然又恢复到了100毫秒，难道ef对automapper有帮助吗？

可问题依然没有得到解决，我想知道的是，为什么我的map只有500条数据却用了1分钟，让我很纠结

于是我仔细的看的entity

忽然有两行代码引起了我的注意

[ForeignKey("AId")]
        public virtual ATable ATable { get; set; }

        [ForeignKey("BId")]
        public virtual BTable BTable { get; set; }

会不会是因为表关系？

我把map删掉后换成了foreach，

AName = c.ATable == null ? string.Empty : c.ATable.Name,
BName = c.BTable == null ? string.Empty : c.BTable.Name,

运行结果

58秒（XD快了几秒）

那如果我把这两行代码删掉呢？

奇迹发生了，15秒！！！

果然，是这两个表关系搞得鬼，紧接着，我就把代码做了优化

先把数据取出来，然后再进行赋值

var aName = string.Empty;
            var bName = string.Empty;
            if (aList != null && aList.Count > 0)
            {
                aName = aList.FirstOrDefault(s => s.Id == item.AId).Name;
            }
            if (bList != null && bList.Count > 0)
            {
                bName = bList.FirstOrDefault(s => s.Id == item.BId).Name;
            }

            AName = aName
            BName = bName

运行结果 18秒XD~

问题终于得到了解决

可是，话又说回来，ef通过virtual 取出来的关系数据效率有这么差吗？

还是这个地方有别的什么可以优化的地方吗？

这个问题还需要深究。

以上是关于关于数据执行效率的主要内容，如果未能解决你的问题，请参考以下文章

VsCode 代码片段-提升研发效率

关于提高python程序执行效率的思路和想法

几条jQuery代码片段助力Web开发效率提升

十条jQuery代码片段助力Web开发效率提升

VS Code配置snippets代码片段快速生成html模板，提高前端编写效率