数据库优化方面

Posted 2020-11-13 chenminklutz

from https://www.quora.com/I-have-a-table-that-has-1-3-billion-sales-rows-and-most-of-the-columns-are-indexed-It-still-takes-a-long-time-for-me-to-run-a-procedure-or-a-view-even-with-conditions-How-do-I-solve-the-issue

 

Several possibilities come to mind:
1) indexing strategy
2) efficient queries
3) resource configuration
4) database design

First - Perhaps your indexing strategy can be improved. 

Here are some general guidelines consider them carefully - Google these concepts to see if they apply to you:

1) Define a primary key (in most database systems this will create an index)
2) Put indexes on columns used in SQL joins.
3) Put indexes on columns that are used as conditions in WHERE clauses (but if this results in indexing LOTS of columns then you may have database design issues).
4) Put indexes on columns that appear in GROUP BY or ORDER BY

You said, "most of the columns are indexed". It‘s important to carefully choose which columns get indexed, and which columns to include together in one index.

For example, consider a CUSTOMER table with these columns:
cust_id,  email, fname, lname, address_1, address_2, city, state, zip, phone

If your queries often include city, state, zip together in the same SQL then it would be best to create a single index that includes all three columns. 

But if a query uses just the zip then the index might not get used at all. In that case creating an index just for the zip makes sense.

Also, "most of the columns are indexed" makes me wonder if you have a lot of unused indexes on your table. If you have 100 columns and most of them are indexed then it‘s a fair guess that most of the indexes are not needed. Query your database system to get an index usage count. Consider dropping unused or seldom-used indexes.

Second - efficient queries:
Just as important as having a good indexing strategy - are your queries written efficiently? Again, some very general guidelines:
1) Identify the slowest queries
2) examine the query execution plan to see if the expected indexes are being used; if they are not being used, refer to guidelines above to see why this might be the case.
3) Are you using just a few SQL statements to satisfy a broad number of uses? I often find that VIEWS are used this way.

And third - does your platform have enough resources?
Run diagnostics to see if your database server is CPU bound or I/O bound or to check for other problems.

Finally - database design. This is last in my list because it is the most time-consuming. This is a huge topic that I won‘t explore here.

Have you considered partitioning your table?

This is a complex topic that requires detailed review on a case-by-case basis. Examine your query plans in detail. I hope I‘ve given you enough material to begin your research.

I want to acknowledge the advice from others to move to another system (I‘ve had a lot of success with Vertica DB), but I think you should start by looking at indexing and query effiecency to see if there is room for immediate improvement.