Kylin-基本知识

Posted 07H_JH

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Kylin-基本知识相关的知识,希望对你有一定的参考价值。

CUBE

  • Table - This is definition of hive tables as source of cubes, which must be synced before building cubes.
    技术分享

  • Data Model - This describes a STAR SCHEMA data model, which defines fact/lookup tables and filter condition.
    技术分享

  • Cube Descriptor - This describes definition and settings for a cube instance, defining which data model to use, what dimensions and measures to have, how to partition to segments and how to handle auto-merge etc.
    技术分享

  • Cube Instance - This is instance of cube, built from one cube descriptor, and consist of one or more cube segments according partition settings.
    技术分享

  • Partition - User can define a DATE/STRING column as partition column on cube descriptor, to separate one cube into several segments with different date periods.
    技术分享

  • Cube Segment - This is actual carrier of cube data, and maps to a HTable in HBase. One building job creates one new segment for the cube instance. Once data change on specified data period, we can refresh related segments to avoid rebuilding whole cube.
    技术分享

  • Aggregation Group - Each aggregation group is subset of dimensions, and build cuboid with combinations inside. It aims at pruning for optimization.
    技术分享

DIMENSION & MEASURE

  • Mandotary - This dimension type is used for cuboid pruning, if a dimension is specified as “mandatory”, then those combinations without such dimension are pruned.
  • Hierarchy - This dimension type is used for cuboid pruning, if dimension A,B,C forms a “hierarchy” relation, then only combinations with A, AB or ABC shall be remained.
  • Derived - On lookup tables, some dimensions could be generated from its PK, so there’s specific mapping between them and FK from fact table. So those dimensions are DERIVED and don’t participate in cuboid generation.
    技术分享

  • Count Distinct(HyperLogLog) - Immediate COUNT DISTINCT is hard to calculate, a approximate algorithm - HyperLogLog is introduced, and keep error rate in a lower level.
  • Count Distinct(Precise) - Precise COUNT DISTINCT will be pre-calculated basing on RoaringBitmap, currently only int or bigint are supported.
  • Top N - For example, with this measure type, user can easily get specified numbers of top sellers/buyers etc.
    技术分享

CUBE ACTIONS

  • BUILD - Given an interval of partition column, this action is to build a new cube segment.
  • REFRESH - This action will rebuilt cube segment in some partition period, which is used in case of source table increasing.
  • MERGE - This action will merge multiple continuous cube segments into single one. This can be automated with auto-merge settings in cube descriptor.
  • PURGE - Clear segments under a cube instance. This will only update metadata, and won’t delete cube data from HBase.
    技术分享

JOB STATUS

  • NEW - This denotes one job has been just created.
  • PENDING - This denotes one job is paused by job scheduler and waiting for resources.
  • RUNNING - This denotes one job is running in progress.
  • FINISHED - This denotes one job is successfully finished.
  • ERROR - This denotes one job is aborted with errors.
  • DISCARDED - This denotes one job is cancelled by end users.
    技术分享

JOB ACTION

  • RESUME - Once a job in ERROR status, this action will try to restore it from latest successful point.
  • DISCARD - No matter status of a job is, user can end it and release resources with DISCARD action.
    技术分享












以上是关于Kylin-基本知识的主要内容,如果未能解决你的问题,请参考以下文章

Kylin-基本知识

Kylin实践——Kylin中必须要理解的基本概念和术语

Kylin3.1.2实战--调优CUBE构建数据倾斜优化

Kylin 基本原理及概念

kylin build报错Execution Error, return code 1

Kylin实践—— Kylin安装启动