Comparative Test Report on esProc, Hive, Impala Clusters (part VI)

6. Hermeneutic Analysis

In general, by observing the results tested from each use case, the data features may be concluded as below:

1.In most cases, the performances of three take on a downgrading trend, that is, esProc ranks top, Impala takes second place, Hive is worst. esProc comfortably stays ahead of Impala, and is several times higher than Hive in performance.

2.The case of big group is somewhat special. Hive presents a lot better performance than Impala. In the use case for common group, Impala is also inferior to Hive.

3.Impala is not sensitive to computation amount, and it degrades slowly even if the computation amount continues to grow.

The reasons why these three tools present the above features come here as follows: 

1. esProc's performance takes the top level, which could benefit from hard disk IO. esProc enables direct access to hard disk bypass HDFS, while Hive/Impala depends on HDFS in this regard. As the most time for big data computation is consumed on the hard disk, esProc may gain the performance boosts of hard disk IO. Of course, this advantage of esProc makes sense only for small and middle scales of clusters, because in large cluster environment, esProc still ensures the data safety by HDFS or by other redundancy performance.

2. In most cases, Impala demonstrates better performance than Hive, which could involve data exchange. Impala supports in-memory computation, it can exchange data in memory; while Hive only supports out-memory computation, it has to do a data exchange by hard disk. In terms of exchange data, Impala's performance is higher than that of Hive at least by an order of magnitude, but from its overall performance, the gap of 3-90 times just like what Cloudera declared does not appear at all, generally only 2-3 times as it surpasses over Hive. 

3. When operating a big group, Impala is far more inferior to Hive. In the use case for common group, Impala also performs worse than Hive on rare occasion. Both cases appear as long as there is a large mass of data. As you think, Impala only provides a support for in-memory computation, so you can guess this phenomenon isdue to the fact that the data volume is massive enough to reach the limit of memory. At this time, JVM needs to proceed frequent memory exchanges. In fact, the memory overflow will appear in Impala when data amount further increases. Unless additional physical memory is available, the computation can't be achieved.

4. Impala is not sensitive to the computation amount, and it degrades slowly even if the computation amount continues to grow. This is because Impala supports dynamical native code generation, while Hive and esProc is interpreted by JAVA with a big gap of executingefficiency. The big data computation, however, mainly consumes the most time on the hard disk IO, not code execution Impala's advantage in native code generation does not often help improve its overall performance. 

Comparative Test Report on esProc, Hive, Impala Clusters (part V)

5.4 Use Case for dimension Table across Nodes

This case is used for testing performance of foreign key join over big data. The dimension table in this case is big enough to exceed the memory size. At this time, it will be segmented to be loaded into several sub-nodes.

Use Case Number Description:

Tested Results :

Data Features:

1. esProc, Impala and Hive degrade in turn, and esProc shows an evident advantage.

2. Data type also produces a very minimal impact on the performances of three test objects.  

5.5 Use Case for Big Group

This case is used for testing performance of big data grouping, and the number of grouped result goes far beyond the physical memory that the computation can’t be done within  memory.

Use Case Description: 

Tested Results 

Data Features:

1. As we can see, the change has taken place in the rank. esProc, Hive, Impala degrade in turn, esProc still significantly takes the lead, and Impala present a worst performance. 

2. The performance of Impala will decrease sharply when there is a huge mass of the data.

3. Data type also produces a very minimal impact on the performances of three test objects.

Comparative Test Report on esProc, Hive, Impala Clusters (part IV)

5.3 Joining Use Case

This case is used for testing performance of foreign key join over big data, involving of integer, digital and string join as well as single-level and multilevel.

Use Case Number Description:

 Tested Results:

Data Features:
1.esProc, Impala and Hive degrade in turn. At top level of performances, esProc is nearly twice times higher than Impala, while 4-6 times than Hive. 

2.Impala lacks sensitivity to computation amount. As you imagine, if the computing computation amount continues to increase, its performance will exceed that of esProc. 

3.When the number of levels needed to be joined increases, the performances of both esProc and Impala have unapparent changes, but Hive’s performance will be reduced significantly.

Comparative Test Report on esProc, Hive, Impala Clusters (part III)

5.2 Use Case for Group

This case is used for testing performance of big data grouping, while separately taking integer, digital, string, date and other grouping into account. The number of grouped results in this case is relatively small, less than physical memory of node.

Use Case Number Description:

Tested Results:

Data Feature:

In general, esProc, Impala and Hive degrade in turn. At top level of performances, esProc is 2-3 times higher than Impala, while 3-4 times higher than Hive. Apart from Use cases 32, 36, 40 which are used for wide table with large amount data at this point, Hive presents higher performance than Impala.

Comparative Test Report on esProc, Hive, Impala Clusters (part II)

5. Test Case

To make you easy to understand, the operation logic of any test will be described in SQL. During the test, Hive and Impala implement SQL statements directly, while in esProc, we will write the codes to achieve SQL function-equivalent operations.

5.1 Use Case for Scan

This case is used for testing performance of full-table scanning of big data, while separately considering simple counting, integer summary, float  summary and numeric summary, as well as filtering of integer, digital, string and date, and other functions.

Use case description:

Tested Results

Note: Time unit is in seconds.

Data Fetures:

1. esProc, Impala and Hive degrade in turn, at top level of performance, esProc is 2-3 times higher than Impala, and 3-4 times than Hive.

2. Computing amount has a little impact on the performances of three test objects.

3. Data type also produces a very minimal impact on the performances of three test objects.

Comparative Test Report on esProc, Hive, Impala Clusters (part I)

1. Objective

By making separate tests on computation capabilities of esProc, Hive and Impala clusters as they are running in the same hardware environment, to demonstrate different performances by comparison.

2. Test Content and Method

Cluster Scale: 4 Nodes.

Data volume: 125G wide fact table and 143G narrow fact table are used as primary data for testing, whose data volume is considerably larger than the physical memory of computer node.

Algorithm classification: Total five typical SQL algorithms: scan, group, join, join big dimension table across nodes and big group, are separately tested. What you should note here is that, the purpose of adopting these simple algorithms is only to help understand testing process and intuitively demonstrate comparative performances, but not means esProc and SQL are fully consistent. In fact, both of them lay stress on different key functions, that is to say, esProc is good at computation with relatively more complex business logic, while SQL is fit to operate some common complex computations. Complex algorithm in SQL will be executed with different plan, which is out of manual control, and thus not good for comparison. We'll not do such test.

Category of use cases: A number of sets of use cases will be designed for test process according to table width, data types and computation amount.

Storage structure: Row-based storage.

Note: The Comparative Test Process for esProc, Hive, Impala Clusters is annexed to this report. For specific data structure, test code, test reproducibility and other contents, refer to this document.

3. Environment Description

Hardware:

       Number of PCs：4

       CPU：Intel Core i5 2500(4 Cores)

       RAM：16G

       HDD：2T/7200rpm

       Ethernet adapter：1000M

Software:

       OS: CentOS6.4

       JDK：1.7

       Hadoop/hdfs 2.2.0

Test Objects:

       Hive  0.11.0

       esProc     3.1

       Impala    1.2.0

4. Data Description

Data scale is defined based on the exported files.

The file formats that help demonstrate the highest performance of each object are used here, of which, esProc uses the proprietary binary files; Hive and Impala use the text files.

4.1 Data Table and Associative Table

Fact Table T1

      

This is a wide Table, used to simulate the fact case with a number of fields, which is designed with 100 fields.

      

Fact TableT2

      

This is narrow Table T2, used to simulate a fact case with lesser fields, which is designed with 11 fields.

The fact table is primary data source in this test, which will be used during scanning, grouping, joining processes.

Dimension Tables DL2, DL6, DD2, DD6, DC2, DC6

The dimension table is only used to test the usecase of join (and multi-level join) operation. These dimension tables will join with the fact table, also join with each other with smaller amount of data.

      

DC11, as a dimension table across nodes, must be loaded as segmented into memory of different computers because the table is too large to load into one computer. Cluster needs to load DC11 across nodes to complete the computation.

4.2 . Data Scale