hibench7.0编译安装

 

引言

HiBench是一个大数据基准套件,可以帮助您评测不同大数据平台的性能、吞吐量和系统资源利用率。本文仅介绍如何对hadoop进行测试,其他大数据平台使用请参考官网https://github.com/intel-hadoop/HiBench

软件依赖

HiBench需要java环境,以及Maven管理。

安装java运行环境

新建目录/home/java 然后上传相应的jdk二进制文件到此目录并解压。(本例为aarch64为平台,x86平台一样的操作步骤,亲试)

root@kylin1:/home/java# ll

总用量 71608

drwxr-xr-x 3 root root     4096 4月   8 11:59 ./

drwxr-xr-x 4 root root     4096 4月   8 11:28 ../

drwxr-xr-x 7 root root     4096 4月   8 11:44 jdk1.8.0_201/

-rw-r–r– 1 root root 73312819 4月   8 11:43 jdk-8u201-linux-arm64-vfp-hflt.tar.gz

添加环境变量:/etc/profile 文件末尾增加如下行

export JAVA_HOME=/home/java/jdk1.8.0_201

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

使其生效

source /etc/profile

查看

root@kylin1:/home/java# java -version

java version “1.8.0_201”

Java(TM) SE Runtime Environment (build 1.8.0_201-b09)

Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

 

安装Maven

下载Maven包

wget http://apache.fayea.com/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.zip

解压缩

unzip apache-maven-3.6.0-bin.zip -d /usr/local/

添加环境变量

cat /etc/profile,在文件末尾增加如下

export M3_HOME=/usr/local/apache-maven-3.6.0

export PATH=$M3_HOME/bin:$PATH

 

source /etc/profile

测试Maven环境

mvn -v

看到相应版本信息输出即表明配置正确:

root@kylin3:/usr/local/HiBench-HiBench-7.0# mvn -v

Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00)

Maven home: /usr/local/apache-maven-3.6.0

Java version: 1.8.0_201, vendor: Oracle Corporation, runtime: /home/java/jdk1.8.0_201/jre

Default locale: zh_CN, platform encoding: UTF-8

OS name: “linux”, version: “4.4.58-20180615.kylin.server.yun+-generic”, arch: “aarch64”, family: “unix”

 

下载HiBench

git clone https://github.com/intel-hadoop/HiBench.git比较慢,建议直接网页下载zip包,然后解压到/usr/local目录下。

root@kylin3:/usr/local/HiBench-HiBench-7.0# pwd

/usr/local/HiBench-HiBench-7.0

 

安装Hibench

切到HiBench下,执行对应的安装操作,可以选择自己想要安装的模块。以安装hadoop框架为例:

root@kylin3:/usr/local/HiBench-HiBench-7.0# mvn -Phadoopbench -Dspark=2.1 -Dscala=2.11 clean package

更多安装方法参考https://github.com/Intel-bigdata/HiBench/blob/master/docs/build-hibench.md

 

因为网络原因,下次模块过程中经常出现暂停现象。当前解决办法暂停下载,ctrl+c 然后继续执行上面的命令。

遇到的问题:

[WARNING] Could not get content

org.apache.maven.wagon.TransferFailedException: Failed to transfer file http://archive.apache.org/dist/hive/hive-0.14.0/apache-hive-0.14.0-bin.tar.gz with status code 503

下载hive出错,解决:

root@kylin3:/usr/local/HiBench-HiBench-7.0# pwd

/usr/local/HiBench-HiBench-7.0

root@kylin3:/usr/local/HiBench-HiBench-7.0# vim hadoopbench/sql/pom.xml

修改该文件中,将http改为https即可

<properties>

    <repo>https://archive.apache.org</repo>

<file>dist/hive/hive-0.14.0/apache-hive-0.14.0-bin.tar.gz</file>

</properties>

 

 

org.apache.maven.wagon.TransferFailedException: Failed to transfer file http://archive.apache.org/dist/nutch/apache-nutch-1.2-bin.tar.gz with status code 503

root@kylin3:/usr/local/HiBench-HiBench-7.0/hadoopbench/nutchindexing# pwd

/usr/local/HiBench-HiBench-7.0/hadoopbench/nutchindexing

vim pod.xml

修改如下文件由http改成https

<configuration>

<url>https://archive.apache.org/dist/nutch/apache-nutch-1.2-bin.tar.gz</url>

</configuration>

 

最终结果

[INFO] Reactor Summary:

[INFO]

[INFO] hibench 7.0 …………………………………. SUCCESS [  0.173 s]

[INFO] hibench-common 7.0 …………………………… SUCCESS [ 10.755 s]

[INFO] HiBench data generation tools 7.0 ……………… SUCCESS [ 12.696 s]

[INFO] hadoopbench 7.0 ……………………………… SUCCESS [  0.004 s]

[INFO] hadoopbench-sql 7.0 ………………………….. SUCCESS [  2.355 s]

[INFO] mahout 7.0 ………………………………….. SUCCESS [  6.676 s]

[INFO] PEGASUS: A Peta-Scale Graph Mining System 2.0-SNAPSHOT SUCCESS [  0.978 s]

[INFO] nutchindexing 7.0 ……………………………. SUCCESS [ 41.968 s]

[INFO] ————————————————————————

[INFO] BUILD SUCCESS

[INFO] ————————————————————————

[INFO] Total time:  01:15 min

[INFO] Finished at: 2019-04-09T11:58:27+08:00

[INFO] ————————————————————————

 

 

发布者

deelaaay

己所不欲,勿施于人。