在windows 64位系统上编译hadoop2.7.1成功经验分享

根据官方的build文件说明,按照要求一步步做下去,应该是没问题的,但每个人计算机软硬件环境不同,可能会有出入,本文针对我个人遇到的问题进行说明,可能有些问题你遇不到,也可能你遇到的问题我没有遇到。届时各位还需具体问题具体分析。
我的硬件:E3+16G
我的软件:
windows server 2012 R2,这个系统只有64位没有32位;
jdk1.8不行,后来按照要求换成1.7;下载地址:32位    64位
eclipse Version: Luna Service Release 2 (4.4.2);
maven 3.3.9;下载地址
ProtocolBuffer 2.5.0;下载地址
cmake version 3.4.1;下载地址
Windows SDK 7.1;下载地址
Cygwin for 64-bit versions of Windows,版本号不详。下载地址
hadoop2.7.1 src;下载地址

把以上软件全部下载到本地,需要解压,安装。注意,Findbugs、Windows SDK 8.1和zip我没有使用。

maven 3.3.9的安装和配置可以参考这篇文章:

Maven在Eclipse的基本使用_蒋艳华_新浪博客


在cmd里执行mvn -v能看到maven版本号就算成功,类似下面:

Apache Maven 3.3.9
Maven home: E:\apache-maven-3.3.9
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: E:\Java\jdk1.7\jre

ProtocolBuffer 2.5.0的安装和配置
其实就是把protoc.exe拷贝到windows的system32目录下就可以了。你也可以设置path环境变量。
翻网络说是要下载源代码和win32-zip二进制文件,实际只要下载win32-zip那个文件就可以,就要里面的protoc.exe。
在cmd里执行protoc –version能看到protoc版本号就算成功,类似下面:

libprotoc 2.5.0

cmake的安装和配置,一样,执行cmake –version,看到类似下面的就算成功:

cmake version 3.4.1

安装Cygwin主要是使用里面的工具:sh, mkdir, rm, cp, tar, gzip,这些工具都要加到环境变量里。安装cygwin会自动处理,如果没有需要手动添加环境变量。
所以只要你去cmd里敲这些命令都能使用,那就说明安装成功。
例如看下sh的版本号,看到下面类似的说明sh就可以正常使用:

D:\>sh --version
GNU bash, version 4.3.42(5)-release (x86_64-pc-msys)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Windows SDK 7.1安装和配置
我反复安装多次都报错:problem occurred while installing selected Windows SDK components.
解决办法:卸载vc2010 Redistributable 10.0.4及以上所有版本,包括x86和64,针对我个例我卸载了vc2010、vc2012、vc2013 x86 x64共6个,然后安装成功。目前正常使用,编译项目也成功。
具体可以参考这篇文章:

安装Windows SDK7.1时发生的一个错误(附解决办法) – HEVC_CJL的专栏 – 博客频道 – CSDN.NET

jdk的安装,要求用1.7,我之前安装的1.8在编译hadoop时出错。另外从高版本到低版本切换,eclipse项目会报错,而且貌似jdk32位或64位要和eclipse对应,具体可以参考这篇文章:

从高版本JDK换成低版本JDK报错Unsupported major.minor version 52.0_neolone_新浪博客

最后把hadoop2.7.1的源代码解压,放在一个较短的路径里,比如d:\hdp

至此,准备工作就完成了。

下面开始编译,要求必须进入 Windows SDK 命令行界面, (点击开始, 所有程序,
Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.)

如果你的系统是64位的,执行如下命令设置环境变量:

set Platform=x64

如果你的系统是32位的,执行如下命令设置环境变量:

set Platform=Win32

编译并安装 hadoop maven plugins 插件

d:
cd d:\hdp\hadoop-maven-plugins
mvn install

然后生成Eclipse工程文件

cd d:\hdp
mvn eclipse:eclipse -DskipTests

以上两步,应该会看到命令行窗口显示大量信息,最后出现build success,那就成功。

接下来开始编译hadoop2.7.1
Windows下面编译打包hadoop2.7.1的命令为:

mvn package -Pdist,native-win -DskipTests -Dtar

同样,应该会看到命令行窗口显示大量信息,中间可能还需要连接网络去下载依赖项,这些都是maven帮我们完成,非常省事,最后出现build success,那就成功。应该会在d:\hdp\hadoop-dist\target\hadoop-2.5.1.tar.gz看到打包好的文件。
我本例实际只到hadoop-common这个模块成功,后面的一个失败,剩下的全部跳过了,但我只需要用hadoop-common,所以就没有继续下去,以后有时间了会再好好研究下怎么回事。
我的hadoop-common在这里找到了:”D:\hdp\hadoop-common-project\hadoop-common\target\hadoop-common-2.7.1.jar”,大小和原包只差了几十字节,导入项目测试可以正常使用。
整个编译过程20多分钟,如果全部编译成功,估计要更久。
具体编译过程和遇到的问题及解决办法可以参考下面这几篇帖子:

Hadoop源代码在Windows下面的编译 – 推酷

Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS x64 – SrcCodes

hadoop2.6在win7上的编译-报错处理-Hadoop2|YARN-about云开发

以下是官方要求和编译命令,大家可以参考
———————————————————————————-
Requirements:

* Windows System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
tools must be present on your PATH.

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/download/win.

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

The Windows SDK 8.1 is available to download at:

http://msdn.microsoft.com/en-us/windows/bg162891.aspx

Cygwin is neither required nor supported.

———————————————————————————-
Building:

Keep the source code tree in a short path to avoid running into problems related
to Windows maximum path length limitation. (For example, C:\hdc).

Run builds from a Windows SDK Command Prompt. (Start, All Programs,
Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.)

JAVA_HOME must be set, and the path must not contain spaces. If the full path
would contain spaces, then use the Windows short path instead.

You must set the Platform environment variable to either x64 or Win32 depending
on whether you’re running a 64-bit or 32-bit system. Note that this is
case-sensitive. It must be “Platform”, not “PLATFORM” or “platform”.
Environment variables on Windows are usually case-insensitive, but Maven treats
them as case-sensitive. Failure to set this environment variable correctly will
cause msbuild to fail while building the native code in hadoop-common.

set Platform=x64 (when building on a 64-bit system)
set Platform=Win32 (when building on a 32-bit system)

Several tests require that the user must have the Create Symbolic Links
privilege.

All Maven goals are the same as described above with the exception that
native code is built by enabling the ‘native-win’ Maven profile. -Pnative-win
is enabled by default when building on Windows since the native components
are required (not optional) on Windows.

If native code bindings for zlib are required, then the zlib headers must be
deployed on the build machine. Set the ZLIB_HOME environment variable to the
directory containing the headers.

set ZLIB_HOME=C:\zlib-1.2.7

At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested
with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in
the zlib 1.2.7 source tree.

http://www.zlib.net/

———————————————————————————-
Building distributions:

* Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]