1.下载mahout
下载地址:http://mahout.apache.org
我下载的最新版
:mahout-distribution-0.9
2.把mahout解压到你想存放的文档,我是放在/Users/jia/Documents/hadoop-0.20.2,即hadoop的安装目录上。
3.为mahout配置环境
打开终端,打开profile文件所在的目录
JIAS-MacBook-Pro:~ jia$ open /etc
把profile文件复制到桌面,然后编辑,在它后面加入环境变量
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/<span style="color: #000000;">Home</p><p>export HADOOP_HOME</span>=Documents/hadoop-0.20.2<span style="color: #000000;"></p><p>export MAHOUT_HOME</span>=Documents/hadoop-0.20.2/mahout-distribution-0.9<span style="color: #000000;"></p><p>export MAVEN_HOME</span>=Documents/apache-maven-3.2.2<span style="color: #000000;"></p><p>export PATH</span>=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$MAVEN_HOME/bin:$MAHOUT_HOME/<span style="color: #000000;">bin</p><p>export HADOOP_CONF_DIR</span>=Documents/hadoop-0.20.2/<span style="color: #000000;">conf</p><p>export MAHOUT_CONF_DIR</span>=Documents/hadoop-0.20.2/mahout-distribution-0.9/<span style="color: #000000;">conf</p><p>export classpath</span>=$classpath:$JAVA_HOME/lib:$MAHOUT_HOME/lib:$HADOOP_CONF_DIR:$MAHOUT_CONF_DIR
然后把桌面上的profile文件覆盖/etc上的profile,期间要输入管理员密码
注意:
1.如果在ubuntu下安装的是hadoop2.6进行配置的话,路径为:
export JAVA_HOME=/usr/lib/jvm/java-<span style="color: #800080;">1.7</span>.<span style="color: #800080;">0</span>-openjdk-<span style="color: #000000;">amd64</p><p>export HADOOP_HOME</span>=/home/sendi/hadoop-<span style="color: #800080;">2.6</span>.<span style="color: #800080;">0</span><span style="color: #000000;"></p><p>export MAHOUT_HOME</span>=/home/sendi/mahout-distribution-<span style="color: #800080;">0.9</span><span style="color: #000000;"></p><p>export MAVEN_HOME</span>=/home/sendi/apache-maven-<span style="color: #800080;">3.3</span>.<span style="color: #800080;">3</span><span style="color: #000000;"></p><p>export PATH</span>=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$MAVEN_HOME/bin:$MAHOUT_HOME/<span style="color: #000000;">bin</p><p>export HADOOP_CONF_DIR</span>=/home/sendi/hadoop-<span style="color: #800080;">2.6</span>.<span style="color: #800080;">0</span>/etc/<span style="color: #000000;">hadoop</p><p>export MAHOUT_CONF_DIR</span>=/home/sendi/mahout-distribution-<span style="color: #800080;">0.9</span>/<span style="color: #000000;">conf</p><p>export classpath</span>=$classpath:$JAVA_HOME/lib:$MAHOUT_HOME/lib:$HADOOP_CONF_DIR:$MAHOUT_CONF_DIR
View Code
2.配置MAHOU_CONF_DIR时有些网站说时export MAHOUT_CONF_DIR=Documents/hadoop-0.20.2/mahout-distribution-0.9/src/conf
0.9版本的正确配置是:export MAHOUT_CONF_DIR=Documents/hadoop-0.20.2/mahout-distribution-0.9/conf ,因为当你打开mahout文件夹时,发现没有src这个目录<br> mahout官网上0.9版本有几个压缩文件,我自己试过,前面两个小压缩文件不行。<br>
这里我选择的是第5个78M的。
4.检验mahout是否配置成功
4.1启动hadoop
JIAS-MacBook-Pro:hadoop-0.20.2 jia$ bin/start-all.sh
4.2查看mahout
JIAS-MacBook-Pro:mahout-distribution-0.9 jia$ bin/<span style="color: #000000;">mahout </p><p>MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.</p><p>hadoop binary is not in PATH,HADOOP_HOME</span>/bin,HADOOP_PREFIX/<span style="color: #000000;">bin, running locally</p><p>An example program must be given as the first argument.</p><p>Valid program names are:</p><p> arff.vector: : Generate Vectors from an ARFF file or directory</p><p> baumwelch: : Baum</span>-Welch algorithm <span style="color: #0000ff;">for</span><span style="color: #000000;"> unsupervised HMM training</p><p> canopy: : Canopy clustering</p><p> cat: : Print a file or resource as the logistic regression models would see it</p><p> cleansvd: : Cleanup and verification of SVD output</p><p> clusterdump: : Dump cluster output to text</p><p> clusterpp: : Groups Clustering Output In Clusters</p><p> cmdump: : Dump confusion matrix in HTML or text formats</p><p> concatmatrices: : Concatenates </span>2<span style="color: #000000;"> matrices of same cardinality into a single matrix</p><p> cvb: : LDA via Collapsed Variation Bayes (0th deriv. approx)</p><p> cvb0_local: : LDA via Collapsed Variation Bayes, in memory locally.</p><p> evaluateFactorization: : compute RMSE and MAE of a rating matrix factorization against probes</p><p> fkmeans: : Fuzzy K</span>-<span style="color: #000000;">means clustering</p><p> hmmpredict: : Generate random sequence of observations by given HMM</p><p> itemsimilarity: : Compute the item</span>-item-similarities <span style="color: #0000ff;">for</span> item-<span style="color: #000000;">based collaborative filtering</p><p> kmeans: : K</span>-<span style="color: #000000;">means clustering</p><p> lucene.vector: : Generate Vectors from a Lucene index</p><p> lucene2seq: : Generate Text SequenceFiles from a Lucene index</p><p> matrixdump: : Dump matrix in CSV format</p><p> matrixmult: : Take the product of two matrices</p><p> parallelALS: : ALS</span>-<span style="color: #000000;">WR factorization of a rating matrix</p><p> qualcluster: : Runs clustering experiments and summarizes results in a CSV</p><p> recommendfactorized: : Compute recommendations using the factorization of a rating matrix</p><p> recommenditembased: : Compute recommendations using item</span>-<span style="color: #000000;">based collaborative filtering</p><p> regexconverter: : Convert text files on a per line basis based on regular expressions</p><p> resplit: : Splits a set of SequenceFiles into a number of equal splits</p><p> rowid: : Map SequenceFile</span><Text,VectorWritable> to {SequenceFile<IntWritable,VectorWritable>, SequenceFile<IntWritable,Text><span style="color: #000000;">}</p><p> rowsimilarity: : Compute the pairwise similarities of the rows of a matrix</p><p> runAdaptiveLogistic: : Score </span><span style="color: #0000ff;">new</span><span style="color: #000000;"> production data using a probably trained and validated AdaptivelogisticRegression model</p><p> runlogistic: : Run a logistic regression model against CSV data</p><p> seq2encoded: : Encoded Sparse Vector generation from Text sequence files</p><p> seq2sparse: : Sparse Vector generation from Text sequence files</p><p> seqdirectory: : Generate sequence files (of Text) from a directory</p><p> seqdumper: : Generic Sequence File dumper</p><p> seqmailarchives: : Creates SequenceFile from a directory containing gzipped mail archives</p><p> seqwiki: : Wikipedia xml dump to sequence file</p><p> spectralkmeans: : Spectral k</span>-<span style="color: #000000;">means clustering</p><p> split: : Split Input data into test and train sets</p><p> splitDataset: : split a rating dataset into training and probe parts</p><p> ssvd: : Stochastic SVD</p><p> streamingkmeans: : Streaming k</span>-<span style="color: #000000;">means clustering</p><p> svd: : Lanczos Singular Value Decomposition</p><p> testnb: : Test the Vector</span>-<span style="color: #000000;">based Bayes classifier</p><p> trainAdaptiveLogistic: : Train an AdaptivelogisticRegression model</p><p> trainlogistic: : Train a logistic regression using stochastic gradient descent</p><p> trainnb: : Train the Vector</span>-<span style="color: #000000;">based Bayes classifier</p><p> transpose: : Take the transpose of a matrix</p><p> validateAdaptiveLogistic: : Validate an AdaptivelogisticRegression model against hold</span>-<span style="color: #000000;">out data set</p><p> vecdist: : Compute the distances between a set of Vectors (or Cluster or Canopy, they must fit in memory) and a list of Vectors</p><p> vectordump: : Dump vectors from a sequence file to text</p><p> viterbi: : Viterbi decoding of hidden states from given output states sequence</span>
这里需要说明下,当你看到下面的代码时,以为是错的,其实不是,原因:
MAHOUT_LOCAL:设置是否本地运行,如果设置这个参数就不会运行hadoop了,一旦设置这个参数,那HADOOP_CONF_DIR 和HADOOP_HOME 这两个参数的
设置就自动失效了。
当初我就在这个问题上纠结了很久。
<span style="color: #000000;">MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.</p><p>hadoop binary is not in PATH,HADOOP_HOME</span>/bin,HADOOP_PREFIX/bin, running locally
5.运行mahout的算法
5.1到下面的地址去下载测试数据
http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data
5.2创建测试目录testdata,并把数据导入到这个testdata目录中
JIAS-MacBook-Pro:hadoop-0.20.2 jia$ bin/hadoop fs -mkdir testdata
5.3把测试数据上传到hdfs上,不能把测试数据存在mac上用pages建立的文档上,而是建立一个新的文件命令:touch data
JIAS-MacBook-Pro:hadoop-0.20.2 jia$ bin/hadoop fs -put workspace/data testdata/
5.4运行mahout上的kmeans算法
JIAS-MacBook-Pro:hadoop-0.20.2 jia$ bin/hadoop jar mahout-distribution-0.9/mahout-examples-0.9-job.jar org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
5.5查看结果
JIAS-MacBook-Pro:~ jia$ cd Documents/hadoop-0.20.2/<span style="color: #000000;"></p><p>JIAS</span>-MacBook-Pro:hadoop-0.20.2 jia$ bin/hadoop fs -ls output/<span style="color: #000000;"></p><p>Found </span>15<span style="color: #000000;"> items</p><p></span>-rwxrwxrwx 1 jia staff 194 2014-08-03 14:42 /Users/jia/Documents/hadoop-0.20.2/output/<span style="color: #000000;">_policy</p><p>drwxr</span>-xr-x - jia staff 136 2014-08-03 14:42 /Users/jia/Documents/hadoop-0.20.2/output/<span style="color: #000000;">clusteredPoints</p><p>drwxr</span>-xr-x - jia staff 544 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-0<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-1<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:42 /Users/jia/Documents/hadoop-0.20.2/output/clusters-10-<span style="color: #0000ff;">final</span><span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-2<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-3<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-4<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-5<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-6<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/clusters-7<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:42 /Users/jia/Documents/hadoop-0.20.2/output/clusters-8<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 204 2014-08-03 14:42 /Users/jia/Documents/hadoop-0.20.2/output/clusters-9<span style="color: #000000;"></p><p>drwxr</span>-xr-x - jia staff 136 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/<span style="color: #000000;">data</p><p>drwxr</span>-xr-x - jia staff 136 2014-08-03 14:41 /Users/jia/Documents/hadoop-0.20.2/output/random-seeds