Apache Hadoop How To Install The
The Hadoop project has gained a lot of notoriety thanks to its great results in implementing a multi-server distributed computing system for handling huge amounts of data.This topic explains how to install the Hortonworks driver for Apache Hive, which is a fully compliant ODBC driver that supports multiple Hadoop distributions.As I needed an exact version for my work to reproduce an issue, I'll go with version 2.4.0 for now. If you wanted to use a different version of Spark & Hadoop, select the one you wanted from drop-downs and.Apache Hadoop was initially developed by Yahoo and the project is a combination between the previous Apache Hadoop Core and Apache Hadoop Common repos. Below I am using my MacBook Pro as my local machine, but As the current latest version in macports is 2.6.x, we need to stick to an earlier version:Related: Apache Spark Installation on Windows.
Run some of the examples provided (that's actually one line.):Bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep input output 'dfs+'Copy the output files from the distributed filesystem to the local filesystem and examine them:View the output files on the distributed filesystem: This is a raw copy from the official docs. Look around in the tags folder for other versions.We need to patch JniBasedUnixGroupsNetgroupMapping:The contents of ~/tools/hadoop-2.4.0/etc/hadoop/hdfs-site.xml:Cat ~/.ssh/id_dsa.pub > ~/.ssh/authorized_keysLet's see what we've did.
Expression: file.exists()At org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:395)At org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:683)At org.codehaus.mojo.jspc.CompilationMojoSupport.findToolsJar(CompilationMojoSupport.groovy:371)At org.codehaus.mojo.jspc.CompilationMojoSupport.this$4$findToolsJar(CompilationMojoSupport. 4:132 in /Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/target/antrun/build-main.xmlIf you don't have JAVA_HOME correctly set, you'll get - Detecting CXX compiler ABI info - Detecting CXX compiler ABI info - done - Detecting CXX compile features - done CMake Error at /opt/local/share/cmake-3.2/Modules/FindPackageHandleStandardArgs.cmake:138 (message): Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH) Call Stack (most recent call first): /opt/local/share/cmake-3.2/Modules/FindPackageHandleStandardArgs.cmake:374 (_FPHSA_FAILURE_MESSAGE) /opt/local/share/cmake-3.2/Modules/FindJNI.cmake:287 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) - Configuring incomplete, errors occurred! See also "/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log".JniBasedUnixGroupsNetgroupMapping.c patch missingIf you don't have the patch for JniBasedUnixGroupsNetgroupMapping.c above, you'll get Building C object CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o /Library/Developer/CommandLineTools/usr/bin/cc -Dhadoop_EXPORTS -g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fPIC -I/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/target/native/javah -I/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/src/main/native/src -I/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/src -I/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/src/src -I/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/target/native -I/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/include/darwin -I/opt/local/include -I/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util -o CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o -c /Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c /Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression ('void' and 'int') make: *** Error 1 make: *** Error 2Without applying the fcloseall patch above, you might get the following error: Undefined symbols for architecture x86_64: _launch_container_as_user in libcontainer.a(container-executor.c.o) ld: symbol(s) not found for architecture x86_64 collect2: error: ld returned 1 exit statusWithout the "export JAVA_HOME=`/usr/libexec/java_home -v 1.7` sudo mkdir $JAVA_HOME/Classes sudo ln -s $JAVA_HOME/lib/tools.jar $JAVA_HOME/Classes/classes.jar" line creating the symlinks above, you'll getException in thread "main" java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/Classes/classes.jar. Without ProtoBufIf you don't have protobuf, you'll get the following error: - hadoop-maven-plugins:2.4.0:protoc (compile-protoc) hadoop-common - failed: java.io.IOException: Cannot run program "protoc": error=2, No such file or directoryIf you don't have the correct version of protobuf, you'll get Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.4.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' -> Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/Users/doma/dev/hadoop-2.4.0/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory around Ant part. They meant to drive you here via google -) Apply the procedure above and all of these errors will be fixed for you.
0 kommentar(er)
