发表于   |   分类于   |     |   阅读次数

下载

sqoop-1.4.6 安装包

配置

sqoop-env.sh

1
2
3
4
5
6
7
8
9
10
#Set path to where bin/hadoop is available 配置Hadoop
export HADOOP_COMMON_HOME=/csh/link/hadoop
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/csh/link/hadoop
#set the path to where bin/hbase is available 配置HBase
export HBASE_HOME=/csh/link/hbase
#Set the path to where bin/hive is available 配置Hive
export HIVE_HOME=/csh/link/hive
#Set the path for where zookeper config dir is 配置Zookeeper
export ZOOCFGDIR=/csh/link/zookeeper/conf

添加MySQL连接jar包到$SQOOP_HOME/lib目录中

mysql-connector-java-5.1.31-bin.jar

验证

1
2
3
4
5
6
7
8
9
[root@node1 sqoop]# bin/sqoop-version
Warning: /csh/link/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /csh/link/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/04/10 05:31:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015

发表于   |   分类于   |     |   阅读次数

导入数据

1
2
3
create database test;
use test;
source /home/xxx/data/test.sql

导出数据

1
bin/mysqldump -u username -p [-d] database_name > xxx.sql

如果没有-d， 则导出表结构和数据；如果有-d，则只导出表结构

例如：

1
bin/mysqldump -u root -p test > test.sql

发表于   |   分类于   |     |   阅读次数

1.下载解压安装包
下载链接：apache-hive-1.2.1-bin.tar.gz

1
tar -xvf apache-hive-1.2.1-bin.tar.gz

2.启动Hadoop
可以参考：Hadoop2.x配置HA

4.在$HIVE_HOME/bin目录下执行

1
./hive

5.创建表test

1
2
3
4
5
[root@node1 bin]# ./hive
Logging initialized using configuration in jar:file:/csh/software/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
hive> create table test (a int);
OK
Time taken: 41.704 seconds

6.访问HDFS文件系统可以查看到test文件夹【创建一个表，就会在/user/hive/warehouse文件下创建对应的文件夹】
Hive安装配置

发表于   |   分类于   |     |   阅读次数
Node Name Master ZooKeeper RegionServer
node1 yes yes no
node2 backup yes yes
node3 no yes yes

1.配置conf/regionservers

1
2
node2
node3

2.创建conf/backup-masters

1
node2

3.配置 hbase-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://node1:9000/hbase</value>
</property>
<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/csh/hadoop/zookeeper</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>指定HBase的运行模式：false：单机模式或者伪分布模式； true：完全分布式模式</description>
</property>
<property>
    <name>hbase.master</name>
    <value>hdfs://node1:60000</value>
    <description>指定master位置</description>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>node1,node2,node3</value>
</property>

4.将配置文件拷贝到node2、node3中

1
2
scp /csh/software/hbase-1.1.4/conf/* root@node2:/csh/software/hbase-1.1.4/conf/
scp /csh/software/hbase-1.1.4/conf/* root@node3:/csh/software/hbase-1.1.4/conf/

5.启动HBase（在node1启动）

1
./start-hbase.sh

6.jps查看结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
//node1
4339 Jps
4122 HMaster
4047 HQuorumPeer

//node2
3313 HQuorumPeer
3810 Jps
3447 HMaster
3374 HRegionServer

//node3
2981 HQuorumPeer
3048 HRegionServer
3230 Jps