一、HDFS基础操作
以root用户登录,创建如下HDFS目录:
/dw/yourname/input
hadoop fs -mkdir -p /dw/zhanggengchen/input
/dw/yourname/output
hadoop fs -mkdir -p /dw/zhanggengchen/output
输出结果:
[root@master hadoop-mapreduce]# hadoop fs -ls /dw/zhanggengchen
Found 2 items
drwxr-xr-x - root supergroup 0 2025-07-03 18:14 /dw/zhanggengchen/input
drwxr-xr-x - root supergroup 0 2025-07-03 18:15 /dw/zhanggengchen/output
上传本地文件 /root/dataExercise/people.txt 到 /dw/yourname/input 目录下。
hadoop fs -put /root/dataExercise/people.txt /dw/zhanggengchen/input/
列出 /dw/yourname/input 目录内容,并显示 people.txt 文件前5行内容。
hadoop fs -cat /dw/zhanggengchen/input/people.txt | head -n 5
输出结果:
[root@master hadoop-mapreduce]# hadoop fs -cat /dw/zhanggengchen/input/people.txt | head -n 5
Michael, 29
Andy, 30
Justin, 19
二、Sqoop数据导入
Mysql数据库连接信息:
主机:localhost
端口:3306
用户名:root
密码:bimao123
需求
使用Sqoop将MySQL数据库的bigdatafamily中的customers表导入到HDFS目录 /dw/yourname/customers,要求:
数据格式为文本,字段分隔符为\t
vim TODO3.sh
sqoop import \
--connect jdbc:mysql://localhost:3306/bigdatafamily \
--username root \
--password bimao123 \
--table customers \
--target-dir /dw/zhanggengchen/customers \
--fields-terminated-by '\t' \
--as-textfile
列出 /dw/yourname/customers 目录内容,并显示部分导入数据内容。
[root@master hadoop-mapreduce]# hadoop fs -ls /dw/zhanggengchen/customers
输出结果:
Found 5 items
-rw-r--r-- 1 root supergroup 0 2025-07-03 18:32 /dw/zhanggengchen/customers/_SUCCESS
-rw-r--r-- 1 root supergroup 3144522 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00000
-rw-r--r-- 1 root supergroup 3143941 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00001
-rw-r--r-- 1 root supergroup 3144303 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00002
-rw-r--r-- 1 root supergroup 3144580 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00003
[root@master hadoop-mapreduce]# hadoop fs -cat /dw/zhanggengchen/customers/part-m-00000 | head -n 3
输出结果:
1000000 Quentin Shepard 32092 West 10th Street Prairie City SD 57649
1000001 Brandon Louis 1311 North 2nd Street Clearfield IA 50840
1000002 Marilyn Ham 25831 North 25th Street Concord CA 94522