Hadoop之HDFS
HDFS的Shell操作
-
启动Hadoop集群（方便后续测试）
[atguigu@hadoop102 ~]$ sbin/start-dfs.sh [atguigu@hadoop102 ~]$ sbin/start-yarn.sh -
-help：输出这个命令参数
[atguigu@hadoop102 ~]$ hadoop fs -help rm -
-ls：显示目录信息
[atguigu@hadoop102 ~]$ hadoop fs -ls / -
-mkdir：在HDFS上创建目录
[atguigu@hadoop102 ~]$ hadoop fs -mkdir -p /user/atguigu/input [atguigu@hadoop102 ~]$ hadoop fs -mkdir /bigdata0523 -
-moveFromLocal：从本地剪切粘贴到HDFS
[atguigu@hadoop102 hadoop-3.1.3]$ vim qiangge.txt --- 只年说：流年同学你好，可以加个微信吗？ --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -moveFromLocal ./qiangge.txt /bigdata0523 -
-appendToFile：追加一个文件到已经存在的文件末尾
[atguigu@hadoop102 hadoop-3.1.3]$ vim pantongxue.txt --- 流年同学说：你说加就加啊。不给！！ --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -appendToFile pantongxue.txt /bigdata0523/qiangge.txt -
-cat：显示文件内容
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -cat /bigdata0523/qiangge.txt 2025-07-05 12:03:57,587 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 只年说：流年同学你好，可以加个微信吗？ 流年同学说：你说加就加啊。不给！！ -
-chgrp、-chmod、-chown：Linux文件系统中的用法一样，修改文件所属权限
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -chmod u+x /bigdata0523/qiangge.txt -
-copyFromLocal：从本地文件系统中拷贝文件到HDFS路径去
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -copyFromLocal pantongxue.txt /bigdata0523-put效果和它相同，一般用put就行
[atguigu@hadoop102 hadoop-3.1.3]$ vim xinge.txt --- 龙哥说：你们俩在干啥，带我一个呗！！！ --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -put xinge.txt /bigdata0523 -
-copyToLocal：从HDFS拷贝到本地
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -copyToLocal /bigdata0523/qiangge.txt .-get效果和它相同，一般用get就行
[atguigu@hadoop102 hadoop-3.1.3]$ vim mengmeng.txt --- 被班主任耽误的舞蹈选手 --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -moveFromLocal mengmeng.txt /bigdata0523 [atguigu@hadoop102 hadoop-3.1.3]$ ll | grep mengmeng [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -get /bigdata0523/mengmeng.txt . [atguigu@hadoop102 hadoop-3.1.3]$ ll | grep mengmeng -rw-r--r--. 1 atguigu atguigu 34 7月 5 12:34 mengmeng.txt -
-cp：从HDFS的一个路径拷贝到HDFS的另一个路径
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -cp /bigdata0523/mengmeng.txt / -
-mv：在HDFS目录中移动文件
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /mengmeng.txt /user/atguigu/input # 同时还具备改名的功能，下方这种写法 dageoge.txt前方没有明确路径，默认是放到 /user/atguigu 下面了。可以理解为类似于linux也有用户家目录的概念 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /user/atguigu/input/mengmeng.txt dagaoge.txt # 移动的同时还可以更改名称 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /user/atguigu/dagaoge.txt /user/atguigu/input/mengmeng.txt -
-get：等同于copyToLocal，就是从HDFS下载文件到本地
# 上方copyToLocal已经说过-get，此处不再操作 hadoop fs -get /bigdata0523/mengmeng.txt . -
-getmerge：合并下载多个文件，比如HDFS的目录/user/atguigu/test 下有多个文件：log.1，log.2，log.3…
# 不用看后缀名，后缀名没啥用，里面是合并后的内容。getmerge用的比较少 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -getmerge /bigdata0523 ./teacherAndStudent.avi [atguigu@hadoop102 hadoop-3.1.3]$ cat teacherAndStudent.avi 被班主任耽误的舞蹈选手 流年同学说：你说加就加啊。不给！！ 只年说：流年同学你好，可以加个微信吗？ 流年同学说：你说加就加啊。不给！！ 龙哥说：你们俩在干啥，带我一个呗！！！ -
-put：等同于copyFromLocal
# 上方copyFromLocal已经说过-put，此处不再操作 hadoop fs -put xinge.txt /bigdata0523 -
-tail：显示一个文件的末尾
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -tail /input/README.txt # -n查看指定多少行不能用！实时查看可以跟-f [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -tail -f /input/README.txt# 当我们把副本数改为5，再去做操作的时候报错 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -appendToFile mengmeng.txt /input/README.txt 2025-07-05 13:11:46,387 WARN hdfs.DataStreamer: DataStreamer Exception java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK]], original=[DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1304)at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1372)at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1598)at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1499)at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:720) appendToFile: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK]], original=[DatanodeInfoWithStorage[
