2

I wish to move files in a hadoop dir in a timely manner.

The hadoop dir contains 1000 files with the same extension. I wish to move 100 of them every 10 minutes. I can setup a cron job to move the files every 10 minutes but I don't know how to specify the number of files to be moved.

hdfs dfs -ls /src/ | tail -100 | xargs hdfs dfs -mv {} / dest/

Any command to use?

Thanks in advance.

Marcellinov
  • 3,349
  • 3
  • 15
  • 23
GulshanS
  • 21

3 Answers3

1

You can use mv like this: mv -t target file1 file2 file3 ...

ls | head -n 100 | xargs mv -t destination
Gustav
  • 459
  • 2
  • 13
1

What about using a script like this:

#!/bin/bash

Source="/where/they/are/now/*"
Destination="/where/they/will/go"

while true; do
  count=0
  for file in $Source; do
    if [ $((count++)) -eq 100 ];then
      break
    else mv "$file" "$Destination"
    fi
  done
  sleep 10m
done
bashBedlam
  • 1,022
0

For the future generations:

hdfs dfs -mv `hdfs dfs -ls -C {your_src_hdfs_path} | head -{number_of_files}` {your_dest_hdfs_path}
user1620719
  • 1