How to combine multiple text files into one text file ordered by date created?

Question

I'm a newbie so please help:

I keep a journal on my iPhone using Scratch that outputs all notes I make to a separate .txt file stored in Dropbox.

I have this synced with my Ubuntu 14.04 system, so in my Files I have a folder with all the text files stored in it here:

/home/stuart/Dropbox/Scratch

I want to run a command that concatenates all these files into one file, with the following provisos:

1. Ordered by date created (earliest file first)
2. Print the date of the file on a separate line before the contents of the file
3. Include a blank line followed by some kind of separator line after each file

So the output file has entries that look something like this:

12-01-2014 11:01 AM:
A coffee shop in Israel. The sign outside reads:
"Coffee" - 9 shekel
"Coffee please" - 8 shekel
"Good morning, could I have a coffee please?" - 7 shekel

--
25-01-2014 11:01 AM:
You cannot outperform your ego - ole Gunnar solskjaer

--

And so on. I used to use a different app that did this kind of auto append but I have no idea how to replicate it.

I've looked over a lot of the help files on here but I've not found any that can help with the output I have in mind.

Any help greatly appreciated!

---

MORE INFO

I tried creating the script suggested below and followed the steps. However I get this response:

stuart@StudioClough:/home$ chmod +x $HOME/my_concat

stuart@StudioClough:/home$ ./my_concat /home/stuart/Dropbox/Scratch > new_concatenated_file

bash: new_concatenated_file: Permission denied

Do I have to somehow run it as sudo?

Answer 1

It can be done with a python script, with one sidenote: I took the modification date instead of the creation date, since the creation date will almost certainly not match the real creation date: it is the date the file was copied to the computer, while the modification date seems unchanged during copying (see discussion at @cOrps answer). You will have to see if it works in your situation.

If that is acceptable for you, you can use the script below to create a combined file with your notes. It reads the notes, sorts them and appends them to a text file (creates it if it doesn't exist).

The good news is that you can append your new notes to the same file without overwriting the old ones.

Example output:

Mon Sep 29 08:48:31 2014
This is my first note.
As you can read, I am not really awake yet.

----------
Mon Sep 29 09:04:06 2014
It is really time I am going to eat something.
I am a bit hungry.
Making it a bit longer.

----------

How to use:

• Copy the script below into an empty file, save it as add_notes.py
• change the directories for files_dir (where your notes are) and the file in which you want to save the notes: combined_file (the script creates the file if it doesn't exist)

• run the script in a terminal window by typing the command:

  python3 /path/to/add_notes.py
  

The script:

#!/usr/bin/env python3

import os
import time
import subprocess

# --------------------------------------------------------
files_dir = "/path/to/your/textfiles"
combined_file = "/path/to/your/combined/file.txt"
# ---------------------------------------------------------
notes = []

if not os.path.exists(combined_file):
    subprocess.Popen(["touch", combined_file])

def read_file(file):
    with open(file) as note:
        return note.read()

def append_file(combined_file, text):
    with open(combined_file, "a") as notes:
        notes.write(text)

for root, dirs, files in os.walk(files_dir):
    for name in files:
        subject = root+"/"+name
        cr_date_text = time.ctime(os.path.getmtime(subject))
        cr_date_n = os.stat(subject).st_mtime
        notes.append((cr_date_n, cr_date_text, subject))

notes.sort(key=lambda x: x[0])

for note in notes:
    text = note[1]+"\n"+read_file(note[2])+"\n"+"-"*10+"\n"
    append_file(combined_file, text)

Answer 2

Here is a bash solution. It should work if you are using the ext4 file system. It is using file creation date that ext4 store in crtime field.

Create this script anywhere. Let's say my_concat in your $HOME directory (in your case it is /home/stuart):

#!/bin/bash

get_crtime() {
    for target in "${@}"; do
        inode=$(ls -di "${target}" | cut -d ' ' -f 1)
        fs=$(df  --output=source "${target}"  | tail -1)
        crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null | 
        grep -oP 'crtime.*--\s*\K.*')
        printf "%s\n" "${crtime}"
    done
}

get_epoch_crtime(){
    date --date "$(get_crtime $1)" +%s
}

get_epoch_mtime() {
    stat -c %Y $1
}

# takes two date as input, returns earlier date
get_earlier_time(){
    if [[ "$1" -lt "$2" ]]; then
        echo $(date -d @$1 +%m/%d/%Y:%H:%M:%S)
    else
        echo $(date -d @$2 +%m/%d/%Y:%H:%M:%S)
    fi
}

if [ $# != 1 ]; then
    echo "Required only one argument - full path to folder"
    echo "Usage example:"
    echo "$0 /var/log/syslog/"
    exit 1
fi

if [ -d "$1" ]; then
    cd $1
    for file in *
    do 
        echo $(get_earlier_time $(get_epoch_crtime $file) $(get_epoch_mtime $file))
        cat $file
        echo -e "\n-------"
    done
else
    echo "The folder specified is not exists ($1). Please enter full path"
fi

Make it executable:

chmod +x $HOME/my_concat

Now go to your $HOME folder and run script. Script will ask you your password because script uses sudo:

./my_concat /home/stuart/Dropbox/Scratch > new_concatenated_file

Now read new_concatenated_file using some editor:

gedit new_concatenated_file

This script uses both creation date and modify date, after comparison it takes the earliest one.

Sources

1. About creation date
2. Creation date in other files systems
3. Script to find creation date