awk -F'|' 'NR == FNR { $1 = ""; $2 = ""; seen[$0]++ } NR != FNR { orig = $0; $1 = ""; $2 = ""; if (!seen[$0]) print orig }' first.txt second.txt
That is, for example when the first file contains:
1234|12|Bill|Blatt|programmer
3243|34|Bill|Blatt|dentist
98734|25|Jack|Blatt|programmer
748567|31|Mark|Spark|magician
And the second file contains:
123|12|Bill|Blatt|programmer
3243|4|Bill|Blatt|dentist
934|25|Jack|Blatt|prograbber
30495|89|Dave|Scratt|slobber
The lines that are unique in the second file ignoring the first two files are:
934|25|Jack|Blatt|prograbber
30495|89|Dave|Scratt|slobber
The one-liner is easier to see expanded to multiple lines:
awk -F'|' -v OFS='|' '
NR == FNR {
$1 = "";
$2 = "";
seen[$0]++;
}
NR != FNR {
orig = $0;
$1 = "";
$2 = "";
if (!seen[$0]) print orig
}' first.txt second.txt
Here's how it works:
-F'|'
-- use pipe as the field separator.NR == FNR
-- this matches lines in the first input file.$1
, $2
), and using the rest ($0
) as the key, and count it.NR != FNR
-- this matches lines not in the first input file.Notice that this approach also preserves the original order of the lines in the second file.